Abstract
Objective
To determine reliability among four experienced and calibrated readers in cross-sectional and longitudinal semi-quantitative MRI assessments of knee osteoarthritis (OA) in the Multicenter Osteoarthritis (MOST) study.
Design
From all MOST participants with at least one knee with readable 60-month and 84-month paired knee MRIs (1.0 T extremity systems), we selected 10 subjects having a spectrum of baseline disease severity of cartilage, bone marrow lesions, and meniscal damage and a spectrum of longitudinal changes in severity at 24 months follow-up. MRIs were independently assessed using the WORMS grading system by four musculoskeletal radiologists with the chronological sequence known to the readers. Kappa statistics were used to determine agreement between each pair of readers and Kendall's coefficient of concordance to determine average agreement across readers.
Results
For most features, cross-sectional reliability was substantial to almost perfect. Regarding longitudinal reliability (detection of longitudinal change), inter-reader reliability as weighted kappa values ranged from 0.62 to 0.78 for cartilage damage, 0.75–0.88 for bone marrow lesions, 0.75–0.92 for meniscal tears, 0.67–0.95 for meniscal extrusion, 0.51–0.77 for bone attrition, 0.43–0.76 for osteophytes, 0.31–0.70 for Hoffa-synovitis, and 0.47–0.85 for effusion-synovitis. Kendall's coefficient ranged from 0.65 to 0.98.
Conclusion
High levels of cross-sectional reliability and moderate to high longitudinal reliability was achieved using four experienced readers in semiquantitative MRI-assessment of most knee OA features.
Keywords: Osteoarthritis, Magnetic resonance imaging, Reliability, Cross-sectional study, Longitudinal study
1. Introduction
Magnetic resonance imaging (MRI)-based semiquantitative assessment of knee osteoarthritis (OA) has proven to be a valuable method for performing whole-organ joint evaluation in observational cross-sectional and longitudinal studies of knee OA including clinical trials [1]. Semiquantitative scoring enables multi-tissue joint assessment including articular cartilage, meniscus and ligaments, bone marrow abnormalities, osteophytes, synovitis and effusion, cystic lesions and loose bodies, using knee MRI acquisition techniques that are commonly applied in the clinical routine [2].
The Whole Organ Magnetic Resonance Imaging Score (WORMS) was the first published scoring system for knee OA assessment and has been used extensively for almost two decades in many OA studies world-wide [3].
For meaningful data interpretation, it is paramount to ensure both cross-sectional and longitudinal reliability between MRI readers in large observational longitudinal OA studies. While cross-sectional reliability results between two trained and calibrated readers have been presented for all available MRI scoring systems [[3], [4], [5], [6]] and assessed in many other studies, data on longitudinal reliability (regarding the detection of change over time) is sparse [7] and particularly, agreement among more than two readers has not been presented to date.
The aim of this study was to determine reliability among four experienced and calibrated musculoskeletal radiologists in cross-sectional and longitudinal knee MRI assessments in the Multicenter Osteoarthritis (MOST) study using the WORMS instrument.
2. Methods
2.1. Study design and subjects
Subjects were participants in the Multicenter Osteoarthritis Study (MOST), a prospective observational study of 3026 participants with the goal of identifying risk factors for incident and progressive knee OA in a population with or at high risk of developing OA. The Health Insurance Portability and Accountability Act-compliant study protocol was approved by the Institutional Review Boards at the University of Iowa, University of Alabama at Birmingham, University of California at San Francisco and Boston University School of Medicine, and written informed consent was obtained from all patients.
The parent MOST study read MRIs from all participants with at least one knee with a readable (based on quality assurance review by a trained MRI reader) 60-month and 84-month pair of knee MRIs. If both knees had readable MRIs, then one knee was randomly selected. Selection of knees for the present study was based on a combination of radiographic status from knee x-rays and the presence, and longitudinal changes, of cartilage damage, bone marrow lesions and meniscal damage on MRI, with a goal of including a spectrum of baseline severity, and of change, in these MRI features. A convenience sample of ten knees was selected to be included in this reliability study: 4 had incident radiographic OA on paired x-rays and change in at least one MRI feature; 3 knees showed progression of existing radiographic OA and change in at least one of the MRI features; and 3 knees had no change on radiographic status and no change in the 3 MRI features.
2.2. MRI acquisition and interpretation
Knee MRIs were acquired at 60-month and 84-month follow-up with a 1.0 T dedicated extremity unit (OrthOne™, GE HealthCare, Milwaukee, WI) using sagittal and axial fat-suppressed fast spin-echo proton density-weighted sequences, and a short tau inversion-recovery-STIR sequence in the coronal plane. MRIs were independently assessed using the WORMS grading system [3] by four musculoskeletal radiologists (R1 (AG), R2 (FWR), R3 (MDC), and R4 (MDM), with the chronological sequence known to the readers who were blinded to all radiographic and clinical information. All readers had extensive experience in semiquantitative MRI assessment of knee OA (R1 – 12 years; R2 – 10 years; R3 and R4 – 4 years each at the time of assessment). The following structures/features were assessed: cartilage damage (0–6), osteophytes (0–7), bone marrow lesions (0–3), subchondral cysts (0–3), subchondral bone attrition (0–3), meniscal damage (0–4), meniscal extrusion (0–2), Hoffa-synovitis (0–3), effusion-synovitis (0–3), and cruciate and collateral ligament pathology (0–2). The following features were assessed as absent (0) or present [1]: popliteal cysts, tibio-fibular cysts, loose intra-articular bodies, anserine bursitis, and pre-patellar bursitis. In addition to the WORMS grades for cartilage damage and bone marrow lesions, the readers assessed within-grade changes over time in order to increase sensitivity to change (Fig. 1) [8]. The most experienced readers (R1 and R2) re-assessed the same sample of knee MRIs at least months after the first reading was performed. Prior to the MRI assessment of included participants, the four readers participated in a 3-h calibration session using paired 60-month and 84-month MRIs from 10 knees from 10 different MOST participants (not included in this study).
Fig. 1.
Coronal fat-suppressed proton density-weighted MRI from 60-month (A) and 84-month (B) visits showing a within-grade increase of a grade 1 bone marrow lesion (BML) size located at the central subregion of the medial femoral condyle (arrow – A; arrowhead – B). Although there is an evident increase of BML size between A and B, both lesions do not reach more than 25% of the whole central subregion of the medial femoral condyle area (BML grade must be considered then as grade 1 in the WORMS system for A and B). In another participant, coronal fat-suppressed proton density-weighted MRI from 60-month (C) and 84-month (D) visits show a full-grade increase bone marrow lesion (BML) size located at the central subregion of the medial tibia, involving less than 25% of the whole area in C – grade 1 (arrow), whereas in D it involves between 25% and 50% of the whole area, consistent with a grade 2 (arrows). Note a small BML near the subspinous region at C (arrowhead).
2.3. Statistical analysis
Linear weighted kappa statistics were applied to determine the reliability between each pair of readers (R1 vs. R2; R1 vs. R3; R1 vs. R4; R2 vs. R3; R2 vs. R4; R3 vs. R4), as well as for intra-reader reliability. For cross-sectional reliability assessment, the full spectrum of WORMS feature's scores from both the 60 and 84-month timepoints were included. For longitudinal reliability assessment, changes from 60-month to 84-month visits were assessed. Longitudinal change was defined on a 5-point scale: 1 – decrease in grade; 2 – no change; 3 – within-grade increase; 4 – full grade increase; 5 – more than a full grade increase. The weighted kappa tested agreement between pairs of readers. For kappa statistics, agreement is defined as poor (<0.2), fair (0.21–0.4), moderate (0.41–0.6), substantial (0.61–0.8), almost perfect (0.81–0.99), and perfect (1.0) [9]. With a ranked outcome and multiple readers, we also computed a Kendall's coefficient of concordance (Kendall's W) which provides an average agreement among readers and ranges from 0 (no agreement) to 1 (complete agreement). All statistical calculations were performed using SAS® software (Version 9.1 for Windows; SAS Institute; Cary, NC).
3. Results
Ten participants were included in the analyses (one knee per participant). Participants were on average 65.4 years old (SD ± 7.4), 12 (60%) were female, and the mean body mass index was 29.8 (SD ± 5.0). Several features exhibited low prevalence (cross-sectional analysis) or few or no incident changes (longitudinal analysis) and were excluded for both cross-sectional and longitudinal analysis, i.e. cruciate and collateral ligament pathology, popliteal cysts, tibio-fibular cysts, loose intra-articular bodies, anserine bursitis, and pre-patellar bursitis.
The paired inter-reader weighted kappa values for cross-sectional and longitudinal reliability analyses for the structures/features assessed are displayed in Table 1. The ranges for inter-reader weighted kappa values for cross-sectional and longitudinal reliability, respectively, were 0.77–0.87 and 0.62 to 0.78 for cartilage damage, 0.80 to 0.89 and 0.75 to 0.88 for BMLs, 0.92 to 0.96 and 0.75 to 0.92 for meniscal tears, 0.67 to 0.83 and 0.67 to 0.95 for meniscal extrusion, 0.71 to 0.88 and 0.51 to 0.77 for bone attrition, 0.47 to 0.80 and 0.43 to 0.76 for osteophytes, 0.16 to 0.60 and 0.31 to 0.70 for Hoffa-synovitis, and 0.57 to 0.89 and 0.47 to 0.85 for effusion synovitis.
Table 1.
Paired cross-sectional (CS) and longitudinal (L) reliability for main features assessed in WORMS. Kappa values (95% confidence intervals). BMLs: bone marrow lesions.
| Category | R1 vs. R2 | R1 vs. R3 | R1 vs. R4 | R2 vs. R3 | R2 vs. R4 | R3 vs. R4 | |
|---|---|---|---|---|---|---|---|
| Cartilage morphology | CS | 0.85 (0.81,0.90) | 0.86 (0.82,0.90) | 0.82 (0.77,0.87) | 0.77 (0.71,0.82) | 0.79 (0.73,0.84) | 0.87 (0.82,0.91) |
| L | 0.78 (0.67,0.90) | 0.77 (0.66,0.88) | 0.63 (0.49,0.78) | 0.63 (0.49,0.77) | 0.62 (0.46,0.77) | 0.70 (0.56,0.84) | |
| Osteophytes | CS | 0.64 (0.57,0.72) | 0.52 (0.46,0.59) | 0.47 (0.40,0.54) | 0.49 (0.42,0.56) | 0.48 (0.41,0.55) | 0.80 (0.76,0.84) |
| L | 0.61 (0.47,0.75) | 0.58 (0.43,0.72) | 0.54 (0.40,0.69) | 0.48 (0.33,0.64) | 0.43 (0.27,0.58) | 0.76 (0.66,0.86) | |
| BMLs | CS | 0.89 (0.84,0.94) | 0.81 (0.74,0.88) | 0.81 (0.74,0.88) | 0.80 (0.74,0.87) | 0.83 (0.76,0.90) | 0.86 (0.81,0.92) |
| L | 0.88 (0.80,0.96) | 0.82 (0.73,0.92) | 0.80 (0.70,0.91) | 0.75 (0.63,0.87) | 0.79 (0.68,0.91) | 0.80 (0.69,0.91) | |
| Subchondral cysts | CS | 0.68 (0.46,0.90) | 0.54 (0.32,0.77) | 0.50 (0.27,0.72) | 0.51 (0.29,0.73) | 0.48 (0.26,0.69) | 0.93 (0.82,1.00) |
| L | 0.60 (0.23,0.97) | 0.70 (0.39,1.00) | 0.70 (0.39,1.00) | 0.60 (0.29,0.91) | 0.60 (0.29,0.91) | 1.00 (1.00,1.00) | |
| Bone attrition | CS | 0.79 (0.70,0.89) | 0.76 (0.69,0.83) | 0.71 (0.63,0.80) | 0.79 (0.70,0.88) | 0.80 (0.70,0.89) | 0.88 (0.80,0.95) |
| L | 0.71 (0.48,0.95) | 0.67 (0.46,0.88) | 0.61 (0.38,0.83) | 0.51 (0.24,0.78) | 0.55 (0.28,0.82) | 0.77 (0.58,0.97) | |
| Meniscal tears | CS | 0.94 (0.90,0.98) | 0.92 (0.88,0.97) | 0.92 (0.87,0.97) | 0.97 (0.93,1.00) | 0.92 (0.87,0.98) | 0.96 (0.92,0.99) |
| L | 0.92 (0.81,1.00) | 0.84 (0.68,1.00) | 0.75 (0.55,0.95) | 0.91 (0.79,1.00) | 0.81 (0.63,1.00) | 0.89 (0.73,1.00) | |
| Meniscal extrusion | CS | 0.83 (0.69,0.98) | 0.82 (0.67,0.97) | 0.86 (0.72,1.00) | 0.71 (0.52,0.90) | 0.67 (0.46,0.88) | 0.81 (0.65,0.97) |
| L | 0.81 (0.62,1.00) | 0.75 (0.55,0.95) | 0.95 (0.87,1.00) | 0.67 (0.42,0.91) | 0.77 (0.57,0.96) | 0.81 (0.60,1.00) | |
| Hoffa-synovitis | CS | 0.60 (0.38,0.83) | 0.58 (0.36,0.80) | 0.45 (0.24,0.66) | 0.16 (0.10,0.42) | 0.24 (0.04,0.52) | 0.59 (0.32,0.82) |
| L | 0.64 (0.00,1.00) | 0.44 (0.21,1.00) | 0.31 (0.27,0.90) | 0.64 (0.00,1.00) | 0.45 (0.15,1.00) | 0.77 (0.35,1.00) | |
| Effusion-synovitis | CS | 0.89 (0.75,1.00) | 0.88 (0.72,1.00) | 072 (0.51,0.92) | 0.78 (0.56,0.99) | 0.62 (0.40,0.84) | 0.57 (0.32,0.82) |
| L | 0.85 (0.57,1.00) | 0.64 (0.28,1.00) | 0.85 (0.57,1.00) | 0.47 (0.08,0.86) | 0.70 (0.28,1.00) | 0.47 (0.08,0.86) |
The multi-reader inter-reader Kendall's coefficients of concordance for cross-sectional and longitudinal reliability analyses for the structures/features assessed are displayed in Table 2. Kendall's coefficients of concordance ranged from 0.65 (Hoffa's synovitis) to 0.98 (meniscal tears) for cross-sectional reliability; for longitudinal reliability, it ranged from 0.65 (Hoffa's synovitis) to 0.91 (meniscal extrusion).
Table 2.
Multi-reader cross-sectional and longitudinal reliability for main features assessed in WORMS: Kendall's Coefficient of Concordance.
| Category | Kendall's Coefficient of Concordance | |
|---|---|---|
| Cartilage morphology | Cross-sectional | 0.91 |
| Longitudinal | 0.80 | |
| Osteophytes | Cross-sectional | 0.76 |
| Longitudinal | 0.72 | |
| Bone Marrow Lesions | Cross-sectional | 0.89 |
| Longitudinal | 0.89 | |
| Subchondral cysts | Cross-sectional | 0.72 |
| Longitudinal | 0.79 | |
| Bone attrition | Cross-sectional | 0.89 |
| Longitudinal | 0.75 | |
| Meniscal tears | Cross-sectional | 0.98 |
| Longitudinal | 0.90 | |
| Meniscal extrusion | Cross-sectional | 0.85 |
| Longitudinal | 0.91 | |
| Hoffa-synovitis | Cross-sectional | 0.65 |
| Longitudinal | 0.65 | |
| Effusion-synovitis | Cross-sectional | 0.88 |
| Longitudinal | 0.81 |
Intra-reader weighted-kappa values for cross-sectional and longitudinal reliability assessments showed substantial to perfect agreement among features evaluated (Table 3).
Table 3.
Cross-sectional (CS) and longitudinal (L) intra-reader reliability for readers R1 and R2. Kappa values (95% confidence intervals). BMLs (bone marrow lesions).
| Category | R1 | R2 | |
|---|---|---|---|
| Cartilage morphology | CS | 0.92 (0.80,1.00) | 0.98 (0.97,1.00) |
| L | 0.79 (0.73,0.86) | 0.99 (0.97,1.00) | |
| Osteophytes | CS | 0.70 (0.50,0.87) | 0.97 (0.93,1.00) |
| L | 0.76 (0.58,0.90) | 0.97 (0.92,1.00) | |
| BMLs | CS | 0.87 (0.71,0.98) | 0.99 (0.97,1.00) |
| L | 0.81 (0.62,0.95) | 0.98 (0.96,1.00) | |
| Subchondral cysts | CS | 1.00 (1.00,1.00) | 0.98 (0.97,1.00) |
| L | 0.91 (0.89,0.92) | 0.98 (0.98,0.99) | |
| Bone attrition | CS | 0.81 (0.74,0.86) | 0.93 (0.93,0.93) |
| L | 0.80 (0.73,0.86) | 0.93 (0.93,0.93) | |
| Meniscal tears | CS | 0.97 (0.93,1.00) | 1.00 (1.00,1.00) |
| L | 0.94 (0.83,1.00) | 1.00 (1.00,1.00) | |
| Meniscal extrusion | CS | 0.85 (0.66,1.00) | 1.00 (1.00,1.00) |
| L | 0.68 (0.47,0.88) | 1.00 (1.00,1.00) | |
| Hoffa Synovitis | CS | 0.74 (0.45,0.98) | 1.00 (1.00,1.00) |
| L | 1.00 (1.00,1.00) | 1.00 (1.00,1.00) | |
| Effusion Synovitis | CS | 0.83 (0.67,1.00) | 1.00 (1.00,1.00) |
| L | 0.51 (0.07,0.95) | 1.00 (1.00,1.00) |
4. Discussion
In this multi-reader reliability study, we demonstrated that the inter-reader assessment of most common and clinically relevant OA features showed substantial to high agreement cross-sectionally and moderate to high agreement when assessing longitudinal changes. To the best of our knowledge, our study is the first to evaluate the inter-reader agreement of longitudinal changes in MRI OA features using multiple readers.
Semiquantitative scoring systems (such as WORMS) helped the understanding of the pathophysiology and the natural history of knee OA, highlighting several factors associated with both incidence and progression of disease [1]. These systems have been applied in clinical trials of knee OA to assess treatment efficacy and to monitor structural change over time in different tissues such as cartilage, BMLs, menisci, and synovitis [10,11]. While in most of these studies the outcome of interest was longitudinal change in MRI features previous longitudinal OA studies mainly reported reliability considering each time point as an independent measurement (cross-sectional reliability) and/or using a maximum of two readers for reliability assessment [[3], [4], [5], [6],[12], [13], [14]]. Ideally, reliability among multiple readers (as is the common situation in large epidemiologic studies) should be tested including agreement regarding longitudinal changes detected over time. Such information is paramount when considering including multiple readers in OA clinical trials when applying MRI. Whether longitudinal reliability in semiquantitative MRI assessment of knee OA using multiple experienced readers is equivalent to the cross-sectional reliability has not been reported to date.
Our study demonstrated that, for most relevant MRI features of knee OA, longitudinal reliability between the four experienced readers was moderate to high. Compared to cross-sectional reliability, longitudinal reliability seems to be slightly inferior for most features assessed. This suggests that there are differences in how readers classified change in structures/features over time using the scoring options for change. According to the scoring system used in our study, an increase in score could happen in three different ways: a within-grade increase, a full grade increase, or more than a full grade increase. For example, readers could agree that there was an increase in grade over time for a given structure or feature but could disagree about how large an increase was. Despite this fact, we demonstrated that, overall, experienced MRI readers were able to provide reliable assessment of longitudinal changes for the most relevant OA features assessed including cartilage damage, bone marrow lesions, meniscal tears, and effusion-synovitis, with moderate to substantial agreement among multiple readers. Regarding cross-sectional reliability, our study showed comparable agreement using multiple pairs of readers as was previously reported using two readers only [[3], [4], [5], [6],[9], [10], [11], [12], [13], [14]], and for most MRI OA features, agreement was substantial to almost perfect. Scoring of Hoffa-synovitis appeared to be slightly less reliable cross-sectionally, and this is probably due to the non-specificity of this feature.
Some limitations need mentioning. The MRIs were presented sequentially, and readers were aware of the chronological order of images but were blinded to any clinical information. This could, perhaps, bias the readers to expect more change. However, there is evidence showing when readers are blinded to the chronological order of images, sensitivity to relevant changes decreases and this may induce to errors when compared to unblinded assessment [16]. An extremity 1.0 T MRI was used in our study, which limits the resolution of images due to the lower signal-to-noise ratio in comparison to higher magnetic fields, such as 3.0 T MRIs used in the Osteoarthritis Initiative [12]. This may have limited the detection of low-grade pathology or small changes over time regarding some features assessed. However, it was demonstrated that semiquantitative assessment of knee OA features at 1.0 T and at 1.5 T MRIs was comparable [15]. Although the cross-sectional reliability found in our study is comparable with previous studies using higher magnetic field systems if longitudinal reliability is also comparable has not been tested to date. Kappa values located at (or near) both ends of the reliability scale must be interpreted carefully, considering the low prevalence and missing change values of some of the features assessed including subchondral cysts and Hoffa synovitis. Kappa, but not Kendall's W, is highly influenced by feature prevalence. Sample size calculation was not performed; however, formal hypothesis testing was not the focus of this analysis.
In conclusion, moderate to high cross-sectional reliability and moderate to high longitudinal reliability could be achieved in our study using multiple experienced readers in semiquantitative MRI assessment of most knee OA features. This supports the feasibility of including multiple experienced MRI readers in cross-sectional and longitudinal knee OA studies.
Declaration of competing interest
Michel D. Crema, Frank W. Roemer, and Monica D. Marra are stockholders of Boston Imaging Core Lab (BICL), LLC. Ali Guermazi is president of BICL, LLC. He is also a consultant for MerckSerono, Pfizer, TissueGene, Roche, Galapagos, and AstraZeneca. There is no conflict of interest for the remaining authors.
Acknowledgements
We would like to thank all st’ at the Coordinating Center at the University of California at San Francisco. We would also like to thank all staff at the clinical sites in Iowa and Alabama. Finally, we would like to express our thanks to all participants of the MOST study.
Contributor Information
Michel D. Crema, Email: michelcrema@gmail.com.
Frank W. Roemer, Email: frank.roemer@uk-erlangen.de.
Michael C. Nevitt, Email: MNevitt@psg.ucsf.edu.
David T. Felson, Email: dfelson@bu.edu.
Monica D. Marra, Email: monicadiasmarra@gmail.com.
John A. Lynch, Email: jlynch@psg.ucsf.edu.
Irina Tolstykh, Email: Irina.Tolstykh@ucsf.edu.
Cora E. Lewis, Email: clewis@dopm.uab.edu.
James Torner, Email: james-torner@uiowa.edu.
Ali Guermazi, Email: guermazi@bu.edu.
Credit author statement
Conception and design: all authors; Analysis and interpretation of the data: MDC, DTF, AG, MCN, JN, JAL, IT, MDM, FWR; Drafting of the article: all authors; Critical revision of the article for important intellectual content: all authors; Final approval of the article: all authors; Provision of study materials or patients: DTF, MCN, JT, CEL; Statistical expertise: JN, IT.
Funding source
Supported by NIH grants from the National Institute of Aging to Drs. Lewis (U01-AG-18947), Torner (U01-AG-18832), Nevitt (U01-AG-19069), and Felson (U01-AG-18820) and NIH AR47785.
References
- 1.Guermazi A., Roemer F.W., Haugen I.K., Crema M.D., Hayashi D. MRI-based semiquantitative scoring of joint pathology in osteoarthritis. Nat. Rev. Rheumatol. 2013;9:236–251. doi: 10.1038/nrrheum.2012.223. [DOI] [PubMed] [Google Scholar]
- 2.Roemer F.W., Hunter D.J., Crema M.D., Kwoh C.K., Ochoa-Albiztegui E., Guermazi A. An illustrative overview of semi-quantitative MRI scoring of knee osteoarthritis: lessons learned from longitudinal observational studies. Osteoarthritis Cartilage. 2016;24:274–289. doi: 10.1016/j.joca.2015.08.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Peterfy C.G., Guermazi A., Zaim S., et al. Whole-organ magnetic resonance imaging score (WORMS) of the knee in osteoarthritis. Osteoarthritis Cartilage. 2004;12:177–190. doi: 10.1016/j.joca.2003.11.003. [DOI] [PubMed] [Google Scholar]
- 4.Kornaat P.R., Ceulemans R.Y., Kroon H.M., et al. MRI assessment of knee osteoarthritis: knee Osteoarthritis Scoring System (KOSS)-inter-observer and intra-observer reproducibility of a compartment-based scoring system. Skeletal Radiol. 2005;34:95–102. doi: 10.1007/s00256-004-0828-0. [DOI] [PubMed] [Google Scholar]
- 5.Hunter D.J., Lo G.H., Gale D., Grainger A.J., Guermazi A., Conaghan P.G. The reliability of a new scoring system for knee osteoarthritis MRI and the validity of bone marrow lesion assessment: BLOKS (Boston Leeds Osteoarthritis Knee Score) Ann. Rheum. Dis. 2008;67:206–211. doi: 10.1136/ard.2006.066183. [DOI] [PubMed] [Google Scholar]
- 6.Hunter D.J., Guermazi A., Lo G.H., et al. Evolution of semi-quantitative whole joint assessment of knee OA: MOAKS (MRI Osteoarthritis Knee Score) Osteoarthritis Cartilage. 2011;19:990–1002. doi: 10.1016/j.joca.2011.05.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Roemer F.W., Frobell R., Lohmander L.S., Niu J., Guermazi A. Anterior cruciate ligament osteoarthritis score (ACLOAS): longitudinal MRI-based whole joint assessment of anterior cruciate ligament injury. Osteoarthritis Cartilage. 2014;22:668–682. doi: 10.1016/j.joca.2014.03.006. [DOI] [PubMed] [Google Scholar]
- 8.Roemer F.W., Nevitt M.C., Felson D.T., et al. Predictive validity of within-grade scoring of longitudinal changes of MRI-based cartilage damage and bone marrow lesion assessment in the tibio-femoral joint – the MOST study. Osteoarthritis Cartilage. 2012;20:1391–1398. doi: 10.1016/j.joca.2012.07.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 10.Roemer F.W., Aydemir A., Lohmander S., et al. Structural effects of srpifermin in knee osteoarthritis: a post-hoc analysis on cartilage and non-cartilaginous tissue alterations in a randomized controlled trial. BMC Muscoskel. Disord. 2016;17:267. doi: 10.1186/s12891-016-1128-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Guermazi A., Kalsi G., Niu J., et al. Structural effects of intra-articular TGF-B1 in moderate to advanced knee osteoarthritis: MRI-based assessment in a randomized controlled trial. BMC Muculoskelet Disord. 2017;18:461. doi: 10.1186/s12891-017-1830-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gersing A.S., Schwaiger B.J., Nevitt M.C., et al. Is weight loss associated with less progression of changes in knee articular cartilage among obese and overweight patients as assessed with MR imaging over 48 months? Data from the Osteoarthritis Initiative. Radiology. 2017;284:508–520. doi: 10.1148/radiol.2017161005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Roemer F.W., Zhang Y., Niu J., et al. Tibiofemoral joint osteoarthritis: risk factors for MR-depicted fast cartilage loss over a 30-month period in the multicenter osteoarthritis study. Radiology. 2009;252:772–780. doi: 10.1148/radiol.2523082197. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Roemer F.W., Kwoh C.K., Hannon M.J., et al. What comes first? Multitissue involvement leading to radiographic osteoarthritis: magnetic resonance imaging-based trajectory analysis over four years in the osteoarthritis initiative. Arthritis Rheum. 2015;67:2085–2096. doi: 10.1002/art.39176. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Roemer F.W., Lynch J.A., Niu J., et al. A comparison of dedicated 1.0 T extremity MRI vs large-bore 1.5 T MRI for semiquantitative whole organ assessment of osteoarthritis: the MOST study. Osteoarthritis Cartilage. 2010 Feb;18:168–174. doi: 10.1016/j.joca.2009.08.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Felson D.T., Nevitt M.N. Blinding images to sequence in osteoarthritis: evidence from other diseases. Osteoarthritis Cartilage. 2009;17:281–283. doi: 10.1016/j.joca.2008.09.008. [DOI] [PMC free article] [PubMed] [Google Scholar]