The MOCART (Magnetic Resonance Observation of Cartilage Repair Tissue) 2.0 Knee Score and Atlas

Abstract

Objective

Since the first introduction of the MOCART (Magnetic Resonance Observation of Cartilage Repair Tissue) score, significant progress has been made with regard to surgical treatment options for cartilage defects, as well as magnetic resonance imaging (MRI) of such defects. Thus, the aim of this study was to introduce the MOCART 2.0 knee score — an incremental update on the original MOCART score — that incorporates this progression.

Materials and Methods

The volume of cartilage defect filling is now assessed in 25% increments, with hypertrophic filling of up to 150% receiving the same scoring as complete repair. Integration now assesses only the integration to neighboring native cartilage, and the severity of surface irregularities is assessed in reference to cartilage repair length rather than depth. The signal intensity of the repair tissue differentiates normal signal, minor abnormal, or severely abnormal signal alterations. The assessment of the variables “subchondral lamina,” “adhesions,” and “synovitis” was removed and the points were reallocated to the new variable “bony defect or bony overgrowth.” The variable “subchondral bone” was renamed to “subchondral changes” and assesses minor and severe edema-like marrow signal, as well as subchondral cysts or osteonecrosis-like signal. Overall, a MOCART 2.0 knee score ranging from 0 to 100 points may be reached. Four independent readers (two expert readers and two radiology residents with limited experience) assessed the 3 T MRI examinations of 24 patients, who had undergone cartilage repair of a femoral cartilage defect using the new MOCART 2.0 knee score. One of the expert readers and both inexperienced readers performed two readings, separated by a four-week interval. For the inexperienced readers, the first reading was based on the evaluation sheet only. For the second reading, a newly introduced atlas was used as an additional reference. Intrarater and interrater reliability was assessed using intraclass correlation coefficients (ICCs) and weighted kappa statistics. ICCs were interpreted according to Koo and Li; weighted kappa statistics were interpreted according to the criteria of Landis and Koch.

Results

The overall intrarater (ICC = 0.88, P < 0.001) as well as the interrater (ICC = 0.84, P < 0.001) reliability of the expert readers was almost perfect. Based on the evaluation sheet of the MOCART 2.0 knee score, the overall interrater reliability of the inexperienced readers was poor (ICC = 0.34, P < 0.019) and improved to moderate (ICC = 0.59, P = 0.001) with the use of the atlas.

Conclusions

The MOCART 2.0 knee score was updated to account for changes in the past decade and demonstrates almost perfect interrater and intrarater reliability in expert readers. In inexperienced readers, use of the atlas may improve interrater reliability and, thus, increase the comparability of results across studies.

Introduction

Cartilage defects of the knee joint are a common orthopedic challenge ¹ and predispose patients to further cartilage loss and the development of osteoarthritis. ² Since the introduction of autologous cartilage implantation, ³ an increasing number of surgical techniques and available scaffolds have been developed. ⁴ Consequently, standardized and reproducible assessment of patient outcome—and of the cartilage repair tissue morphology—became increasingly important for treatment monitoring of individual patients and for comparison between a growing number of different surgical techniques and scaffolds.^1,5-7

Clinical scores, such as the Lysholm Knee Score, ⁸ the IKDC (International Knee Documentation Committee) score, ⁹ or the KOOS (Knee Injury and Osteoarthritis Outcome Score) score, ¹⁰ reflect the individual disease burden and overall joint health. However, these scores lack specificity regarding the quality and state of the repair tissue itself. With improvements in hardware and the development of new sequences, magnetic resonance imaging (MRI) became the method of choice for the morphological assessment of cartilage defects and maturing repair tissue throughout the postoperative period.^6,11-13

However, morphological MRI is based on qualitative assessment and thus suffers from a fundamental lack of standardization and objectivity. To overcome this limitation, the Magnetic Resonance Observation of Cartilage Repair Tissue (MOCART) score was introduced over a decade ago.^14-16 It was based on 9 pertinent variables and facilitated a standardized, reproducible, semiquantitative approach for the morphological assessment of cartilage repair. Since then, the MOCART score has been used in numerous clinical studies as a primary or secondary morphological endpoint.^17-22 Likewise, in daily clinical routine, it forms the basis of standardized reporting about cartilage repair tissue in many centers.

However, since the first introduction of the MOCART score, cartilage repair techniques and MR imaging have undergone significant change. Novel, scaffold-based surgical treatment options^23,24 render the assessment of subchondral bone more important. ²⁵ At the same time, other aspects, such as the assessment of adhesions, which were formerly frequent complications with periosteal flaps, ²⁶ became less relevant due to the introduction of second-generation autologous chondrocyte implantation. Furthermore, the continuous development of MR software and hardware, in particular, better phased array coil designs and increase in available gradient strength as well as further dissemination of MR scanners that operate at high field strengths (3 T), have improved routine clinical MR examinations and MR protocols over the last decade.^27-34 These developments will be addressed with this update to the MOCART score.

Furthermore, we recognized that the linguistically defined categories of the original MOCART score may be interpreted in different ways by different readers, thus introducing variability and decreasing interrater reliability.

Hence, the aim of this study was to develop an incremental update to the original MOCART score for the assessment of cartilage repair of the knee joint, which would account for the above-mentioned advancements and address issues identified in the clinical routine. Intrarater and interrater reliability should be evaluated by expert readers. Furthermore, we aimed to develop an atlas that would depict all variables, using a native MR image next to a color-coded overlay that would emphasize crucial features, and to assess its impact on the interrater reliability of readers with little or no experience.

Materials and Methods

This single-center study was approved by the Ethics Committee of the Medical University of Vienna. Twenty-four patients, who underwent surgical cartilage repair for a femoral cartilage lesion and who received follow-up MR examinations, were randomly selected and retrospectively included in the study. All patients were treated with matrix-associated autologous chondrocyte transplantation (MACT). MACT was performed as a two-stage procedure. In the first procedure, a cartilage biopsy was obtained arthroscopically from a non-weight-bearing area of the knee. After cell extraction, cells were cultivated and subsequently transferred onto a scaffold. For the second procedure, a mini-arthrotomy was used. First, debridement of the cartilage defect to the subchondral bone was performed. Then, the cell matrix implants were cut to size, implanted, and held in place using fibrin glue. ¹⁴

Patients

Twenty-four patients (11 female, 13 male), with a mean age of 34.8 ± 10.9 years at the MRI examination, were retrospectively included in the study. The median postoperative follow-up interval was 2.3 years, ranging from 10 months to 17 years. All patients were treated for a single cartilage lesion. In 10 patients, the left knee was affected, while in 14 patients, the right knee was affected. Sixteen lesions were located at the medial femoral condyle and 8 at the lateral femoral condyle. Twenty patients suffered from ICRS (International Cartilage Repair Society) grade IV lesions, whereas 4 patients had ICRS grade III lesions. Median lesion size was 3.8 cm², ranging from 0.9 cm² to 12 cm². The following scaffolds were used in this study population: Novocart 3D (TETEC AG, Reutlingen, Germany); IGOR.CHONDRO-SYSTEMS (Institute for Tissue and Organ Reconstruction, Wels, Austria); Hyalograft (Fidia Advanced Polymers, Albano Terme, Italy); and CaRes (Arthrokinetics, Esslingen, Germany). In nine patients, additional autologous bone grafting was performed for osteochondral defects. Autograft spongiosa zylinders were harvested from the iliac crest in eight patients and from the proximal tibia in one patient using an OATS harvester and transferred into the recipient socket. ³⁵

MRI Acquisition

All MR imaging studies were performed on 3T MR systems (MAGNETOM Tim Trio, MAGNETOM Verio, MAGNETOM Prisma, Siemens Healthineers, Erlangen, Germany) using a dedicated knee coil. Patients were positioned in the supine position with the knee extended and the joint space in the middle of the coil. The assessed MRI examinations were part of the routine clinical follow-up and not conducted for a prospective study. Thus, sequence parameters varied slightly between patients. Imaging studies for the assessment of knee cartilage and cartilage repair should contain the following sequences: a set of localizers in all three planes; a sagittal non-fat-saturated high-resolution proton-density-weighted turbo spin-echo (sag PDw TSE) sequence; a sagittal fat-saturated (fs) PDw TSE sequence; a sagittal T1-weighted (T1w) TSE; and a coronal fat-saturated PDw TSE sequence. For patients with cartilage repair of the patellofemoral joint, the imaging protocol should be complemented with an axial version of the fs PDw TSE sequence. Whereas the original MOCART score required the additional acquisition of a gradient echo sequence for scoring of the variable signal intensity, this is not required in the MOCART 2.0 knee score. While three-dimensional (3D) gradient echo (GRE) sequences suffer from low sensitivity for intrachondral signal alterations and bone marrow abnormalities with a higher vulnerability to susceptibility artifacts, such as postoperative metal abrasion artifacts, a 3D TSE sequence suffers from long acquisition times and has not been validated for cartilage lesions. ³⁶

The main sequence with the highest sensitivity for intrachondral signal alterations and the structure of repair tissue is a fat-saturated PDw TSE sequence with an echo time (TE) of 40-60 ms, which provides excellent contrast between cartilage and joint fluid and can be performed with a high spatial resolution. ¹² For the accurate evaluation of surface defects of cartilage repair tissue and fissure-like delamination, a sufficiently high spatial resolution with an in-plane resolution of 0.3 mm or less is highly recommended. ¹² Fat-suppressed sequences are necessary for the evaluation of subchondral bone marrow edema-like signal alteration, while non-fat-saturated PDw TSE sequences enable the characterization of intrachondral osteophytes and bony overgrowth in the repair tissue area. The intactness of the subchondral lamina and the visualization of bony defects is also best evaluated on non–fat-saturated sequences and a further differentiation of subchondral bone marrow edema-like signal alterations, such as osteonecrosis-like lesions, is possible with non-fat-saturated sequences.

An exemplary MRI protocol, which contains all sequences that are recommended to enable adequate scoring of the MOCART 2.0 knee score, is provided in Table 1 .

Table 1.

Exemplary MRI Protocol That Fulfills the Recommended Requirements in Terms of Sequences and Resolution for Adequate Assessment of the MOCART 2.0 Knee Score at 3 T.

	Example Parameters for a 3 T Protocol
	Sag PDw TSE	Sag PDw TSE fs	Sag T1w TSE	Cor PDw TSE fs
Coil	8-ch knee	8-ch knee	8-ch knee	8-ch knee
TE (ms)	37	42	12	27
TR (ms) ± deviation	2000 ± 10%	3090 ± 10%	600	4250
Flip angle	90 ± 10%	90 ± 10%	90	180
Fat suppression	No	Yes	No	Yes
FOV (mm)	120	160	150	150
RFOV (%)	100	100	100	100
Acq. matrix	384	384	448	384
Scan (%)	85	100	100	100
Slices	19	25	25	25
Slice thickness (mm)	2	3	3	3
Interslice gap (%)	10	20	20	10
Slice orientation	Sagittal	Sagittal	Sagittal	Coronal
Acquisition time (TA)	03:20	04:06	02:48	03:29

MRI = magnetic resonance imaging; MOCART = Magnetic Resonance Observation of Cartilage Repair Tissue; Sag PDw TSE = sagittal proton density turbo spin-echo; Sag T1w TSE = sagittal T1w TSE; Cor PDw TSE = coronal proton density turbo spin-echo.

Variables of the MOCART 2.0 Knee Score

Seven variables were adopted and modified to add up to a total score ranging from 0 to 100 points. Adhesions, which used to be regularly observed in first-generation autologous chondrocyte implantation, ²⁶ became rare findings with the introduction of second- and third-generation autologous chondrocyte implantation. Thus, the variable “adhesions” of the original MOCART score was removed from the scoring system. In an effort to put more emphasis on the morphological assessment of the repair tissue itself, the variable “effusion” was discarded as well. While an effusion might cause symptoms, it is not necessarily associated with the success of the cartilage repair but may originate from additional pathology. Furthermore, effusion as well as additional pathology of the joint will be captured by CROAKS. ²⁹ All variables are depicted in schematic drawings, as seen in Figures 1 to 7 .

Figure 1.

Volume of cartilage defect filling compared to native cartilage.

Figure 7.

Subchondral changes.

Furthermore, a color-coded atlas containing a native MR image and a processed image with a colored overlay for each value of each variable was developed. In the processed images, the cartilage repair tissue is always marked red, with adjacent healthy cartilage labeled green.

Volume of Cartilage Defect filling

The volume of cartilage defect filling must be assessed in relation to the adjacent native reference cartilage and must be described as a percentage of the hypothetical volume of intact cartilage that covers the defect. Developments in MR hardware, such as dedicated multi-element phased array knee coils and further dissemination of high-field (3 T) scanners, have significantly increased the achievable signal-to-noise-ratio (SNR), which can be invested in increased spatial resolution. ³⁷ Taking advantage of this increased spatial resolution, an additional subdivision in increments of 25% was performed to increase the sensitivity of the MOCART 2.0 knee score to different degrees of defect filling. The filling is considered to be complete (100%) when the repair site is as thick as the surrounding reference cartilage, with a repair tissue volume equivalent to the hypothetical volume of healthy cartilage that covers the defect ( Fig. 1 ). An incomplete repair with inferior cartilage filling compared to adjacent native regions is classified as underfilled and can be classified as “minimal underfilling” (75% to 99%; Fig. 1_2b ), “minor underfilling” (50% to 74%; Fig. 1_3 ), “moderate underfilling” (25% to 49%; Fig. 1_4 ), or “severe underfilling” (<25%; Fig. 1_5a ). Complete delamination in situ ( Fig. 1_5b ) receives the same score as severe underfilling as it bears the risk of dislocation and exposed subchondral bone. Morphologically, delamination is characterized by a complete fluid-like interface that surrounds the repair tissue, which renders healing unlikely. Since Kreuz et al. ³⁸ found that hypertrophy <150% does not negatively affect clinical outcome, hypertrophic filling of <150% will be scored the same as complete filling. Hypertrophy of ≥150% ( Fig. 1_2a ) will be rated with the same score as minimal (75% to 99%) underfilling. Evaluation in at least 2 different sequences and planes is essential to avoid misinterpretations and to avoid underappreciating graft hypertrophy, especially in sequences with fat suppression.

Integration into Adjacent Cartilage

This variable evolved from the variable “integration with border zone” of the original MOCART score. It serves as a measure of the integration of the cartilage repair tissue into the neighboring native cartilage by evaluating the interface between these two tissues. Integration is classified as complete ( Fig. 2_1 ) in cases of an indiscernible interface between the repair tissue and the adjacent cartilage. In case of a split-like demarcation line between the repair tissue and the adjacent cartilage, the width of this defect must be determined. The variable discriminates between split-like defects ≤2 mm ( Fig. 2_2 ), defects >2 mm but <50% of the repair tissue length ( Fig. 2_3 ), and defects ≥50% of the repair tissue length ( Fig. 2_4 ).

Figure 2.

Integration into adjacent cartilage.

Surface of the Repair Tissue

The surface of the repair tissue is classified as “intact” in case of a preserved, congruent articular surface ( Fig. 3_1 ). Irregularities of the articular surface may range from minor fibrillations to fissures and ulcerations. These irregularities are further differentiated based on severity. In the original MOCART score, this discrimination was based on the depth of the surface damages rather than on the surface extension, which overlapped with the assessment of the defect fill. This was amended in the MOCART 2.0 knee score. Now, irregularities of the articular surface need to be assessed with regard to their extent in respect to the total repair tissue diameter and are subdivided into two grades with either an extension over less ( Fig. 3_2 ) or more ( Fig. 3_3 ) than 50% of the repair tissue diameter. It is important to assess the surface of the repair tissue independently of the volume of cartilage defect filling. The surface has to be evaluated with respect to present irregularities, regardless of perfect filling, present hypertrophy, or underfilling. To be able to visualize fine fibrillations and fissures on the surface of the repair tissue, high-resolution MR imaging protocols are essential.

Figure 3.

Surface of the repair tissue.

Structure of the Repair Tissue

This variable was kept unchanged. The structure of the repair tissue is defined as homogeneous when typical cartilage layers are formed over the entire repair tissue or the repair tissue appears homogeneous ( Fig. 4_1 ). It is classified as inhomogeneous ( Fig. 4_2 ) when the tissue appears disorganized with alterations in signal intensity indicating a heterogeneous repair tissue structure.

Figure 4.

Structure of the repair tissue.

Signal Intensity of the Repair Tissue

In the original MOCART score, the variable “signal intensity of the repair tissue” was assessed on a dual T2w TSE and a T1 3D GRE fs sequence separately that were state-of-the-art for cartilage imaging more than a decade ago. Currently, imaging protocols in both clinical routine and studies frequently do not contain both sequences, and if a 3D sequence is part of the imaging protocol, it is mostly a T2w 3D GRE sequence. This has led to the increasing use of “modified” MOCART scores that either omit the assessment of signal intensity based on the 3D GRE sequence with a maximum score of only 85 points ³⁹ or apply a correction factor. ⁴⁰ This, however, reduces comparability across trials.

Hence, in the MOCART 2.0 knee score, signal intensity of the repair tissue is recommended to be assessed on PDw TSE sequences, which offer high sensitivity for the intrachondral structure of cartilage. The signal intensity of repair tissue can be rated as “normal” (isointense to adjacent native cartilage) ( Fig. 5_1 ), “minor abnormal” ( Fig. 5_2a : “minor hyperintense,” and Fig. 5_2b : “minor hypointense”), and “severely abnormal” ( Fig. 5_3a : “almost fluid-like signal,” and Fig. 5_3b : “close to subchondral plate signal”). In contrast to the original MOCART score, signal alterations of the repair tissue can be rated hyper- or hypointense. The signal intensity should be evaluated on all fat-saturated as well as non-fat-saturated PDw TSE sequences. However, the pathology should be present in more than one slice to avoid inaccurate interpretation of a partial volume effect or artifact. In addition, the worst present feature defines the scoring, for example, if the repair tissue shows minor hypointensity and major hyperintensity in different regions, it should receive 0 points. Furthermore, the magic angle effect ⁴¹ 0 has to be considered when evaluating the signal intensity of the repair tissue. Should the repair tissue be located at the anterior or posterior condyles at an angle close to 55° to the B₀, the intensity should be evaluated in reference to healthy cartilage, which is positioned at the same angle to the magnetic field, to avoid false-positive scorings. While a hyperintensity of the repair tissue may represent a higher water content and disorganization of the collagen fiber network, a hypointensity of the repair tissue on the same sequence may result from fibrous tissue formation. Overall, this variable may be an indicator of tissue maturation during the early postoperative phase, up to one year. ⁴⁰

Figure 5.

Signal intensity of the repair tissue.

Bony Defect or Bony Overgrowth

Recently, subchondral bone has become the focus of attention in cartilage repair, since primary damage in osteochondritis dissecans, or secondary damage on the basis of a chondral lesion, is frequent. This is reflected in the development of biphasic scaffolds designed to restore the entire osteochondral unit. ⁴² Thus, assessment of bony defects and subchondral bone is featured more prominently in the MOCART 2.0 knee score. In the original MOCART score, the failure or success of restoration of subchondral bone after treatment of osteochondral lesions was reflected only in the assessment of the variable “subchondral lamina” and “subchondral bone,” which subsumed different pathological changes. Hence, the variable “bony defect or bony overgrowth” was introduced in the MOCART 2.0 knee score. A perfect outcome with intact subchondral bone and no presence of intrachondral osteophytes should be rated as “no bony defect or bony overgrowth” ( Fig. 6_1 ). Bony defects should be subcategorized in defects shallower than the thickness of the adjacent native cartilage ( Fig. 6_2a ) and as deep or deeper ( Fig. 6_3a ) than the thickness of the adjacent native cartilage. Bony overgrowth should be subcategorized as bony overgrowth <50% ( Fig. 6_2b ) and ≥50% ( Fig. 6_3b ) of the thickness of the adjacent native cartilage. The depth of the bony defect or bony overgrowth should always be assessed using the adjacent native cartilage as reference, especially in case of an underfilling of the defect, in which the repair tissue thickness used as a reference might produce a false-positive result.

Figure 6.

Bony defect or bony overgrowth.

Subchondral Changes

In the original MOCART score, this variable differentiated only between the accountable area as being “intact” or as pathologically altered without defining the underlying reason. Due to the importance of the assessment of the subchondral bone and the varying importance of different pathologies, this variable was expanded. In case of an intact subchondral lamina and no additional pathologies, the variable “subchondral changes” is rated “intact” ( Fig. 7_1 ) with 20 points. An edema-like marrow signal can be subdivided into minor, with a maximum diameter less than 50% of the repair tissue diameter ( Fig. 7_2 ) scored with 15 points; or severe, which exceeds 50% of the repair tissue diameter ( Fig. 7_3 ) scored with 10 points. For subchondral cysts with an individual or combined diameter ≥5 mm (i.e., multiple small cysts have a combined diameter ≥5 mm) ( Fig. 7_4a ), or an “osteonecrosis-like signal” ( Fig. 7_4b ), zero points are allocated in this variable. “Osteonecrosis-like signal” may be assessed on T1-weighted images (as depicted in Fig. 7_4b ) or on PD-weighted TSE images with hyperintense bone marrow signal and a hypointense demarcation line. If more than one subcategory of this variable is present in one patient, the subcategory with the less favorable scoring defines the points allocated, that is, if minor edema-like marrow signal and a subchondral cyst are observed, 0 points are selected for this variable.

Image Analysis

Image analysis was performed on a picture archiving and communication system (PACS) workstation (IMPAX EE R20, Agfa Healthcare N.V., Mortsel, Belgium) by four independent readers: one expert reader with more than 25 years of experience in musculoskeletal imaging and extensive familiarity with the original MOCART scoring (reader 1); one expert reader with more than 30 years of experience in musculoskeletal imaging, but no familiarity with the original MOCART scoring (reader 2); and two radiology residents with 1 year of experience in musculoskeletal MR imaging (readers 3 and 4) and no familiarity with the original MOCART score. Imaging studies were assessed under supervision of the study coordinator in random order, and all readers were blinded to all patient details. The expert reader 1 and both inexperienced readers 3 and 4 performed the evaluation twice, separated by a four-week interval to avoid recall bias. The MOCART 2.0 knee score was assessed according to the newly introduced scoring system, which now contains seven variables that add up to a total score between 0 and 100 points ( Table 2 ). The first reading was based on the MOCART 2.0 knee score evaluation sheet only. For the second reading, readers 3 and 4 had access to the newly introduced atlas to support decision making.

Table 2.

MOCART 2.0 Knee Score: Cartilage Repair Tissue Assessment: Grading and Point Scale.

			Scoring
1	Volume fill of cartilage defect
	1	Complete filling OR minor hypertrophy: 100% to 150% filling of total defect volume	20
	2	Major hypertrophy ≥150% (1_2a) OR 75% to 99% filling of total defect volume (1_2b)	15
	3	50% to 74% filling of total defect volume	10
	4	25% to 49% filling of total defect volume	5
	5	<25% filling of total defect volume (1_5a) OR complete delamination in situ (1_5b)	0
2	Integration into adjacent cartilage
	1	Complete integration	15
	2	Split-like defect at repair tissue and native cartilage interface ≤2 mm	10
	3	Defect at repair tissue and native cartilage interface >2 mm, but <0% of repair tissue length	5
	4	Defect at repair tissue and native cartilage interface ≥50% of repair tissue length	0
3	Surface of the repair tissue
	1	Surface intact	10
	2	Surface irregular <50% of repair tissue diameter	5
	3	Surface irregular ≥50% of repair tissue diameter	0
4	Structure of the repair tissue
	1	Homogeneous	10
	2	Inhomogeneous	0
5	Signal intensity of the repair tissue
	1	Normal	15
	2	Minor abnormal—minor hyperintense (5_2a) OR minor hypointense (5_2b)	10
	3	Severely abnormal—almost fluid like (5_3a) OR close to subchondral plate signal (5_3b)	0
6	Bony defect or bony overgrowth
	1	No bony defect or bony overgrowth	10
	2	Bony defect: depth < thickness of adjacent cartilage (6_2a) OR overgrowth <50% of adjacent cartilage (6_2b)	5
	3	Bony defect: depth ≥ thickness of adjacent cartilage (6_2a) OR overgrowth ≥50% of adjacent cartilage (6_2b)	0
7	Subchondral changes
	1	No major subchondral changes	20
	2	Minor edema-like marrow signal—maximum diameter <50% of repair tissue diameter	15
	3	Severe edema-like marrow signal—maximum diameter ≥50% of repair tissue diameter	10
	4	Subchondral cyst ≥5 mm in longest diameter (7_4a) OR osteonecrosis-like signal (7_4b)	0

MOCART = Magnetic Resonance Observation of Cartilage Repair Tissue.

Statistical Analysis

All statistical calculations were performed using IBM SPSS Statistics for Windows version 25 (IBM, Armonk, NY). Metric data are described using mean ± standard deviation. Intraclass correlation coefficients (ICCs; 2-way mixed, absolute agreement) and their 95% confidence intervals (CIs) were calculated as an index of intrarater and interrater reliability for the overall MOCART score. Weighted kappa statistics and their 95% CIs were calculated as an index for intrarater and interrater reliability for categorical MOCART subscales. ICCs were interpreted according to Koo and Li ⁴³ : an ICC of less than 0.5 indicated poor agreement, an ICC of 0.50 to 0.75 moderate agreement, an ICC of 0.75 to 0.90 good agreement, and an ICC of above 0.90 excellent agreement. Weighted kappa statistics were interpreted according to the criteria of Landis and Koch ⁴⁴ : a kappa value of 0.01 to 0.20 indicated poor agreement, a kappa value of 0.21 to 0.40 indicated fair agreement, a kappa value of 0.41 to 0.60 indicated moderate agreement, a kappa value of 0.61 to 0.80 indicated substantial agreement, and a kappa value of 0.81 to 1.00 indicated almost perfect agreement.

Results

MOCART 2.0 Knee Score and Reliability

The overall intrarater (ICC = 0.88, P < 0.001) as well as the interrater (ICC = 0.84, P < 0.001) reliability of the expert readers (reader 1 vs. reader 2) was good ( Table 3 ).

Table 3.

Intrarater Reliability of Expert Reader 1, Interrater Reliability of Expert Readers 1 and 2, as Well as Interrater Reliability of Inexperienced Readers 3 and 4 and versus Reader 1 Given as Weighted Kappa with 95% Confidence Intervals for Categorical Variables and as Intraclass Correlation Coefficient with 95% Confidence Intervals for the Overall MOCART Score.

Intrarater and Interrater Reliability Given as Weighted Kappa for Every Variable and 2-Way Mixed Total Agreement Intraclass Correlation Coefficients (ICCs) Including 95% Confidence Intervals (CIs) and P Value
	Expert Reads
	Intrarater reliability—Expert Reader 1			Interrater reliability—Expert Reader 1 vs. 2
	Weighted Kappa (95% CI)		P Value	Weighted Kappa (95% CI)	P Value
Degree	0.807 (0.618 to 0.997)		<0.001	0.794 (0.628 to 0.960)	<0.001
Integration	0.870 (0.747 to 0.992)		<0.001	0.769 (0.529 to 1.008)	<0.001
Surface	0.571 (0.290 to 0.852)		<0.001	−0.022 (−0.286 to 0.242)	0.873
Structure	0.731 (0.457 to 1.005)		<0.001	1.000 (1.000 to 1.000)	<0.001
Signal	0.808 (0.611 to 1.004)		<0.001	0.574 (0.265 to 0.883)	<0.001
Bony defect/overgrowth	0.812 (0.610 to 1.014)		<0.001	0.724 (0.465 to 0.983)	<0.001
Subchondral changes	0.627 (0.394 to 0.860)		<0.001	0.664 (0.448 to 0.880)	<0.001
	ICC (95% CI)		P Value	ICC (95% CI)	P Value
Total score	0.878 (0.737 to 0.945)		<0.001	0.843 (0.674 to 0.929)	<0.001
	First Reading without Atlas
	Reader 1 vs. Reader 3		Reader 1 vs. Reader 4		Reader 3 vs. Reader 4
	Weighted Kappa (95% CI)	P Value	Weighted Kappa (95% CI)	P Value	Weighted Kappa (95% CI)	P Value
Degree	0.352 (0.076 to 0.628)	0.011	0.365 (0.148 to 0.583)	<0.001	0.388 (0.207 to 0.570)	<0.001
Integration	0.374 (0.057 to 0.692)	0.008	0.500 (0.180 to 0.820)	0.001	0.431 (0.196 to 0.667)	0.002
Surface	0.169 (−0.138 to 0.475)	0.264	0.272 (−0.027 to 0.570)	0.101	0.308 (0.020 to 0.596)	0.046
Structure	0.191 (−0.139 to 0.521)	0.111	0.263 (−0.156 to 0.682)	0.195	−0.077 (−0.212 to 0.059)	0.555
Signal	0.283 (0.014 to 0.552)	0.019	0.114 (−0.137 to 0.365)	0.399	0.129 (−0.036 to 0.295)	0.079
Bony defect/overgrowth	0.639 (0.418 to 0.860)	<0.001	0.633 (0.407 to 0.859)	<0.001	0.337 (0.046 to 0.628)	0.039
Subchondral changes	0.415 (0.190 to 0.640)	<0.001	0.371 (0.143 to 0.598)	<0.001	0.611 (0.334 to 0.888)	<0.001
	ICC (95% CI)	P Value	ICC (95% CI)	P Value	ICC (95% CI)	P Value
Total score	0.435 (−0.046 to 0.736)	0.001	0.681 (0.398 to 0.847)	<0.001	0.336 (−0.033 to 0.637)	0.019
	Second Reading with Atlas
	Reader 1 vs. Reader 3		Reader 1 vs. Reader 4		Reader 3 vs. Reader 4
	Weighted Kappa (95% CI)	P Value	Weighted Kappa (95% CI)	P Value	Weighted Kappa (95% CI)	P Value
Degree	0.476 (0.148 to 0.583)	<0.001	0.642 (0.420 to 0.865)	<0.001	0.562 (0.283 to 0.841)	0.001
Integration	0.566 (0.291 to 0.842)	<0.001	0.592 (0.275 to 0.908)	0.001	0.441 (0,112 to 0,770)	0.441
Surface	0.427 (0.132 to 0.721)	0.005	0.310 (0.005 to 0.615)	0.043	0.428 (0.149 to 0.707)	0.009
Structure	0.654 (0.310 to 0.998)	0.001	0.417 (0.086 to 0.747)	0.025	0.333 (0.052 to 0.614)	0.028
Signal	0.073 (−0.311 to 0.456)	0.623	0.275 (−0.075 to 0.625)	0.051	0.661 (0.341 to 0.981)	<0.001
Bony defect/overgrowth	0.665 (0.396 to 0.933)	<0.001	0.509 (0.204 to 0.814)	0.001	0.491 (0.175 to 0.806)	0.001
Subchondral changes	0.514 (0.297 to 0.732)	<0.001	0.393 (0.150 to 0.635)	<0.001	0.743 (0.544 to 0.942)	<0.001
	ICC (95% CI)	P Value	ICC (95% CI)	P Value	ICC (95% CI)	P Value
Total score	0.666 (0.304 to 0.849)	<0.001	0.646 (0.330 to 0.830)	<0.001	0.573 (0.227 to 0.790)	0.001

In the first reading, readers 3 and 4 performed the scoring based on the scoring sheet; in the second reading, readers 3 and 4 had access to the atlas. Reference of expert reader 1 for interrater reliability was the second read.

Based on the evaluation sheet of the MOCART 2.0 knee score, the overall interrater reliability of the inexperienced readers 3 and 4 was poor (ICC = 0.34, P = 0.019), ranging from poor (“structure”: weighted kappa = −0.08, P = 0.56) to substantial (“subchondral changes”: weighted kappa = 0.61, P < 0.001) for different variables. Using the atlas as an additional reference, the overall interrater reliability of the inexperienced readers 3 and 4 increased to moderate agreement (ICC = 0.57, P = 0.001) in the second reading, ranging from poor (“structure”: weighted kappa = 0.33, P = 0.028) to substantial agreement (“subchondral changes”: weighted kappa = 0.74, P < 0.001), as depicted in Table 2 . The variable “surface of the repair tissue” reached the second-worst interrater reliability in the second read of the inexperienced readers, with a weighted kappa of 0.43, P = 0.009. Similar to the expert readers, disagreement was most often found in this variable between “surface intact” and “surface irregular <50% of the repair tissue” also for the inexperienced readers. This emphasizes the importance of high-resolution imaging, which is pivotal for the correct assessment of cartilage fraying.

Discussion

Since the introduction of the MOCART score nearly 15 years ago, significant progress both in surgical techniques as well as in MR imaging has been made. Since the early alteration in the second MOCART publication, ¹⁴ in which the variable “synovitis” was replaced with the more practical variable “effusion,” the MOCART score has not been revised.

In 2009, Welsch et al. introduced the 3D MOCART score ⁴⁵ in an effort to simplify the imaging protocol by using a single 3D sequence and to improve the scoring system by expanding it to a 3D assessment using multiplanar reformations. In the original paper, the 3D-True-FISP (fast imaging with steady-state free precession) sequence was employed, which was compared to a less artifact-prone PDw SPACE (3D FSE) sequence in a subsequent investigation. ⁴⁶ However, for a number of reasons, most studies still use the original MOCART score: while acquisition of a single 3D sequence does allow a shortening of measurement time, only one MR contrast is acquired, which decreases the amount of available information. In addition, a higher incidence of image artifacts and lower image quality, as well as lower cartilage contrast, were observed for the isotropic 3D-True-FISP and 3D PDw SPACE sequences, when compared with the standard 2D TSE images. ⁴⁶ Furthermore, many studies are still conducted on 1.5 T systems, on which the acquisition of isotropic 3D sequences with sufficient spatial resolution in a feasible measurement time is particularly challenging and has been shown to be inferior to typical PD-weighted 2D TSE sequences. ⁴⁷ Based on these considerations, the 2D imaging protocol of the original MOCART score was maintained for the MOCART 2.0 knee score.

Furthermore, the maximum total score was kept at 100 points. The subdivision of the variable “volume of cartilage defect filling” in steps of 25% was motivated by the gain in signal-to-noise ratio (SNR) due to technical advancements over the past decade, which allowed for an increase in spatial resolution and should facilitate more accurate assessment. One key change to the variable defect fill is that the volume of the repair tissue rather than the thickness should be quantified and described as a percentage of the hypothetical volume of intact cartilage that covers the defect. This is particularly helpful in the assessment of cases in which the repair tissue exhibits areas of different thickness. The original MOCART score tended to underscore these cases, when using the worst area as a reference for the scoring. The variable “integration into adjacent cartilage” now assesses only the integration to adjacent cartilage rather than to subchondral bone as well. This change was introduced to eliminate overlap with the variables “bony defect or bony overgrowth” and “subchondral changes.” The variable “structure of the repair tissue” was kept unchanged; however, assessment is now recommended on PDw TSE sequences. Signal intensity is now graded as “normal,” “minor,” or “severely abnormal” for both hyperintense as well as hypointense signal alterations. This change was made to account for the fact that many routine protocols currently lack a 3D GRE sequence. This led to the increasing use of “modified” MOCART scores that either omitted the assessment of signal intensity based on the 3D GRE sequence and could be summed to only 85 points ³⁹ or used a correction factor ⁴⁰ that reduced comparability across trials. It is recommended that signal intensity is assessed on a PDw TSE sequence with a TE of less than 60 ms in the MOCART 2.0 knee score and not on a T2 TSE sequence, which would overestimate hyperintensity and underestimate hypointensity. However, it is key that TEs are kept the same between subjects within a study and also across follow-ups. Variance in TE influences the rating and thus jeopardizes comparability across studies. The variable “bony defect or bony overgrowth” was introduced to better depict subchondral bone, as well as bony overgrowth and intrachondral osteophytes, which may occur in particular after microfracturing. ⁴⁸ The variable “subchondral changes,” which emerged from the variable “subchondral bone” of the original MOCART, is of particular importance as it now allows for the discrimination between minor and severe edema-like marrow signal.

The evaluations of expert readers 1 and 2 revealed significant overall intrarater and interrater reliability. In addition, reader 1 achieved substantial and almost perfect intrarater agreement for all independent variables. What is noteworthy, is the low interrater (weighted kappa = −0.02, P = 0.87) reliability also between the expert readers with regard to the variable “surface of the repair tissue.” Neither of the expert readers used the atlas for additional reference. Whereas reader 1 had extensive experience with the original MOCART score since its introduction, reader 2 had 30 years of experience in musculoskeletal MRI, but only limited experience with the original MOCART score. Hence, reader 1 was confident to make extreme judgements (in 10 cases the surface was assessed as intact), whereas a tendency to the mean was observed for reader 2 with not a single rating for “surface intact” but 21 ratings for “surface irregularities <50% of the repair tissue length.” One could speculate that less familiar readers with this scoring system, when in doubt, tend to choose the less extreme option, explaining the significant difference.

In the literature, assessment of the MOCART score is frequently performed exclusively by expert readers, and high values of interrater reliability have been reported. However, in the clinical routine and in clinical trials, assessments are often performed by less experienced readers, which might negatively affect reproducibility and comparability across studies. The interrater reliability of the inexperienced readers improved with access to the atlas for all variables. Thus, we hypothesize that, especially for less experienced readers, additional access to a visual atlas might improve reliability. Furthermore, many other grading systems, such as the RAMRIS ⁴⁹ and the OARSI OA, ⁵⁰ have published atlases and clinical trials typically begin with reader training.

Studies have suggested that the MOCART score correlates with morphological and clinical changes alike, although this correlation is missing in some reported collectives.^19,51 In a systematic review and meta-analysis, 22% of identified studies reported a clinical correlation with defect fill. ⁵² In addition, correlation with quantitative MRI, namely, T₂ mapping, has been reported.^53,54 One explanation for inconsistent findings regarding the correlation with clinical scores might be that the cartilage lesion itself or the state of the repair tissue does not directly cause pain, but rather, neighboring structures are affected by the subsequent joint degeneration. Thus, it was not the aim of this study to change the variables or point-scale in a way that would entail maximum correlation with clinical outcome, but rather to modify the score in such a way as to take into account the changes over the last decade in surgical treatment and MRI method development.

This study has some limitations that need to be addressed. First, this was not a prospective but rather a retrospective study. Hence, patients were scanned on different 3 T systems and sequence parameters were not identical for all patients. However, all MRI protocols used comparable sequences, as previously described, and this variability reflects the actuality in the clinical routine. Furthermore, only 24 patients were included in this study. Still, the study cohort was large enough to observe a difference in ICC between conventional evaluation and evaluation with the support of the newly introduced atlas. Also, the increase in ICC between the conventional evaluation and evaluation with support of the atlas could be due to a “learning curve” effect. However, inexperienced readers 3 and 4 did not receive any feedback or additional training regarding their first readings, which renders it unlikely that the observed increase in reliability is attributed to a learning curve. Another limitation is that all of the study cases were treated with MACT, although with different scaffolds and additional bone grafting in a few cases. Hence the validation of the new MOCART 2.0 knee score with different surgical techniques is warranted in subsequent studies.

Conclusion

The modification of key variables and omission of the assessment of the variables “adhesions” and “effusion” in the MOCART 2.0 knee score improves the scoring system to be more sensitive to important factors in cartilage repair morphology. Most importantly, this study demonstrates almost perfect intrarater and interrater reliability for expert readers and the importance of a visual reference for correct assessment, as evidenced by the increase in interrater reliability for inexperienced readers when using the established atlas as an additional reference. The use of the supplemented atlas as an additional reference for decision making may increase interrater reliability and, thus, the comparability of reported results across studies, especially for less experienced readers.