Inter-rater agreement of rotator cuff tendon and muscle MRI parameters evaluated preoperatively and during the first postoperative year following rotator cuff repair

Abstract

Background:

MRI is standard-of-care for rotator cuff evaluation, with clinical interpretation usually limited to qualitative judgments. The reliability of MRI–based measurements and scoring systems have been evaluated only preoperatively or ≥6 months following rotator cuff repair (RCR) when repairs are in the later stages of healing. This study describes the MRI assessments and inter-rater agreement of various rotator cuff tendon and muscle parameters evaluated preoperatively and four times during the first postoperative year.

Methods:

Two musculoskeletal radiologists independently assessed MRI scans of 42 patients preoperatively and 3, 12, 26 and 52 weeks after RCR. Using standardized reading rules, readers assessed tendon integrity (5-point Sugaya), tear dimensions, muscle fat (5-point Goutallier) and atrophy (4-point Warner), muscle cross-sectional areas and myotendinous junction (MTJ) distance. Raw exact agreement proportions, kappa statistics, and correlation coefficients were used to quantify inter-rater agreement.

Results:

Readers showed moderate to substantial above-chance agreement in scoring RC tendon integrity and supraspinatus muscle atrophy, good to excellent agreement on tear dimensions and muscle cross-sectional areas, but only fair to moderate agreement for fatty infiltration and MTJ distance. Only fatty infiltration grades evidenced observer bias. Inter-rater agreement did not appear time dependent.

Conclusion:

Using defined reading rules in a research setting, MRI evaluations of RC tendon integrity, tear dimensions, muscle atrophy and cross-sectional areas have reasonable reliability at all time points in the first postoperative year. However, the presence of clinically significant disagreements, even in such favorable circumstances, indicates need for improved imaging tools for precise rotator cuff evaluation.

Level of evidence:

Level IV; Diagnostic Study

Full-thickness rotator cuff tears are present in at least 25% of individuals in their 60s and 50% of individuals in their 80s⁴⁸ and can be associated with pain, weakness and loss of shoulder function⁹. At least 250,000 rotator cuff repairs (RCRs) are performed annually in the United States⁴⁴. Despite decades of advances in surgical techniques and rehabilitation strategies, re-tear rates following RCR are reported to be at least 20-30%¹⁷.

Shoulder magnetic resonance imaging (MRI) is routinely used for both diagnosing rotator cuff tears and assessing repair integrity after surgery^{20; 24; 43}. MRI prior to surgery is considered the standard of care modality for rotator cuff imaging with excellent sensitivity and specificity for full-thickness cuff tear diagnosis¹³. However, clinical interpretation is usually limited to the presence/absence of a tear, tendinosis and qualitative judgment of muscle atrophy and fatty infiltration. MRI cannot accurately discern the type or quality of the tendinous tissues³⁶.

Re-tears most frequently occur within the first 6 postoperative months^{20; 31}, when the MRI appearance of the healing RCR is considerably variable¹¹, making image interpretation challenging. In attempts to standardize MR assessments of the rotator cuff, particularly to enable research aimed to improve diagnosis and treatment of rotator cuff pathology, some groups have employed a comprehensive set of parameters describing rotator cuff pathology, including RCR status (re-tear vs. intact, tear location, tendon thickness, and tendon retraction), rotator cuff muscle atrophy and fatty infiltration, and humeral head edema-like marrow signal^{15; 42; 46}. Fair to excellent inter-rater agreement has been reported for these assessments ^{18; 21; 23; 28; 41; 49}, however, most of these studies evaluated shoulder MRIs obtained preoperatively or ≥6 months postoperatively when repairs are in the later stages of healing. No study has prospectively assessed inter-rater agreement on longitudinal MRIs obtained preoperatively and at multiple intervals over the first year following RCR when the rotator cuff tissues are healing and the MRI signal is evolving.

The objective of this study was to describe the MRI assessments of various rotator cuff tendon and muscle parameters over the first year following RCR and measure inter-rater agreement between two fellowship-trained musculoskeletal radiologists preoperatively and at several postoperative intervals. We hypothesized that inter-rater agreement preoperatively and 12 months postoperatively would be similar and greater than at intermediate postoperative time points (3 weeks, 12 weeks and 6 months) when ongoing tissue healing may introduce more ambiguity in interpretation of the MRI appearance.

METHODS

Study Design

This is a cross-sectional study of observer agreement in interpretation of MRI scans before and after rotator cuff repair surgery. Each of 42 patients prospectively enrolled in an IRB approved natural history study of RCR [https://clinicaltrials.gov/ct2/show/NCT02716441] had 5 shoulder MRIs obtained preoperatively within 3 months of surgery and postoperatively at 3 weeks, and 3, 6 and 12 months after surgery. All patients underwent primary arthroscopic RCR of their supraspinatus and/or infraspinatus tendons by one of four shoulder surgeons at our institution between 2016 and 2018. All tears were fully repaired by a double row or double-row equivalent technique using non-metallic anchors (type and manufacturer at the discretion of the surgeon). No patient had a re-operation within the 12 month imaging follow-up interval. The 210 shoulder MRIs were de-identified with respect to patient identity and time relative to surgery and reviewed independently by 2 fellowship-trained musculoskeletal radiologists, with 30 and 16 years of experience, who were also blinded to clinical and demographic patient information. Imaging studies were reviewed individually in random order with patients and time points intermingled on AquariusNET viewer software v4.4 (TeraRecon, Inc., Durham, NC, USA). All area and distance measurements were performed using AquariusNET software and manually recorded in a standardized REDCap^® database. Reading rules for all assessments were prospectively defined and refined on a separate image set by the two readers prior to the study, and the rule book was available to these readers during independent scoring of each MRI.

Detailed Methods

Magnetic Resonance Imaging

Two hundred ten pre- and postoperative MR studies were evaluated, all obtained using a dedicated shoulder coil. One hundred ninety (90.5%) were obtained at our institution on a 1.5T system (Aera, Symphony, Avanto, or Espree, Siemens Medical Solutions, Malvern, PA, USA). Twenty pre-existing preoperative shoulder MRIs from outside centers (13 performed at 1.5T, 7 at 3T) were accepted into the study after screening for adequate quality and completeness. Patients were imaged lying supine with their humerus in neutral rotation, i.e. elbow extended, thumb up. All MRIs included oblique coronal, and sagittal intermediate weighted (IW, TE 36-38ms, TR 2340-3400ms) images with fat suppression, oblique coronal T2-weighted images without fat suppression (T2W, TE 60-65ms, TR 2040-3200ms), and sagittal T1-weighted (T1W, TE 10-14ms, TR 526-907ms) images without fat suppression. Field of view was 12 cm with 3-4 mm slice thickness, interslice gap of 15-20%, and in-plane resolution of 0.23 x 0.23 mm. Sagittal images were prescribed to include at least one image medial to the spinoglenoid notch for muscle grading. Eleven (5%) studies lacked a coronal non-fat suppressed T2W sequence, 4 (2%) had sagittal T1W sequences that extended only to the spinoglenoid notch but not medial and 1 (0.5%) did not include the inferior margin of the infraspinatus muscle. However, in each instance, all were considered adequate for analysis with no patients requiring repeat imaging.

MRI Evaluation

Tear and re-tear classification

Preoperative and postoperative rotator cuff integrity was assessed from coronal T2W non-fat suppressed and sagittal IW fat-suppressed images using the Sugaya classification⁴²: 1, sufficient thickness with homogenously low intensity; 2, sufficient thickness with partial high intensity area; 3, less than half normal thickness but no discontinuity; 4, minor discontinuity on only 1 or 2 slices on both oblique coronal and sagittal images; 5, major discontinuity on >2 slices on both oblique coronal and sagittal images. Using this scoring classification, Sugaya 1 and 2 imply absence of a tear, Sugaya 3 a partial thickness delaminated tear, Sugaya 4 a small full thickness tear and Sugaya 5 a medium-large full-thickness tear. The radiologists’ Sugaya grades (Read 1) were compared for clinically significant discrepancies (i.e., 1 or 2 vs. 3, 4 or 5; 3 vs. 4 or 5). Scans with such discrepancies were re-examined jointly by the readers without knowledge of their prior grades and assigned a consensus grade (Consensus Read). The median time interval between Read 1 and Consensus Read was 20 days (range, 5-239) for one reader and 54 days (range, 4-364) for the other, with respectively 65% and 81% over 10 days between reads.

Tear and re-tear dimensions

Tear dimensions were measured independently by both readers for studies with a Sugaya grade of 4 or 5 either before (Read 1) or after consensus (Read 2). Anterior-posterior (AP) tear width was determined as the distance between the anterior and posterior tear margins at the midpoint of tendon thickness. AP tear width was measured on the sagittal slice at the midportion of the footprint’s medial-lateral (ML) dimension when the tear was at the footprint. For tears medial to the footprint, the most lateral sagittal slice demonstrating the tear was selected for AP tear width measurement. ML tear length was determined as the distance from the lateral edge of the footprint to the lateral edge of the tendon stump at the midpoint of the tendon thickness using the coronal slice where ML tear length appeared maximal.

Muscle atrophy and fatty infiltration classifications

Supraspinatus atrophy and supraspinatus and infraspinatus fatty infiltration were evaluated on the oblique sagittal T1W non-fat suppressed image immediately medial to the spinoglenoid notch. Supraspinatus atrophy was graded by the Warner classification with aid of a line drawn between the scapular spine and coracoid process as 0, none-muscle convex above the line; 1, mild–muscle even with the line; 2, moderate–concave below the line ≤ 50% of the height of the supraspinatus fossa; and 3, severe -concave below the line >50% of the fossa height⁴⁶. Fatty infiltration of only the intra-muscular regions³⁸ of the supraspinatus and infraspinatus muscles was graded by the Goutallier classification as 0, none to <10%; 1, 10 to <25%; 2, 25 to <50%; 3, 50%; and 4, >50%¹⁴.

Cross sectional areas (CSAs) of muscles

Each radiologist manually outlined the supraspinatus and infraspinatus muscles on the sagittal T1W image from which muscle atrophy and fatty infiltration were evaluated. CSA of the outlined region was then automatically calculated by the AquariusNET software. Perimuscular fat accumulation was excluded. Since teres minor atrophy can occur without rotator cuff pathology ^{2; 22; 30; 39}, this muscle was excluded from measurement. The boundary between the infraspinatus and teres minor was defined by best judgment since the interface of these muscles was often indistinguishable.

Supraspinatus myotendinous junction (MTJ) distance

The maximal distance between the lateral edge of the greater tuberosity and the supraspinatus MTJ, defined as the medial- most point on the tendon at which no muscle was present, was measured using coronal T2W non-fat suppressed and fat suppressed images as the “MTJ distance.”

Statistical Analysis

Inter-rater Sugaya grade agreement was assessed using Read 1 data, first including all 5 categories of the Sugaya classification, secondly by differentiating intact (Sugaya 1-2) versus partial tear or re-tear (Sugaya 3) versus full thickness tear or re-tear (Sugaya 4-5) on a trichotomous scale, and thirdly by dichotomously differentiating only intact (Sugaya 1-2) from torn or re-tom (Sugaya 3-5) tendon. Tear dimensions from Read 1 where present, and Read 2 otherwise, were used to assess inter-rater agreement of length measurements. Ordinal categorizations (Sugaya grade, muscle atrophy and muscle fatty infiltration) were depicted graphically at each examination and overall using agreement charts ⁵. For each variable the raw exact agreement proportion, defined as the fraction of examinations classified identically by the two radiologists, and two chance-corrected observer agreement measures, the simple (Cohen) and quadratically-weighted (Fleiss-Cohen) kappa statistics, were used to quantify the inter-rater agreement for scans at each time interval. Simple kappa statistics penalize all disagreements equally, whereas quadratically-weighted kappa statistics penalize 2- and 3-category disagreements respectively 4 and 9 times as much as adjacent-category disagreements ¹⁰. Scatterplots, concordance correlation coefficients (CCCs) and intraclass correlation coefficients (ICCs), the latter defined as the ratio of between patient to the sum of between and within patient variance components, were used to assess inter-rater agreement for the continuous measurements of tear dimensions, CSAs of muscles and supraspinatus fossa, and supraspinatus MTJ distance. The former accounts for observer bias and is thus more appropriate for our purpose, but we also report the latter for interpretation of our results in the context of other papers where the latter may have been used. Bland-Altman plots were used to qualitatively assess dependence of inter-rater differences on the magnitudes of measurements.

To summarize observer agreement overall across time points, kappa statistics were calculated from the pooled classifications across all examination times. Associated confidence intervals, accounting for any statistical dependence across times, were obtained from 5000 bootstrap resamples of patients, with all of each patient’s examinations included or excluded simultaneously. CCCs summarizing all examinations were calculated as ratios of restricted maximum likelihood (REML) estimates of summed variance components from Gaussian mixed linear regression models including fixed effects of time, reader and their interaction, and random effects of patients, patient×reader, and patient×time interactions. Confidence intervals for these overall concordance correlations were obtained by applying the delta method to their inverse hyperbolic tangents and back-transforming^{6; 25}. ICCs were analogously calculated from main effect Gaussian mixed models with fixed time and random reader effects, with confidence intervals obtained by the delta method without transformation^{16; 19}.

Kappa statistics were described by ranges as almost perfect, 0.81-1.00; substantial, 0.61-0.80; moderate, 0.41-0.60; fair, 0.21-0.40; slight, 0.00-0.20; and poor, < 0.00²⁶, and CCCs and ICCs by ranges as excellent 0.75-1.00; good 0.60-0.74; fair 0.40-0.59; and poor < 0.40⁷.

Results

Patient demographics and tear characteristics

Demographics and operative findings of the 42 patients are summarized in Table 1. Intra-operatively, four of 42 (9.5%) tears were high grade partial thickness and subsequently converted to full thickness and repaired. Tear dimensions averaged 2.2±1.2 (range, 0.5-5.0) cm AP and 1.2±0.6 (range, 0.5-3.0) cm ML based on intra-operative measurement.

Table 1.

Baseline demographics and tear characteristics of patients who underwent RCR. Statistics are presented as percentages or mean ± SD (range).

Characteristics		N	Statistics
Age (years)		42	59.2±7.7 (40-71)
Sex	Male	21	50%
	Female	21	50%
Body Mass Index		42	30±6 (17-46)
Race	White	37	88%
	Black or African American	5	12%
Laterality	Right	29	69%
	Left	13	31%
Tear location	Supraspinatus only	27	64%
	Supraspinatus and infraspinatus	15	36%
Intra-operative tear dimensions (cm)	AP tear width	42	2.2±1.2 (0.5-5.0)
	ML tear length	42	1.2±0.6 (0.5-3.0)

Tear and repair classification

Exact agreement on Read 1 full Sugaya classifications was 0.77 overall and ranged from 0.71 to 0.83 at individual times, and simple κ coefficients showed moderate to substantial above-chance agreement, with values ranging from 0.53-0.64 at individual time points and 0.67 overall (Table 2). Weighted κ coefficients were moderate preoperatively (0.59) but almost perfect at each postoperative time point (0.85-0.92) and overall (0.91). Evidence of observer bias, i.e. a tendency for one reader to consistently use a lower or higher Sugaya grade than the other as indicated by asymmetries of the blocks on the agreement plots around the diagonal line, was minimal (Supplementary Figure S1).

Table 2.

Agreement statistics of MRI parameters from all time points of acquisition and overall. Agreement statistics for AP and ML tear dimensions were calculated from a combination of Read 1 and Read 2 measurements and included N=39, 5, 7, 10, 10 and 71 measurements for pre-op, 3 weeks, 12 weeks, 26 weeks, 52 weeks and overall respectively. Agreement statistics for all other parameters were calculated from Read 1 assessments and included N=42 for all time points and N=210 overall. Exact agreement and simple and weighted kappa (κ) coefficients where appropriate are provided for categorical variables. Concordance correlation coefficients (CCC) are provided for continuous variables. Cross-sectional area (CSA), myotendinous junction (MTJ).

	Statistic	Pre-op (95% CI)	3 weeks (95% CI)	12 weeks (95% CI)	26 weeks (95% CI)	52 weeks (95% CI)	Overall (95% CI)
Sugaya classification (full, 5-point)	Exact agreement	0.83	0.83	0.71	0.76	0.71	0.77
	Simple κ	0.56 (0.31, 0.81)	0.64 (0.41, 0.87)	0.53 (0.31, 0.74)	0.63 (0.44, 0.83)	0.55 (0.35, 0.76)	0.67 (0.58, 0.74)
	Weighted κ	0.59 (0.33, 0.85)	0.92 (0.84, 1.00)	0.87 (0.79, 0.95)	0.86 (0.75, 0.98)	0.85 (0.74, 0.96)	0.91 (0.88, 0.94)
Sugaya classification (trichotomized)	Exact agreement	0.93	0.86	0.81	0.79	0.74	0.82
	Simple κ	0.38 (−0.15, 0.91)	0.66 (0.45, 0.87)	0.66 (0.46, 0.86)	0.58 (0.31, 0.86)	0.58 (0.37, 0.79)	0.72 (0.63, 0.79)
	Weighted κ	0.23 (−0.20, 0.65)	0.84 (0.72, 0.96)	0.77 (0.59, 0.95)	0.85 (0.72, 0.99)	0.75 (0.57, 0.92)	0.86 (0.8, 0.91)
Sugaya classification (dichotomized)	Exact agreement	0.93	0.98	0.98	0.95	0.88	0.94
	Simple κ	0.38 (−0.16, 0.91)	0.92 (0.76, 1.00)	0.86 (0.67, 1.00)	0.90 (0.69, 1.00)	0.66 (0.39, 0.93)	0.87 (0.79,0.93)
Atrophy of supraspinatus	Exact agreement	0.95	0.9	0.95	0.95	0.9	0.93
	Simple κ	0.80 (0.55, 1.00)	0.29 (−0.21, 0.79)	0.77 (0.48, 1.00)	0.83 (0.61, 1.00)	0.72 (0.48, 0.97)	0.73 (0.88, 0.94)
	Weighted κ	0.80 (0.51, 1.00)	0.29 (−0.21, 0.79)	0.77 (0.48, 1.00)	0.93 (0.84, 1.00)	0.90 (0.78, 1.00)	0.85 (0.7, 0.92)
Fatty infiltration supraspinatus	Exact agreement	0.86	0.83	0.74	0.74	0.67	0.77
	Simple κ	0.34 (−0.05, 0.73)	0.45 (0.12, 0.77)	0.43 (0.19, 0.68)	0.41 (0.16, 0.67)	0.37 (0.13, 0.61)	0.42 (0.3, 0.54)
	Weighted κ	0.24 (−0.12, 0.60)	0.45 (0.12, 0.77)	0.38 (0.16, 0.61)	0.56 (0.28, 0.83)	0.70 (0.45, 0.94)	0.55 (0.39, 0.72)
Fatty infiltration infraspinatus	Exact agreement	0.71	0.69	0.69	0.69	0.64	0.69
	Simple κ	0.45 (0.22, 0.69)	0.46 (0.23, 0.68)	0.48 (0.26, 0.71)	0.49 (0.26, 0.73)	0.44 (0.24, 0.65)	0.48 (0.38, 0.58)
	Weighted κ	0.74 (0.60, 0.89)	0.75 (0.61, 0.89)	0.75 (0.55, 0.94)	0.77 (0.59, 0.94)	0.80 (0.67, 0.94)	0.77 (0.69, 0.84)
AP tear width	CCC	0.87 (0.77, 0.93)	0.96 (0.70, 1.00)	0.61 (−0.10, 0.91)	0.93 (0.76, 0.98)	0.89 (0.63, 0.97)	0.86 (0.77, 0.91)1
ML tear length	CCC	0.86 (0.75, 0.92)	0.88 (0.35, 0.98)	0.81 (0.31, 0.96)	0.76 (0.48, 0.90)	0.72 (0.29, 0.91)	0.82 (0.71, 0.89)1
Supraspinatus CSA	CCC	0.98 (0.96, 0.99)	0.97 (0.95, 0.99)	0.98 (0.96, 0.99)	0.98 (0.97, 0.99)	0.97 (0.95, 0.99)	0.98 (0.97, 0.98)2
Infraspinatus CSA	CCC	0.77 (0.61, 0.87)	0.68 (0.48, 0.81)	0.89 (0.80, 0.94)	0.84 (0.73, 0.91)	0.92 (0.85, 0.95)	0.82 (0.75, 0.88)1
Supraspinatus MTJ distance	CCC	0.65 (0.46, 0.79)	0.68 (0.50, 0.80)	0.48 (0.28, 0.64)	0.53 (0.33, 0.68)	0.65 (0.49, 0.77)	0.59 (0.49, 0.68)1

¹Based on a Gaussian mixed linear model with fixed effects of time, reader and their interaction, independent patient, patient × reader, and patient × time random effects and residuals.

²Based on a main effect Gaussian mixed linear model as above, but with first-order autoregressive (AR(1)) covariance structure.

Trichotomizing (1-2 vs. 3 vs. 4-5) and dichotomizing (1-2 vs. 3-5) the Sugaya scale both significantly increased the fractions of scans with exact agreement between the readers at all times (0.74-0.93 and 0.88-0.98, respectively) and overall (0.82 and 0.94, respectively) (Table 2). Trichotomizing the Sugaya scale did not substantially improve simple or weighted κ coefficients from those for the full scale, whereas dichotomization improved simple κ coefficients for postoperative scans compared to both the full and trichotomized scales.

Clinically significant inter-rater Sugaya disagreements occurred for 37 of 210 scans (18%) and were found at all time points but more frequently as time increased after surgery (Table 3). Of these, 33 of 37 (89.2%) were adjacent category disagreements. Following Consensus Read to resolve clinically significant disagreements, 39 of 42 preoperative tears (93%) were classified as full-thickness (Sugaya 4-5) and 3 (7%) as partial-thickness (Sugaya 3), while 10 (24%) repairs were classified by one year postoperative scan as full-thickness re-tears and 8 (19%) repairs as partial-thickness re-tears (Table 4). After consensus, six (16.2%) tears or re-tears required second individual reads for tear dimensions (Read 2, Table 3).

Table 3.

Summary of the number of MRI scans assessed at different time points for Read 1, Consensus Read and Read 2.

	Pre-op	3w	12w	26w	52 w	Total
Read 1	42	42	42	42	42	210
Consensus Read (%)	3 (7.1%)	6 (14.3%)	8 (19.0%)	9 (21.4%)	11 (26.3%)	37 (17.6%)
Disagree between 2 & 3	0	5	7	7	6	25
Disagree between 3 & 4	1	1	1	1	4	8
Disagree between 2 & 4	1	0	0	1	1	3
Disagree between 3 & 5	1	0	0	0	0	1
Read 2 *	1	0	0	2	3	6

^*MRI scans receiving a Sugaya grade of 4 or 5 on Consensus Read were re-read (Read 2) by both readers independently for tear dimension measurements.

Table 4.

Percent of cases assigned to each Sugaya grade after Consensus Read (average percentage of 42 reads from two readers)

		Pre-op	3 weeks	12 weeks	26 weeks	52 weeks
Sugaya Score	1	0%	8.3%	2.4%	0%	0%
	2	0%	77.4%	57.1%	50%	57.1%
	3	7.1%	2.4%	23.8%	26.2%	19.1%
	4	16.7%	2.4%	4.8%	8.3%	8.3%
	5	76.2%	9.5%	11.9%	15.5%	15.5%

Tear and re-tear dimensions

The average dimensions of all tears measured at each examination time are shown in Table 5. Reader agreement for measuring tear dimensions was excellent overall (CCC=0.86 for AP width and 0.82 for ML length) and preoperatively (0.87 and 0.86, respectively) and good to excellent postoperatively, ranging from 0.61-0.96 for AP width and 0.72-0.88 for ML length (Table 2). Analogous results for the ICC were qualitatively, and in most instances quantitatively, very similar (Supplementary Table S1). Between-reader correlation plots for AP and ML tear dimensions showed that 9 of 71 (13%) and 7 of 71 (10%) measures of AP tear width and ML tear length, respectively, differed by ≥10mm (considered by the authors to be clinically relevant) between readers (Supplementary Figures S2 and S3). Bland-Altman plots did not show correlation between inter-rater differences and tear size tear size (not shown).

Table 5.

Descriptive statistics for AP and ML tear/re-tear dimensions were calculated from the average of Read 1 and Read 2 measurements from both readers at each time point; mean ± SD (range).

	Pre-op (n=39)	3 weeks (n=5)	12 weeks (n=7)	26 weeks (n=10)	52 weeks (n=10)
AP tear width (cm)	2.4±1.1 (0.7-5.5)	3.7±1.5 (1.3-5.0)	2.6±1.2 (1.2-4.3)	2.2±1.6 (0.4-4.5)	2.3±1.6 (0.6-5.0)
ML tear length (cm)	2.6±1.1 (0.8-5.7)	3.4±1.4 (1.2-4.8)	4.3±1.1 (2.7-5.5)	3.8±7.9 (2.7-4.8)	4.1±1.1 (2.3-5.5)

Muscle atrophy and fatty infiltration classifications

Readers classified supraspinatus muscle atrophy as grade 0 or 1 (“none” or “minimal”) in > 90% of patients preoperatively, and the distribution of atrophy classifications did not shift substantially over one-year postoperatively (Table 6). Exact agreement on supraspinatus atrophy classification was 0.9-0.95 at individual times and 0.93 overall, and simple κ coefficients showed substantial above-chance agreement, with values ranging from 0.72-0.80 at preop, 12 weeks, 26 weeks and 52 weeks after surgery and 0.73 overall although only fair, 0.29, at 3 weeks postoperatively (Table 2). Quadratic weighting did not improve the coefficients substantially. Agreement plots showed minimal evidence of observer bias (Supplementary Figure S4).

Table 6.

Percent of cases assigned to each supraspinatus atrophy, supraspinatus fatty infiltration and infraspinatus fatty infiltration grade (average percentage of 42 reads from two readers)

Variables	Score	Pre-op	3 weeks	12 weeks	26 weeks	52 weeks
Supraspinatus Atrophy	0	86.9%	92.9%	88.1%	84.5%	79.8%
	1	6.0%	7.1%	11.9%	7.1%	11.9%
	2	7.1%	0%	0%	8.3%	6.0%
	3	0%	0%	0%	0%	2.4%
Supraspinatus Fatty Infiltration	0	88.1%	82.1%	69.0%	71.4%	61.9%
	1	10.7%	17.9%	28.6%	25.0%	32.1%
	2	1.2%	0%	1.2%	3.6%	3.6%
	3	0%	0%	1.2%	0%	1.2%
	4	0%	0%	0%	0%	1.2%
Infraspinatus Fatty Infiltration	0	64.3%	58.3%	45.2%	46.4%	41.7%
	1	22.6%	29.8%	45.2%	40.5%	44.0%
	2	11.9%	8.3%	6.0%	9.5%	8.3%
	3	1.2%	1.2%	0%	0%	2.4%
	4	0%	2.4%	3.6%	3.6%	3.6%

Fatty infiltration of the infraspinatus was graded higher than the supraspinatus and the distribution in both muscles appeared to shift modestly towards increasing severity over the first 12 weeks postoperatively and remain constant thereafter (Table 6). Exact agreement on fatty infiltration ranged from 0.67-0.86 and 0.77 overall for supraspinatus, and from 0.64-0.71 and 0.69 overall for infraspinatus (Table 2). Simple κ coefficients showed fair to moderate above-chance agreement on fatty infiltration for supraspinatus (0.34-0.45, overall 0.42) and moderate for infraspinatus (0.44-0.49, overall 0.48) (Table 2). Quadratic weighting substantially improved the κ coefficients for infraspinatus muscle fatty infiltration but for supraspinatus only at 52 weeks (Table 2). Observer bias and (predominantly adjacent category) disagreement were observed for fatty infiltration classifications (Supplementary Figures S5 and S6).

Cross sectional areas (CSAs) of muscles

The average CSAs of supraspinatus and infraspinatus muscles at each examination time are shown in Table 7. The magnitudes and ranges of measurements were similar across time for each variable. Reader agreement for CSA measurements was excellent at each examination time and overall for supraspinatus muscle (CCC, 0.97-0.98). Agreement for infraspinatus muscle CSA was lower than for supraspinatus CSA at each time and overall, but still good to excellent (CCC, 0.68-0.92) in all instances. Supplementary Table S1 shows analogous ICC results that are qualitatively, and in most instances quantitatively, very similar. Correlation plots for CSAs are provided in Supplementary Figures S7 and S8. Bland-Altman plots did not show correlation between inter-rater differences and CSA size (not shown).

Table 7.

Descriptive statistics for cross-sectional areas (CSA) and myotendinous junction (MTJ) distance were calculated from the average of measurements from both readers at each time point; mean ± SD (range).

	Pre-op (n=42)	3 weeks (n=42)	12 weeks (n=42)	26 weeks (n=42)	52 weeks (n=42)
Supraspinatus CSA (cm2)	4.9±1.4 (2.0-7.9)	5.0±1.1 (3.1-8.6)	4.9±1.0 (3.4-7.8)	4.9±1.2 (1.6-8.2)	5.0±1.3 (1.4-8.5)
Infraspinatus CSA (cm2)	8.3±2.5 (3.8-15.1)	7.8±2.1 (4.3-14.4)	7.2±1.9 (4.2-14.4)	7.5±2.1 (3.2-14.2)	7.6±2.3 (3.4-14.0)
Supraspinatus MTJ distance (cm)	3.9±0.9 (2.5-6.2)	3.1±1.0 (1.3-5.9)	3.5±0.9 (2.1-5.2)	3.6±0.9 (2.3-5.8)	3.6±1.1 (2.2-6.1)

Supraspinatus myotendinous junction (MTJ) distance

Table 7 also shows the average MTJ distances at each time. MTJ distance was largest, i.e. furthest medial, preoperatively (3.9 ± 0.9 cm) and shortest at 3 weeks following repair (3.1 ± 1.0 cm). Inter-rater CCCs showed fair to moderate agreement on MTJ distance, 0.48-0.68 and 0.59 overall (Table 2). Correlation plots for supraspinatus MTJ distance are provided in Supplementary Figure 9. Bland-Altman plots did not show evidence that inter-rater differences were dependent on MTJ distance (not shown).

DISCUSSION

We assessed 10 rotator cuff tendon and muscle parameters in 42 patients preoperatively and at four time points over the first year following RCR, and described inter-rater agreement between two fellowship-trained musculoskeletal radiologists in making these assessments. Contrary to our hypothesis, exact and chance-corrected agreement between readers were generally similar across time points for each of the parameters assessed. In the few instances where a discrepant κ was observed, it was at the preoperative or 3 week time point and imprecisely estimated with a wide confidence interval. Because exact agreement was high in each instance, these discrepancies are explainable by limited sample size and extreme stringency of kappa statistics when exact agreement is high, and thus are considered to have little or no clinical significance. We did observe that the frequency of clinically significant disagreement in Sugaya grading increased with time since surgery, concomitant with the divergence of tendon pathology over more categories as partial and full-thickness re-tears developed over time.

Repair integrity is arguably the most important and conventionally reported structural outcome following RCR. Our 24% full-thickness re-tear rate at six-months and one year are consistent with most recent reports of one or two tendon rotator cuff tears arthroscopically repaired with double row techniques¹⁷. Using the 5-point Sugaya classification, exact inter-rater agreement was very good (71%-83%) for classifying 42 scans at all time points. Simple κ coefficients showed moderate to substantial (0.53-0.64) chance-corrected agreement at individual times and weighted κ coefficients almost perfect (0.85-0.92) agreement except preoperatively (0.59). Hasegawa et al reported inter-rater agreement using 5-point Sugaya scoring of 68 MRI scans read by 3 orthopedic surgeons of κ=0.31 and 0.49 at 6 and 24 months postoperatively, respectively¹⁸. Kluger et al reported weighted κ=0.72 for 2 radiologists reading 94 MRI scans at an average of 5 years (range, 1-7 years) postoperatively²³, and Niglis et al report κ=0.39 for 50 MRI scans read by 2 orthopedic surgeons and 1 radiologist at minimum of 10 years postoperatively³². Our agreement is relatively high as compared to these prior reports, even for reading MRIs obtained in the early phase of healing, possibly because we applied prospectively defined reading rules.

Repair integrity is most commonly reported using tri- (intact / partial tear or re-tear / full-thickness tear or re-tear) or dichotomized (intact / tear or re-tear) classifications^{23; 32} In our study, trichotomizing the Sugaya scale around 3 (1-2 vs. 3 vs. 4-5) did not substantially improve agreement coefficients on postoperative scans from those for the full scale. Despite high levels of statistical agreement in assessing repair integrity, however, even adjacent category disagreements can be clinically significant when the readers’ grades are mutually exclusive (i.e., Sugaya 1 or 2 vs. 3, 4 or 5; Sugaya 3 vs. 4 or 5). These clinically significant Sugaya disagreements occurred for 17.6% of scans and at all time points. Of such disagreements, almost all were adjacent-category disagreements involving Sugaya grade 3 indicating partial thickness tear, demonstrating the well-appreciated challenge of interpreting the meaning of an abnormal MRI signal intensity without a clear recurrent defect^{4; 21; 35; 37; 41}.

Inter-rater agreement for AP and ML tear dimensions was excellent preoperatively (CCC, 0.87 and 0.86, respectively) and good to excellent postoperatively, ranging from 0.61-0.96 for AP tear width and 0.72-0.88 for ML tear length. Postoperatively, there were far fewer re-tears to be measured and most CCCs were imprecisely measured as reflected in their wide confidence intervals, possibly accounting for the lower CCCs at some postoperative time points. Furthermore, when assessing tear dimensions the complex morphology of the tendon end and the frayed and partially torn margins of a tear make it difficult to accurately and consistently define the tendon margin, thus impeding accurate tear size measurement²¹. In addition, full-thickness tears can have partial thickness components at their margins, introducing uncertainty in measuring AP dimensions.

Using the Warner grading system, patients in our cohort were classified as having no or minimal supraspinatus muscle atrophy at time of surgery, and the distribution of atrophy classifications did not appear to shift substantially over one-year postoperatively. Exact agreement on classification of supraspinatus atrophy was excellent (0.90-0.95) at all time points, and simple κ coefficients generally showed substantial above-chance agreement (0.72-0.83) except for a single discrepant value at 3 weeks. Khazzam et al report simple κ=0.31 for 31 postoperative MRI scans read by 7 fellowship trained orthopedic surgeons²¹, Spencer et al report simple κ=0.25 for 27 preoperative MRI scans read by 10 fellowship-trained shoulder surgeons⁴¹ and Lippe et al report simple κ=0.25 for 31 preoperative MRI scans read by 3 fellowship-trained shoulder surgeons²⁸. Agreement in this study was higher than reported previously, possibly explained at least in part by a homogeneously low degree of atrophy.

The supraspinatus muscle is well-defined within its fossa, and accordingly the CCCs for CSA measurements of the supraspinatus muscle exceeded 0.97 at all intervals. Aleem et al showed similar, 0.90, agreement for computed tomography (CT) based supraspinatus muscle CSA measurements as assessed by the ICC ³. CCCs for measuring infraspinatus muscle CSA, while still 0.68 to 0.92, were lower than for the supraspinatus CSA measures at each interval, likely because of its less well-defined inferior border where, just proximal to the MTJ, the muscle anatomically merges inseparably with teres minor⁸. These findings and conclusion are consistent with the interrater Pearson correlation of 0.59 reported by Zanetti et al for measuring the isolated infraspinatus muscle CSA⁵⁰. These authors noted that correlation improved to 0.92 when measurements combined infraspinatus and teres minor muscles because of the “difficulty to differentiate the infraspinatus and teres minor muscles, especially in the absence of fatty atrophy”⁵⁰.

Most patients in our cohort exhibited grade 0 fatty infiltration (<10% fat) of both the supraspinatus and infraspinatus muscles at the time of surgery, though the distributions of fatty infiltration classification in both muscles appeared to shift modestly toward grade 1 (10-25% fat) during the first 12 weeks postoperation. These findings are consistent with previous quantitative reports showing a small, significant progression in muscle fat fraction from baseline to early (3-6 month) follow-up after RCR, that was greatest for failed repairs^{27; 47} Whereas evidence of observer bias was minimal for classification of muscle atrophy, observer bias and (predominantly adjacent category) disagreement was observed for fatty infiltration classifications. Accordingly, exact agreement (0.64 to 0.86) and simple κ coefficients (0.34 to 0.49) were lower for fatty infiltration than atrophy classification. The observer bias and reduced agreement for fatty infiltration scoring compared to (supraspinatus) atrophy scoring may be explained by the ability to clearly define the supraspinatus fossa on MRI, making judgment of muscle size relatively straightforward, compared to the need to interpret an abnormal MRI signal intensity in order to grade fatty infiltration. Agreement for grading supra- and infraspinatus fatty infiltration on pre- and postoperative MRIs in this study was consistent with the poor to moderate simple κ coefficients of 0.10-0.43 reported previously in the literature^{21; 28; 38; 41}. Despite the use of a standardized grading system, the reliability of MRI (or CT-based) Goutallier classification is inherently challenged by assessment of only one muscle cross-section and using a qualitative scale with wide ranges⁴⁰. Dixon MRI and proton MR spectroscopy allow for volumetric quantification of muscle fat with high reliability, although both require more intensive image processing and neither has yet gained widespread application^{1; 12; 33; 34}.

We measured the supraspinatus MTJ distance as a possible surrogate for tendon retraction during healing in our ongoing work evaluating failure with continuity in RCR²⁹. Preoperatively, the MTJ distance averaged 3.9±0.9cm, but was shorter (i.e. further lateral) by almost a centimeter at 3 weeks, most likely explainable by lateralization of the MTJ by the RCR. Subsequently, the average MTJ distance increased (shifted medially) over time approaching the average preoperative distance by six months and one year. CCCs for supraspinatus MTJ distance were fair to moderate (0.48-0.68), reflecting reader uncertainty in identifying a precise location for the supraspinatus MTJ. This uncertainty can be expected since the supraspinatus MTJ spans approximately two centimeters in its AP dimension, and its tendinous portion is longer anteriorly (mean 5.4cm, range 4.2–7.7cm, n=20) than posteriorly (2.8, 2–3.7cm)⁴⁵. More precise rules for measuring MTJ distance as a possible surrogate of tendon retraction during healing are needed.

The strength of this study is the prospective longitudinal evaluation of inter-rater agreement for making common rotator cuff assessments on MRI, preoperatively and over the first postoperative year. We found no clear evidence of inter-rater agreement being dependent on the time following surgery for any assessment suggesting that the more precise overall agreement statistic could be used to appropriately describe agreement for each measure at any of the examinations. However, this study was limited in that it can confirm only reliability but not accuracy of postoperative assessments. Furthermore, inter-rater agreement was investigated between only two fellowship-trained musculoskeletal radiologists using well-defined reading rules in the research setting where nearly all MRIs were obtained using a common protocol and assured to be of adequate quality for assessment, and assessed on a relatively small number of patients. Additionally, this study only evaluated MRI studies from patients with fully reparable tears. Hence the reported agreement statistics and their lack of apparent time-dependence may not be fully generalizable to a broader patient population with less or more rotator cuff pathology, or assessed in a less controlled, clinical setting.

CONCLUSION

In summary, despite the well-appreciated challenges in interpreting MRI of rotator cuff tissues, this study showed that using defined reading rules in a research setting, two fellowship trained radiologists could reliably evaluate tendon integrity, tear dimensions, muscle atrophy and cross-sectional areas, even in the early postoperative period. Trichotomizing the Sugaya scale around grade 3 did not substantially improve unweighted or weighted agreement coefficients from those for the full scale on postoperative scans, suggesting the possibility that the full 5-point scale might be used by trained readers in the research setting to provide greater discrimination of repair integrity than the trichotomy with little loss of reliability. Consensus reading of categorical variables and averaging of independently measured continuous variables are strategies to mitigate unreliability and potentially improve the accuracy of research assessments. However, the presence of clinically significant disagreements, even in such favorable circumstances as the research setting, indicates need for improved imaging tools for precise rotator cuff evaluation.

Supplementary Material

1

Figure S1. Agreement plots of tendon integrity (Sugaya classification, Read 1) between the two readers preoperatively, at 3, 12, 26 and 52 weeks postoperatively, and overall.

2

Figure S2. Correlation plots for AP tear width (Read 1&2). Nine of 71 (13%) measures of AP tear width differed by ≥ 10mm (considered a clinically relevant amount) between readers.

3

Figure S3. Correlation plots for ML tear length (Read 1&2). Seven of 71 (10%) measures of ML tear length differed by ≥ 10mm (considered a clinically relevant amount) between readers.

4

Figure S4. Agreement plots of supraspinatus muscle atrophy (Warner classification) between the two readers preoperatively, at 3, 12, 26 and 52 weeks postoperatively and overall.

5

Figure S5. Agreement plots of supraspinatus muscle fatty infiltration (Goutallier classification) between the two readers preoperatively, at 3, 12, 26 and 52 weeks postoperatively, and overall.

6

Figure S6. Agreement plots of infraspinatus muscle fatty infiltration (Goutallier classification) between the two readers preoperatively, at 3, 12, 26 and 52 weeks postoperatively, and overall.

7

Figure S7. Correlation plots for CSAs of supraspinatus muscle; N=42 for all time points and N=210 overall.

8

Figure S8. Correlation plots for CSAs of infraspinatus muscle; N=42 for all time points and N=210 overall.

9

Figure S9. Correlation plots for supraspinatus MTJ distance; N=42 for all time points and N=210 overall

10

Supplementary Table S1. Interclass correlation coefficients (ICC) and 95% CIs of the inter-rater agreement of the continuous MRI parameters from all time points of acquisition and overall. Agreement statistics for AP and ML tear dimensions were calculated from a combination of Read 1 and Read 2 measurements and included N=39, 5, 7, 10, 10 and 71 measurements for preop, 3 weeks, 12 weeks, 26 weeks, 52 weeks and overall respectively. Agreement statistics for all other parameters were calculated from Read 1 assessments and included N=42 for all time points and N=210 overall. Cross-sectional area (CSA), myotendinous junction (MTJ).