Reliability of assessment methods for scapular dyskinesis in asymptomatic subjects: A systematic review

Abstract

Objective

The aim of this systematic review was to evaluate the available published evidence on the intra- and inter-rater reliabilities of assessment methods used for identifying and measuring scapular dyskinesis (SD) in asymptomatic subjects.

Methods

A systematic electronic literature search was performed in PubMed, Physiotherapy Evidence Database, Scopus, and the Cochrane Library, and studies on the intra- and inter-rater reliabilities of physical examination tests used for identifying SD in asymptomatic people were identified. Methodological quality of the studies meeting the inclusion criteria was assessed using the Quality Appraisal of Reliability Studies (QAREL) checklist by two reviewers. The overall level of evidence of this systematic review was determined by the Modified Cochrane Back Pain Criteria based on previous research which was modified for reliability studies of the shoulder complex.

Results

The literature search generated 388 results, and only 14 articles met the inclusion criteria. In these studies, reliabilities of two qualitative and five quantitative methods for the assessment of SD were analyzed. The QAREL checklist revealed that 12 studies had moderate risk of bias and 2 had high risk of bias. Additionally, none of the studies were of high quality. On the basis of the Modified Cochrane Back Pain Criteria, the overall level of evidence was moderate. Most of the studies including quantitative measurement methods found good to excellent inter- and intra-rater reliability values. Most of the studies including qualitative methods found low-to-moderate intra- and inter-rater reliability values.

Conclusion

Considering the available published evidence, there is lack of high-quality studies evaluating the inter- and intra-rater reliabilities of qualitative or quantitative methods used for the assessment of SD. There are no qualitative methods with high reliability that are fit for clinical applications. Some quantitative methods with higher reliability are present, but clinicians should be aware of the methodological flaws that studies evaluating these methods suffer from.

Level of Evidence

Level II, Diagnostic study

Introduction

Proper functioning of the upper extremity requires normal scapulothoracic kinematics in order to maintain the integrity of many structures around the shoulder complex (1). Any alteration in the position or motion of the scapula is referred to as scapular dyskinesis (SD) (2). SD is characterized by a prominent scapular medial border, uncoordinated motion with an increased scapular elevation when the arm is ascending and increased downward rotation when the arm is descending (3). SD has been associated with many musculoskeletal pathologies in previous studies (3). However, it is still difficult to conclude whether SD represents a cause or effect of any pathological symptom owing to the observational nature of the studies (3, 4). Furthermore, previous research suggests that SD is highly prevalent in asymptomatic people and especially in overhead athletes.

A recent systematic review by Burn, McCulloch showed that overhead athletes had a greater reported prevalence (61%) of SD in comparison with non-overhead athletes (33%) (5). In addition, the systematic review of Hickey, Solvig (4) showed that asymptomatic athletes with SD have 43% greater risk of developing shoulder pain than those without SD. Furthermore, studies have shown that SD affects performance in overhead athletes (6, 7) and may increase the prevalence of shoulder pain in the entire population (3, 8). Thus, it remains important for clinicians to have a reliable assessment method for SD screening.

Reliability is the degree to which a measurement produces stable and consistent results (9). Both intra- and inter-rater reliabilities are very useful for tracking changes and progression over time as well as for detecting abnormalities in scapular posture among different clinicians efficiently (10). Therefore, examining the inter- and intra-rater reliabilities of SD assessment methods is clinically imperative.

To our knowledge, systematic reviews in this area are lacking. The only relevant systematic review is by Lange, Struyf (11). In their systematic review, they included all relevant studies in symptomatic subjects of the entire population and they reported lack of high-quality studies evaluating the intra-rater and the inter-rater reliabilities in the methods used for the evaluation of SD. Additionally, the review of the literature from Struyf, Nijs (12) offered recommendations on the assessment of SD in the entire population. However, this was not a systematic review and reliability was not assessed exclusively. Thus, the quality of the studies mentioned was not assessed.

Apart from evaluating the reliability of SD methods in symptomatic subjects, it seems reasonable to evaluate similar studies in asymptomatic subjects as well. As there is evidence to suggest that SD precedes symptomatic shoulder problems, it can be speculated that immediate SD diagnosis can prevent the onset of these symptoms. Therefore, evaluating SD methods in asymptomatic subjects is extremely important for all clinicians. Methods that have been conducted in asymptomatic subjects can be used for screening purposes and these should be reviewed and critically evaluated.

Thus, the aim of this systematic review was to evaluate the intra-rater and the inter-rater reliabilities of the methods used for the assessment of SD in asymptomatic subjects.

Materials and Methods

Study design

For this review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was used. This systematic review was registered a priori within the International prospective register of systematic reviews (PROSPERO: CRD42019133383).

Inclusion/exclusion criteria

In this review, studies that examined the intra-rater and inter-rater reliabilities of any physical examination test that aimed to identify SD in asymptomatic people were considered. Studies on subjects of every age and setting were considered eligible. Studies that assessed SD with a single test or as part of other examination tests were included. Only English articles were included. All the articles that were included in this systematic review had clearly stated that reliability evaluation was one of the main aims of the study.

Studies that did not report the measurement procedure or did not provide a relevant citation for the measurement procedure were excluded. Studies that did not name the measurement procedure, so it could be traced in the text, were also excluded. Similarly, studies that used a combination of tests and did not assess reliability for each test separately were also excluded. Studies that recruited subjects with any shoulder-related symptom or studies that did not use the terms asymptomatic or healthy or pain-free participants were also excluded. Studies that used a mixture of symptomatic and asymptomatic participants were also excluded (13).

Search strategy

PubMed, Physiotherapy Evidence Database, Scopus, and the Cochrane Library were searched and a “Building Block Search Strategy” was used (14). The searching strategy was developed in Pubmed and then was applied to the other databases. Reference lists of all articles were searched for relevant articles, as well as other grey literature and key journals.

The searching strategy terms were as follows: dyskinesia OR dyskinesis OR asymmetry OR asymmetries OR kinematics OR kinematic OR abnormalities OR abnormality OR positioning OR position OR motion OR static OR dynamic AND scapula OR scapular AND (evaluation OR evaluations OR rating OR test OR tests OR diagnosis OR diagnostic OR examination OR examinations OR assessment OR assessments OR measurements OR measurement AND reproducibility of results OR observer variation AND reliability OR reproducibility OR repeatability OR agreement ICC OR Intra Class Correlation Coefficient OR interrater or inter-rater or inter and rater OR interexaminer OR inter-examiner OR inter and examiner OR intertester OR inter-tester OR interobserver OR inter-observer OR inter and observer intrarater OR intra-rater OR intra and rater OR intraexaminer OR intra-examiner OR intra and examiner OR intratester or intra-tester OR intra and tester OR intraobserver OR intra-observer OR intra and observer.

Selection of studies

Two independent reviewers identified relevant articles from the titles and the abstracts. Then, the reviewers read the full text of each article and decided whether the article was eligible for inclusion. If consensus was not reached, a third reviewer was consulted. Procedures for study selection were discussed a priori to avoid lack of agreement. An inclusion and exclusion criteria checklist assisted this procedure. Articles were exported to EndNote for removal of duplicates and further processing.

Then, one reviewer performed data extractions, which were subsequently checked by the second reviewer. Data related to the objectives of the studies were initially extracted to ensure that each study meets the objectives of our review and then the rest of the data such as subjects, examiners, tests, outcome measures, and results were extracted.

Quality assessment

Methodological quality of the included studies was assessed by the Quality Appraisal of Reliability Studies (QAREL) checklist. QAREL has been shown to be reliable for the assessment of reliability studies (15). The QAREL consists of an 11-item checklist that covers several key domains when assessing the quality of reliability studies (15). Each item of the checklist can be answered with “yes”, “no” or “unclear”, and “not applicable”. The criteria for choosing yes or no were defined a priori by the reviewers. The number of items that checked “yes” defined the risk of bias of each study. A score of 4 or less (≤4) was set as high risk of bias, five (5) to seven (7) as moderate, and eight or more as low (≥8) (16).

Item 2 was checked “Yes” if the authors of the study provided information on specific training in the measurement method that was evaluated. Item 5 was not applicable as there was no accepted reference standard for scapular position measurement. Item 8 was “Yes” if there was some form of randomization performed between the examination procedures. Item 11 was checked as “Yes” for studies that provided at least the Intraclass Correlation Coefficient (ICC) and standard error of measurement values. The overall level of evidence, from the studies selected, was established based on the Modified Cochrane Back Pain Criteria as suggested in a similar study (box 1) (16).

Box 1.

• Modified Cochrane Back Pain Group criteria for overall level of evidence

• Strong evidence: Consistent findings from 2 or more low risk of bias studies with consistent findings (8 or more of 11 items on QAREL).

• Moderate evidence: One low risk of bias study, 2 or more moderate risk of bias (5–7/11 items) studies, or 1 low and 1 or more moderate-risk studies; with consistent findings.

• Limited evidence: One and more high risk of bias study (1–4/11 items) with consistent findings.

• Lacking evidence: No published studies found.

• Conflicting evidence: Conflicting results within a quality level.

*Can ignore high risk of bias studies, when they conflict with low- or moderate-risk of bias studies in the same context; where studies indicate that a measure’s reliability is different across contexts, the overall quality ratings and conclusions can be separated by context, and these contextual differences should be identified and discussed.

In order to ensure a thorough understanding of the QAREL rating procedure, the first study was conducted by both reviewers at the same time as a calibration exercise. Then, the two reviewers independently evaluated all the other studies using the QAREL 11-item checklist. After completion of the evaluation procedure, the reviewers compared the results. In case of a disagreement, the reviewers discussed their evaluation until consensus was reached. A third reviewer was available in case of on-going disagreement.

Data synthesis

Agreement of the reviewers’ final study selection was calculated with the Cohen’s kappa statistic. For assessing the reliability measures of each study, the ICC values were used. These values represented poor (0.40), moderate (0.40–0.75), and excellent (>0.75) reliability as described by Fleiss (17). For categorical variables, the kappa statistic was used. Kappa values indicated poor (0.00), slight (0.00–0.20), fair 0.21–0.40, moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect agreement (>0.80) (18). In order to establish agreement for the quality of the studies reviewed between the reviewers, the ICC was calculated.

Results

The flow of the search and the selection of the articles for review are presented in the PRISMA chart (Figure 1). The literature search generated 398 results and after removal of duplicates, 331 articles were screened. From those, only 14 articles met the inclusion criteria after separately screening the abstract along with the title, and then the full text (19–32) (Table 1).

Figure 1.

PRISMA

Table 1.

Description of the subjects, methods, measurement procedures, positions, and outcome measures of the included studies

	Study	Subjects	Method	Measurement procedure	Position	Outcome measure
1.	da Costa et al., 2010	30, 15 M and 15 F, 26.5 (±3.8) years old. Unilat.	Protraction & Depression	Palpation Meter (PALM)	Standing, static, rest/semidynamic elev. (90 scaption, full scaption)	Quantitative translation in cm
2.	De Groef et al., 2017	30 F, 40–60 years old, Bilat.	Inclinometry shoulder ROM, acromion–table distance, pectoralis minor muscle length, Scapular upward rotation with two inclinometers, PPT at different locations in the upper body	Sliding Caliper, inclinometer, algometer	Standing, Supine (dynamic 45°, 90°, 135° scaption, dynamic flexion and abduction)	Quantitative translation in cm, kg for pressure pain thresholds
3.	Du et al., 2017	29, 14 M and 15 F, 23 (±2.1) years old Bilat.	Scapular medial border and inferior angle prominence measurement	Modified Weon Scapulometer, 3-D data for validation	Standing static at rest	Quantitative translation in mm
4.	Ellenbecker et al., 2012	71 M 18–32 years old, Bilat.	Scapular test – Classification of Scapular dyskinesis	Video recorder	Semidynamic Scapular Plane Elevation with 2-pound dumbbell in each hand	Categorization with Scoring (4-type)
5.	Hong et al., 2011	16 M, 19.5 (±1.3) years old Bilat.	Scapular medial border posterior displacement in static and semi-dynamic testing positions	Standardize scapular medial border posterior displacement measurement tool	Standing at rest, Press up exercise on chair	Quantitative translation in cm
6.	Madsen et al., 2011	14, 7 M 7 F, 17 (14–22) years old, Bilat.	Wall push up, Scaption	Video recorder	Standing, dynamic movements	Yes/No in dyskinesis (either winging or dysrhythmia)
7.	O’Connor et al., 2016	15 M, 19.46 (±0.63) years old Bilat.	Scapular Control Test	Observation	Standing, semidynamic abduction to 180°	Categorization with Scoring
8.	Pontin et al., 2013	30, 13M 17 F, 24.5 (±7.1), Bilat.	Burkart Protocol (vertical distance of scapulae from the spine and rotation asymmetries)	Tape measurement	Standing static at rest	Differences >1.5 cm or > 5° were considered abnormal (Presence of dyskinesis)
9.	Rossi et al., 2017	75, 45 M and 30 F, 23.7 (±7.0) M 17.6 (±5.7) F years old, Bilat.	4-type classification, Yes/No classification, Scapular dyskinesis test	Observation	Standing, elevation in scapular plane (participants who weighed <68.1 kg and ≥68.1 kg used 1.5-kg and 2.5-kg dumbbells)	Categorization with Scoring (4-type classification, Yes/No classification, and scapular dyskinesis test (SDT)
10.	Shadmehr et al., 2014	30 M, 22.53 (±3.72) years old Bilat.	Modified Lateral Scapular Slide Test	Digital Caliper, goniometer	Neutral position, unloaded scaption, loaded scaption, full scaption	Quantitative translation in cm
11.	Struyf et al., 2009	30, 17 M and 13 F, 21.5 (±5.8) years old, Bilat.	Scapular observation, Pectoralis minor length test, Lateral Scapular Slide Test (LSST)	Sliding caliper & tape, Observation	Standing (resting with both arms relaxed (thumbs facing forward), hands placed on ipsilateral hips (thumbs facing backward), and arms in 90° of humeral abduction in the frontal plane (thumbs facing up)	Quantitative translation in cm, Categorization (Presence of dyskinesis)
12.	Tucker and Ingram, 2012	30 M, 21.9 (±2.3) years old. Unilat.	Scapular Upward Rotation	Electrical and Digital Inclinometer	Standing at rest and standing semidynamic at 60°, 90°, and 120° scaption	Quantitative translation in degrees
13.	Gibson et al., 1995	32, 17 M and 15 F, 26.16 years old. Bilat.	Lateral Scapular Slide Test (LSST) in three positions	Ruler and unmarked string	Standing semidynamic (static rest and dynamic abd./int.rot., arm on hip and 90° abd.)	Quantitative translation in cm
14.	Plafcan et al., 1997	40, 21 M 19 F, 26.6 (±4.9) Y. Bilat.	Posterior Displacement of Scapula	The Perry Tool	Unloaded in standing, static, rest. Loaded position 10% BW	Quantitative translation in degrees

M: male; F: female; Y: year; Bilat.: bilateral; cm: centimeters; mm: millimeters; kg: kilos; ±: standard deviation; BW: bodyweight

Reviewers’ agreement in regards to title and abstract selection yielded a Cohen’s kappa statistic of 0.94 (95% CI: 0.8–0.99) and of full-text articles 0.77 (95% CI: 0.56–0.97). The raters of the QAREL checklist showed excellent agreement with an ICC (2,1) of 0.87 (95% CI: 0.65–0.95). The QAREL methodological assessment is summarized in Table 2. The quality of the studies ranged from 4/11 to 7/11 positive ratings. These results showed that none of the studies were of high quality. The vast majority of the included studies were rated as having a moderate risk of bias except from two studies that were rated as having a high risk of bias (23, 31). The overall level of evidence was moderate according to the Modified Cochrane Back Pain Criteria (16, 33), since all studies were rated as having a moderate risk of bias except from two studies with high risk of bias (23, 31).

Table 2.

Quality appraisal of included studies using QAREL checklist

Study	Study type/Design	QAREL											Risk of bias (Quality)
		1	2	3	4	5	6	7	8	9	10	11
(da Costa et al., 2010)	1/C	Y	Y	?	N	NA	NA	?	Y	Y	Y	Y	6/11 Moderate (Moderate)
(De Groef et al., 2017)	1/A	Y	Y	Y	NA	NA	NA	?	?	NA	Y	Y	5/11 Moderate (Moderate)
(Du et al., 2017)	1/C	Y	Y	Y	Y	NA	NA	?	Y	N	Y	Y	7/11 Moderate (Moderate)
(Ellenbecker et al., 2012)	2/A	Y	Y	Y	NA	NA	NA	?	?	NA	Y	Y	5/11 Moderate (Moderate)
(Hong et al., 2011)	1/C	Y	Y	N	N	NA	NA	?	N	?	Y	Y	4/11 High (Low)
(Madsen et al., 2011)	2/A	Y	Y	Y	NA	NA	NA	?	?	NA	Y	Y	5/11 Moderate (Moderate)
(O’Connor et al., 2016)	2/C	Y	Y	Y	?	NA	NA	?	?	Y	Y	Y	6/11 Moderate (Moderate)
(Pontin et al., 2013)	2/A	Y	Y	Y	NA	NA	NA	?	N	NA	Y	Y	5/11 Moderate (Moderate)
(Rossi et al., 2017)	1+2/C	Y	Y	Y	N	NA	NA	?	N	N	Y	Y	5/11 Moderate (Moderate)
(Shadmehr et al., 2014)	1/C	Y	Y	Y	N	NA	NA	?	N	Y	Y	Y	6/11 Moderate (Moderate)
(Struyf et al., 2009)	1+2/A	Y	Y	Y	NA	NA	NA	?	N	NA	Y	Y	5/11 Moderate (Moderate)
(Tucker and Ingram, 2012)	1/B	Y	Y	NA	Y	NA	NA	?	N	N	Y	Y	5/11 Moderate (Moderate)
(Gibson et al., 1995)	1/C	Y	Y	?	N	NA	NA	?	N	N	Y	Y	4/11 High (Low)
(Plafcan et al., 1997)	1/C	Y	Y	Y	Y	NA	NA	?	N	N	Y	Y	6/11 Moderate (Moderate)

1= Quantitative measurement; 2= Qualitative classification; A= Inter-rater reliability; B= Intra-rater reliability; C= Intra-rater and Inter-rater reliability; QAREL= Quality Appraisal for Reliability Studies; Y= Yes; NA= Not Applicable; N= No; ?= Unclear. QAREL Questions: 1=Was the test evaluated in a sample of subjects who were representative of those to whom the authors intended the results to be applied?; 2=Was the test performed by raters who were representative of those to whom the authors intended the results to be applied?; 3=Were raters blinded to the findings of other raters during the study?; 4=Were raters blinded to their own prior findings of the test under evaluation?; 5=Were raters blinded to the results of the reference standard for the target disorder?; 6=Were raters blinded to clinical information that was not intended to be provided as part of the testing procedure or study design?; 7=Were raters blinded to additional cues that were not part of the test?; 8=Was the order of examination varied?; 9=Was the time interval between repeated measurements compatible with the stability (or theoretical stability) of the variable being measured?; 10=Was the test applied correctly and interpreted appropriately?; 11=Were appropriate statistical measures of agreement used?

The results of the reliability values and the methodological characteristics of the studies are presented in Table 3. Most of the studies that assessed intra-rater reliability did not mention whether the raters were blinded to their measurement results (19, 23, 27, 28, 31) and in some others, it was implied that blinding of the measurement results was not possible due to the nature of the measurement procedure (i.e., observation and measurement setting) (20, 22, 24, 26, 29, 34).

Table 3.

Data extraction of selected studies

Study	Test	Results: Reliability type (reliability index) Reliability result Measurement or Position of subject: Test-Landmark=Result	Measures of Uncertainty: Reliability type (Uncertainty index) Measurement or position of subject: Test-Landmark=Result	Raters and Training	Feasibility and Scapular Dyskinesis classification	Methodological Flaws
da Costa et al., 2010	Assessment of Scapular Protraction and Depression with the Palpation Meter (PALM)	Intra-rater: ICC Moderate to Excellent Neutral Position: Inferior angle=0.89 Root of the spine of the scapula=0.81 Moderate to Excellent 90° of Scaption Position: Inferior angle=0.89 Roots of the spine of the scapula=0.78 Moderate to Excellent Full Scaption Position: Inferior angle=0.71 Root of the spine of the scapula=0.85 Inter-rater: ICC Neutral Position: Excellent Inferior angle=0.89 Root of the spine of the scapula=0.77 90° of Scaption Position: Moderate Inferior angle=0.74 Roots of the spine of the scapula=0.74 Full Scaption Position: Moderate to Excellent Inferior angle=0.85 Roots of the spine of the scapula=0.67	Intra-rater: SEM Neutral Position: Inferior angle=0.70 Root of the spine of the scapula=0.69 Normalized distance=0.18 Vertical distance=0.79 90° of Scaption Position: Inferior angle=0.56 Roots of the spine of the scapula=0.79 Full Scaption Position: Inferior angle=0.98 Roots of the spine of the scapula=0.79 Inter-rater: Neutral Position: Inferior angle=0.59 Root of the spine of the scapula=0.69 Normalized distance=0.19 Vertical distance=0.64 90° of Scaption Position: Inferior angle=0.98 Roots of the spine of the scapula=0.84 Full Scaption Position: Inferior angle=0.74 Roots of the spine of the scapula=0.87	3 PTs 10 h training	CA with training and equipment No SD classification	Raters not experienced with the Palpation Meter (PALM)
De Groef et al., 2017	Inclinometry shoulder ROM, acromion–table distance, pectoralis minor muscle length, Scapular upward rotation with two inclinometers, PPT at different locations in the upper body	ICC Scapular Positioning Using a Sliding Caliper Excellent Acromion-Table=0.91 (Neutral position) Acromion-Table=0.87 (Retraction) Pectoralis minor index=0.91 Acromion-Table-Distance=0.91 (Neutral position) Acromion-Table-Distance=0.87 (Retraction) Pectoralis minor length=0.91 Shoulder Mobility Using a Gravity Inclinometer: Moderate to Excellent Forward flexion=0.67 Abduction=0.77 Scapular Upward Rotation Using Two Gravity Inclinometers: Poor In resting position – Scaption (45°, 90°, 135°, maximal)=0.14–0.39 Inter-rater second cohort: ICC Poor to moderate Shoulder Mobility Using a Gravity Inclinometer: Forward flexion=0.83 Scapular Upward Rotation Using Two Gravity Inclinometers: In resting position – Scaption (45°, 90°, 135°, maximal)=0.26–0.52	Inter-rater first cohort: SEM Scapular Positioning Using a Sliding Caliper: Acromion-Table=0.22 % of body length (Neutral position) Acromion-Table=0.28 % of body length (Retraction) Pectoralis minor index=0.16% of body length Acromion-Table-Distance=0.38 mm (Neutral position) Acromion-Table-Distance=0.49 mm (Retraction) Pectoralis minor length=0.28 mm Shoulder Mobility Using a Gravity Inclinometer: Forward flexion=6° Abduction=5° Scapular Upward Rotation Using Two Gravity Inclinometers: In resting position–Scaption (45°, 90°, 135°, maximal)=3°–8° Inter-rater second cohort: SEM Shoulder Mobility Using a Gravity Inclinometer: Forward flexion=5° Scapular Upward Rotation Using Two Gravity Inclinometers: In resting position–Scaption (45°, 90°, 135°, maximal)=2–8°	2 Raters 4-h training	CA with training and equipment No SD classification	Subjects’ age 37–88 instead of the proposed 40–60 Single measurements
Du et al., 2017	Posterior displacement of the scapular with modified scapulometer	Intra-rater: ICC Excellent Posterior Displacement=0.99 Inter-rater: Excellent Posterior Displacement=0.95	Intra-rater: SEM Posterior Displacement=0.7 Inter-rater: Posterior Displacement=0.8	2 Raters No information on training	NCA No SD classification	No measurement training
Ellenbecker et al., 2012	1)4-type method 2)Yes/No method 3)Scapular symmetry evaluation	Inter-rater Kappa Poor 1) 4-type: Left Upper Extremity K: 0.245 Right Upper Extremity K: 0.186 2) Yes/No Method: Left Upper Extremity K: 0.264 Right Upper Extremity K: 0.157 3) Scapular symmetry/asymmetry K: 0.084	Not reported	4/no info on training	CA/SD classification	Raters from different professional backgrounds/No measurement training
Hong et al., 2011	Posterior displacement of the scapular with modified scapulometer	Intra-rater: ICC Excellent Posterior Displacement At Rest=0.97 And Posterior Displacement with the Sitting Hand Press Up=0.95 Inter-rater: ICC Excellent Posterior Displacement At Rest=0.89 Posterior Displacement with the Sitting Hands Press Up=0.88	Intra-rater: SEM Posterior Displacement At Rest=0.16 cm And Posterior Displacement with the Sitting Hand Press Up= 0.15 cm Inter-rater: SEM Posterior Displacement At Rest=0.16 cm Posterior Displacement with the Sitting Hands Press Up=0.15 cm	2 Raters No information on training	CA No SD classification	No information on training No information on randomization No information on blinding procedures Single measurements
Madsen et al., 2011	Yes/No method for SD with: 1. Wall-push-up 2. Scaption	Inter-rater: Kappa Substantial Wall push up=0.69 Scaption=0.75	Not reported	2 Raters No information on training	CA SD classification	No information on randomization No information on measurement training
O’Connor et al., 2016	Scapular-Control Test	Intra-rater: ICC Moderate-Excellent Test Items: 1)Winging=0.92 2)Control of scapula when lifting=1 3)Control of scapula when lowering=0.63 4)Symmetry between both scapulae=0.60 Inter-rater: ICC Excellent Test Items: 1)Winging=0.93 2)Control of scapula when lifting=1 3)Control of scapula when lowering=0.80 4)Symmetry between both scapulae=0.93	Intra-rater: SEM Test Items: 1) Winging=0.08 2) Control of scapula when lifting=0 3) Control of scapula when lowering:0.09 cm 4) Symmetry between both scapulae:0.12 cmInter-rater: SEM Test Items: 1) Winging=0.02 cm 2) Control of scapula when lifting=0 3) Control of scapula when lowering=0.06 cm 4) Symmetry between both scapulae=0.04 cm	3 Raters Training included (training duration not reported)	CA SD classification	No information on randomization
Pontin et al., 2013	Evaluation of Scapular Differences in a static position in comparison with an X-ray	Inter-rater: ICC Moderate-Poor 1)Vertical distance of the superior angle of the two scapulae=0.73 2)Distance between the superior angle of the scapula and the midline of the spinal column=0.68 3)Angle between the medial border and the vertical line of the superior angle of the scapula Right and Left side= 0.38–0.20	Not reported	2 Raters Trained (training durations not reposted)	CA SD classification	X-rays performed by different technicians Unknown whether the examiners could properly evaluate X-rays
Rossi et al., 2017	Three SD classifications: 1) 4-type classification, 2) Yes/No classification, and 3) Scapular Dyskinesis Test(SDT)	Intra-rater: Kappa Moderate 1)4-type=0.67 2)Yes/No=0.79 3)Scapular Dyskinesis Test(SDT)=0.66 Inter-rater: Kappa Coefficient Moderate 1)4-type=0.71–0.85 Almost Perfect 2)Yes/No= 0.81 Substantial 3)Scapular Dyskinesis Test (SDT)= 0.74	Not reported	2 Raters 9-hour training	CA SD classification	No information on randomization Evaluation only with load
Shadmehr et al., 2014	Modified lateral scapular slide Test. Measurement of dyskinesis in 7 positions: Arms at the side (P1) 90° of abduction (P2) 90° of scaption without having a weight in hands (P3) 90° of scaption with three different weights (1, 2, and 4 kg) in hands (P4, P5, and P6, resp.) 180° of scaption without weight in hands (P7)	Intra-rater: ICC Excellent P1=0.97 P2=0.88 P3=0.90 P4=0.92 P5=0.94 P6=0.95 P7=0.93 Inter-rater: ICC Moderate to Excellent P1=0.87 P2=0.77 P3=0.85 P4=0.70 P5=0.80 P6=0.78 P7=0.71	Intra-rater: SEM P1=0.32 P2=0.74 P3=0.59 P4=0.45 P5=0.45 P6=0.41 P7=0.37 Inter-rater: SEM P1=0.71 P2=1.08 P3=0.82 P4=0.97 P5=0.78 P6=0.78 P7=0.91	2 Raters 30 minutes training	CA No SD classification	Not adequate training and blinding procedures One measurement only for each rater in each position
Struyf et al., 2009	1) Observation at rest, with arms at hips and with arms at 90° of abduction 2) Acromial Distance (Pectoralis minor length test) 3)Modified lateral scapular slide test (LSST)	Inter-rater Kappa 1) Visual Observation At rest = 0.41 (Moderate) Unloaded abduction to 180° =0.63 (Substantial) Loaded abduction to 180° = 0.36 (Fair) Scapular tilting: Rest=0.48 (Moderate) Unloaded=0.52 (Moderate) Loaded=0.24 (Fair) Scapular winging: Rest=0.42 (Moderate) Unloaded=0.78 (Substantial) Loaded=0.50 (Moderate) ICC 2) Moderate Acromial distance (Pectoralis minor length): Relaxed Shoulders=0.72 (Moderate) Retracted shoulders=0.75 (Moderate) 3) LSST: 90° of humeral elevation with a 1 kg-load=0.63 180° of humeral elevation=0.58	Inter-rater Acromial distance: Relaxed Shoulders SEM=1.7 MDC=4.71 Retracted shoulders SEM=1.36 MDC=3.77 Inter-rater reliability for LSST: 90° of humeral elevation with a 1kg-load SEM=1.85 MDC=5.13 180° of humeral elevation SEM=1.18 MDC=3.27	2 Raters Training included	CA SD classification	No information on randomization Single measurements Muscle fatigue may have influenced reliability in loaded measurements
Tucker and Ingram, 2012	Measurement of Scapular upward Rotation with a modified inclinometer	Intra-rater: ICC Excellent Positions: Rest=0.89 60° elevation=0.95 90° elevation=0.95 120° elevation=0.97	Intra-rater: SEM Positions: Rest=1.8 60° elevation=1.5 90° elevations=1.6 120° elevation=1.1	1 Rater No information on training	CA No SD classification	Not clear on blinding procedures No information on measurement training
Gibson et al., 1995	1) Scapular test (De Vita technique) 2) Scapular test in 3 positions K-1=arms relaxed at the side K-2= arms rested at hips K-3= arms at 90° of abduction All tests performed on both the Dominant (D) and Non-Dominant (ND) side	Inter-rater: ICC De Vita technique Excellent D = 0.91 ND = 0.92 K-1 Moderate D = 0.67 ND = 0.69 K-2 Moderate D = 0.52 ND = 0.53 K-3 Poor D = 0.28 ND = 0.18 Intra-rater: De Vita technique Excellent D = 0.92 ND = 0.93 K-1 Excellent D = 0.92 ND = 0.91 K-2 Excellent D = 0.92 ND = 0.88 K-3 Excellent D = 0.89 ND = 0.91	Inter-rater: SEM De Vita technique D = 0.65 cm ND = 0.60 cm K-1 D = 1.04 cm ND = 1.02 cm K-2 D = 1.20 cm ND = 1.17 cm K-3 D = 1.59 cm ND = 1.65 cm Intra-rater: De Vita D = 0.49 cm ND = 0.53 cm K-1 D = 0.44 cm ND = 0.49 cm K-2 D = 0.64 cm ND = 00.61 cm K-3 D = 0.79 cm ND = 0.56 cm	2 Raters No information on training	CA SD classification	No information on randomization No information on measurement training
Plafcan et al., 1997	Posterior scapular displacement “Perry Tool” Test performed with weight and without weight	Intra-rater: ICC Excellent 1)Unweighted =0.98 2)Weighted =0.97 Inter-rater: Excellent 1) Unweighted =0.92 2)Weighted =0.95	Intra-rater: 1) Unweighted = 0.9 cm 2) Weighted =1.1 cm Inter-rater: 1) Unweighted =1.4 cm 2)Weighted =1.7 cm	2 Raters No information on training	CA with training and equipment No SD classification	No information on measurement training

CA: clinically applicable; SD: scapular dyskinesis; ICC: intraclass correlation coefficient / interclass correlation coefficient; K: Kappa Coefficient; SEM: standard error measurement (in centimeters); D: dominant; ND: non dominant; MDC: minimal detectable change with 95% confidence Interval (in centimeters); P: position; PPT: pressure pain threshold; PTs: physiotherapists; K- l = kibler measurement #I, K-2: kibler measurement #2, K-3: kibler measurement #3, Info: information, CM: centimeters; ROM: range of motion

From the studies that examined inter-rater reliability, three studies did not provide clear information into whether the raters were blinded to the results of other raters (19, 23, 31). The most important aspects of the quality assessment were that none of the studies mentioned whether the raters were blinded to additional cues of the subjects involved, such as skin markers or tattoos. Furthermore, none of the studies mentioned whether the raters were blinded to other clinical information. From the QAREL assessment, we found that 9% of the items were judged as unclear (Table 2). In addition, for the inter-rater reliability, re-measurement in every study did not follow the recommended time limits to avoid a memory effect, which may lead to overestimation of the reliability values (35).

From the 14 studies selected, 10 studies assessed scapular position quantitatively with several tools such as handmade scapulometers (21, 23, 32), inclinometers (20, 30), and calipers (20, 28, 29), specially made measurement devices of scapular motion in the horizontal plane such as the Palpation Meter (19) and measurement tapes (Table 4) (29, 31). From these studies, eight were classified as of moderate quality and two were classified as of low quality based on the QAREL. The methods that were classified as quantitative were based on scapular position measurement methods and tools that measured differences between the scapulae with a metric system (i.e., cm).

Table 4.

Quantitative assessment of scapular dyskinesis

Measurements	Method/Number of studies	Intra-rater Reliability (ICC)/Number of studies*	Inter-rater Reliability (ICC)/Number of studies*
1. Protraction and Depression	1. Palpation Meter (PALM)/1	1. Good to Excellent/27	1. Moderate–Excellent/27
2. Pectoralis Minor Muscle Length	1. Sliding Caliper/2	-	1. Excellent/28
			2. Moderate/37
3. Scapular Upward Rotation	1. Inclinometer/2	1. Excellent/382.	1. –
		2. -	2. Moderate–Poor/28
4. Scapular Posterior Displacement	1. Scapulometer/3	2. Excellent/29	1. Excellent/29
		3. Excellent/40	2. Excellent/40
		4. Excellent/31	3. Excellent/31
5. Scapular slide test 3z	1. Digital caliper and goniometer/1	1. Excellent/36	1. Excellent-Moderate/36
	2. Sliding Caliper and tape/1	2. –	2. Moderate/37
	3. Ruler and Unmarked String/1	3. Excellent/39	3. Excellent/39

ICC: Intraclass Correlation Coefficient,

^*ICC interpretation as poor (0.40), moderate (0.40–0.75) and excellent (>0.75) based on Fleiss (26)^z

From the 14 studies reviewed, 5 studies assessed the presence of SD qualitatively, through direct visual observation (25, 27, 29) or through video recording (22, 24) (Table 5). It should be mentioned that one study used both quantitative and qualitative measurement methods (29). On the basis of QAREL, the methodological quality of these studies was classified as moderate. The methods that were classified as qualitative were based on clinical observation of scapular position.

Table 5.

Qualitative assessment of scapular dyskinesis

Measurements	Method/Number of studies	Intra-rater Reliability (K)*	Inter-rater Reliability (K)*
1. 4-Type Classification	1. Video recording/1	1. -	1. Poor/44
	2. Live observation/1	2. Moderate/35	2. Moderate/35
2. Yes/No method for presence of Scapular Dyskinesis	1. Video recording/1	1. –	1. Substantial/32
	2. Live observation/3	2. Moderate/35	2. Moderate-Fair/35
3. Scapular Dyskinesis Test	1. Live observation/1	1. Substantial/35	1. -

K: kappa coefficient

^*K interpretation as poor (0.00), slight (0.00–0.20), fair 0.21–0.40, moderate (0.41–0.60), substantial (0.61–0.80) and almost perfect agreement (>0.80) based on Landis and Koch (26)

Discussion

The aim of this systematic review was to evaluate the quality and the content of the studies that examined the intra-rater and inter-rater reliabilities of the methods used for the assessment of SD in asymptomatic subjects. From the literature search, we identified 14 studies that assessed the reliability of 5 quantitative and 3 qualitative assessment methods of SD (Tables 4 and 5). Most of the studies that we reviewed showed moderate risk of bias based on the QAREL and thus methodological quality was considered as moderate.

From the aforementioned studies that we reviewed, only seven studies used evaluation methods that aimed to classify the scapula as dyskinetic or non-dyskinetic (22, 24–27, 29, 31). From the evaluation that was performed in each study, it was evident that the studies that aimed to classify scapulae as dyskinetic or non-dyskinetic showed a great variation of reliability values, which might be a result of the study designs and quality rather than the quality of the clinical tests (22, 24–27, 29, 31).

Qualitative methods

On the basis of results (Table 5), the Yes/No, the 4-Type Classification, and the SD Test, which were the main methods used with several modifications for the assessment of SD, showed varied reliability results. The values of these methods varied from 0.20 to 0.97 for the inter-rater reliability and from 0.67 to 1 for the intra-rater reliability. Surprisingly, only two studies evaluated the intra-rater reliability of any qualitative assessment method (25, 27). None of the studies demonstrated high reliability values for any of the methods used in more than one study.

This is not surprising since previous studies have shown that, at best, clinicians are likely to see shoulder asymmetries approximately 60% of the time (36). Furthermore, previous research suggests that visual observation of SD provides clinicians with a more practical approach when assessing patients in clinical practice (37). However, in many occasions, participants will display a combination of SD patterns and predominance of one type of SD is not evident (37).

Although there are problems with the reliability of the qualitative assessment, SD evaluation remains a qualitative assessment, at least in the first observation by a clinician, and this part of the assessment should not be excluded. Moreover, previous research in symptomatic participants suggests that the Yes/No method of SD assessment is much more reliable when compared with the 4-Type Classification (34). This suggestion may strengthen the clinical utility of qualitative assessment of SD. Thus, based on the aforementioned suggestions and the reliability values found in the studies that we reviewed, it may be more useful for clinician to use qualitative assessment such as the Yes/No method in combination with quantitative methods, in order to properly screen and identify improvements during follow up in patients with SD.

Quantitative methods

Most of the studies that assessed SD (Table 4) with quantitative measurement methods found moderate to excellent ICC values. This indicates that quantitative measurement of SD is probably a more reliable way of estimating SD. However, an important aspect of the quantitative methods used in these studies was that all measurements were performed in static positions. Thus, proper assessment of the three-dimensional movement of the scapula during shoulder motion was not possible to be performed by any method. Previous research findings in symptomatic subjects have also identified that all quantitative assessment methods of SD are limited to static positions and should not be used uniquely in clinical practice (11).

Furthermore, when looking at the methodological quality of the studies, it seems that there are substantial flaws such as lack of raters’ training, raters’ clinical experience and professional background, single measurements, and lack of blinding procedures (Table 3). In contrast with the studies that used qualitative methods, which were all of moderate methodological quality, among the studies that used quantitative methods, two (23, 31) were classified as low quality, based on the QAREL. This should be taken into consideration when interpreting the results of the reliability studies and deciding on the superiority of the quantitative methods.

Establishing reliability of SD diagnosis test in asymptomatic subjects is way more important for two reasons: 1) in symptomatic subjects, the inclusion of symptom alteration tests such as the scapular assistance test may assist the clinician in making a proper diagnosis for the influence of SD in patient presentation. Thus, it can guide the clinician more efficiently in designing a proper treatment plan. However, in asymptomatic subjects with SD, symptom alteration tests are not applicable and thus the clinicians rely only on scapular measurement differences to identify people at risk for developing shoulder symptoms. 2) Since asymptomatic SD may increase the risk of future shoulder pain (4), the overall diagnosis and management of SD before symptom initiation may inform clinical practice and aid clinicians in the development of a preventative exercise programme. Thus, more high-quality studies are needed in this area, in order to be able to provide clear suggestions for clinical practice.

Overall assessment of studies

The statistical procedures used in the studies should be questioned as a priori sample size was calculated only in one study (19). This has a direct impact on the statistical power of the studies and the precision of the results (38). Calculating a priori sample size for each study is highly important for all reliability studies (39). Furthermore, the statistical procedures that were used to assess reliability were not exhaustive in the aforementioned studies. For example, only one study used Bland and Altman plots to investigate possible systematic biases and identify the limits of agreement (19). The Bland and Altman analysis is important in reliability studies, in order to identify possible systematic bias in the measurements (40, 41).

ICCs, which were used in the studies that we reviewed, are appropriate for presenting the results. However, they could not detect systematic errors. For example, raters may consistently use a method inappropriately and still obtain high reliability values (42, 43). Additionally, this problem can be accentuated when only two raters are recruited (43). Furthermore, most studies did not apply proper randomization procedures to limit bias in the recorded measurements (23, 28–31). Thus, it is suggested that proper methodological procedures should be taken into account when undertaking similar reliability studies in order to ensure high methodological quality and eliminate bias.

Raters’ educational level and experience

From the selected studies, only three studies recruited raters with postgraduate education in musculoskeletal physiotherapy (19, 20, 27) and only one study recruited raters with many years of experience (44). Some of the studies did not provide any relevant information in regards to the raters’ background (23, 26, 28, 30–32). Education and experience are important indicators of increased agreement and it should be reported in these kinds of reliability studies (42). However, this was not the case in the studies selected for this review and this relationship was not detected. It is suggested that homogenous groups of raters should be recruited in reliability studies, in terms of experience and training, in order to avoid jeopardizing the quality of the study.

Limitations

For this review, we used a number of synonyms to detect all the desired studies. However, there is always the potential that some synonyms were missed and that we did not find all desired studies. In order to minimize this limitation, all reference lists were searched from the studies that we selected and from previous similar systematic reviews but we did not find more suitable studies. Another limitation is the language bias that may have occurred since we selected only English articles. However, we did not find any relevant articles with an English title and non-English text. Thus, only relevant studies with a non-English title and non-English text may have been missed.

The agreement among the reviewers for the selection of the studies based on the titles and then the full texts was high. However, when deciding on study selection, the recruitment of two pairs of independent reviewers may have reduced the chance for random error and selection bias (16). Additionally, the inclusion of studies that showed high risk of bias may have improperly estimated the value of some tests and may have affected the conclusions of our review. However, the lack of published studies in this field did not allow for these exclusions to be made.

Important points

In this review, we expected to find better results in terms of reliability since the subjects were asymptomatic and thus they had no restrictions in performing any of the instructions that they were given by the examiners. Previous studies have shown that subjects with shoulder pathology demonstrate increased scapular positional differences (45, 46). Thus, measuring in this symptomatic population should have been more challenging. However, the results of the overall reliability both for the qualitative and the quantitative methods were not different in the asymptomatic subjects.

Additionally, it is important to consider that most quantitative studies have reported excellent reliability values in this review according to Fleiss’ criteria (>0.75) (Table 3) (17). The superiority of the quantitative methods, in terms of reliability, can be explained from the fact that qualitative methods largely depend on the clinical experience of the raters whereas this is not the case with the quantitative methods, unless training is provided. However, it is important to note that, in clinical practice, only tests that score above 0.90 should be taken into consideration and responsibly inform our clinical practice (16, 47). Thus, it is suggested that even quantitative methods should be used in combination and not exclusively when assessing SD.

Conclusion

A number of studies have examined the reliability of qualitative methods in the assessment of SD in asymptomatic subjects. These tests did not show high reliability values and hence their use in clinical practice is not suggested. In addition, some studies have examined the reliability of quantitative methods in the estimation of SD in asymptomatic subjects. These methods showed higher reliability values. However, the reliability values that these studies give are questionable due to their methodological flaws and clinicians should be aware of that. Further research with high-quality studies is needed, for safe clinical applications.

HIGHLIGHTS.

• The inter-rater and intra-rater reliability of quantitative methods for the assessment of SD seems to be superior to qualitative methods.

• This study found lack of high-quality studies evaluating the reliability of SD assessment.

• Several methodological flaws were identified in the studies selected in this review and should be carefully considered in similar reliability studies in the future.

• It is suggested that, even the more reliable quantitative methods should be used in combination with the qualitative methods and not exclusively, when assessing SD.