Inter-examiner reliability of diplomats in the mechanical diagnosis and therapy system in assessing patients with shoulder pain

Abstract

Objective:

To investigate the inter-examiner reliability of Mechanical Diagnosis and Therapy (MDT)-trained diplomats in classifying patients with shoulder disorders. The MDT system has demonstrated acceptable reliability when used in patients with spinal disorders; however, little is known about its utility when used for appendicular conditions.

Methods:

Fifty-four clinical scenarios were created by a group of 11 MDT diploma holders based on their clinical experience with patients with shoulder pain. The vignettes were made anonymous, and their clinical diagnoses sections were left blank. The vignettes were sent to a second group of six international McKenzie Institute diploma holders who were asked to classify each vignette according to the MDT categories for upper extremity. Inter-examiner agreement was evaluated with kappa statistics.

Results:

There was ‘very good’ agreement among the six MDT diplomats for classifying the McKenzie syndromes in patients with shoulder pain (kappa = 0.90, SE = 0.018). The raw overall level of multi-rater agreement among the six clinicians in classifying the vignettes was 96%. After accounting for the actual MDT category for each vignette, kappa and the raw overall level of agreement decreased negligibly (0.89 and 95%, respectively).

Discussion:

Using clinical vignettes, the McKenzie system of MDT has very good reliability in classifying patients with shoulder pain. As an alternative, future reliability studies could use real patients instead of written vignettes.

Introduction

It is accepted that an accurate diagnosis is an important prerequisite for developing an effective treatment strategy.¹ Interventions are ideally targeted to a specific diagnosis; hence, an incorrect diagnosis may well lead to inappropriate management of a pathological condition and an increased likelihood for a poor treatment outcome. If the procedures and tests used in an examination are not reliable and valid, an incorrect diagnosis is the likely sequela.² A key to accurate diagnosis is the reliability of the diagnostic tests being used by the clinician. Inter-rater reliability has been defined as ‘the extent to which examiners, using the same test on the same patients, agree on the results of the test’.³

The literature has highlighted the fact that establishing an accurate diagnosis in patients with shoulder pain is problematic.^4–8 Many commonly used examination procedures and orthopedic special tests for the shoulder lack reliability^2,8 and validity.^4,9,10 Additionally, there is a growing body of evidence suggesting that the findings from imaging tests, such as US, CT, or MRI, should not be relied upon entirely for clinical decision making, as the incidence of pathological findings in clinically asymptomatic shoulders is significant.^11–14 This clearly compromises the clinician’s ability to make an accurate patho-anatomical diagnosis. As a result, there have been calls for^6,8 and the development of^7,8,15,16 non-patho-anatomic shoulder subgroups so that interventions can be more accurately matched to the patients who are classified within a given subgroup.

One widely used non-patho-anatomical classification scheme is the Mechanical Diagnosis and Therapy (MDT) system, which was initially introduced by Robin McKenzie in 1981 as a new approach to the classification and management of patients with low back pain.¹⁷ He later described application of this system to the cervical and thoracic spines.¹⁸ The MDT system classifies patient presentations based on analyzing the symptomatic and mechanical effect of different loading strategies, positions, and postures.¹⁹ Each MDT syndrome requires its own particular management approach.

A series of systematic reviews support the efficacy of the MDT system in the management of acute and chronic low back pain.^20–27 The MDT system for patients with spinal disorders has also demonstrated acceptable reliability,^28–34 as well as diagnostic and prognostic validity,^35–45 among experienced physiotherapists. McKenzie proposed that this system of diagnosis and treatment could also be applied to extremity disorders.¹⁷ McKenzie’s book on the application of MDT to human extremities⁴⁶ contains a detailed explanation of its clinical application to patients with peripheral joint disorders.

According to McKenzie, patients with extremity disorders can be classified into the following four syndromes.⁴⁶

• Derangement syndrome: identified by a rapid response to a direction-specific loading strategy, known as the directional preference. A lasting improvement in symptoms, range of motion, and enhanced function will be achieved once the directional preference has been established and utilized.

• Articular dysfunction: distinguished by intermittent and consistent pain only produced at a diminished end range with a slower response to specific tissue loading strategy.

• Contractile dysfunction: distinguished by intermittent pain consistently produced, but this time only when the musculo-tendinous unit is loaded, for instance, with an isometric contraction against resistance.

• Postural syndrome: intermittent pain only produced by prolonged postures that, once avoided, result in a return to a normal pain-free state. The remainder of the physical examination is normal.

• Other: patients who cannot be classified under any of the mechanical syndromes. Examples include trauma, articular structurally compromised, recent surgery, and chronic pain syndrome (Appendix).

These categories allow for the full spectrum of musculoskeletal presentations to be classified within the MDT system.

The use of MDT in the extremities has not been investigated to the same extent as it has been in the spine. Currently, the scientific literature in this area has been limited to individual case studies, which generally reveal a very good treatment response.^47–54 One survey of the prevalence, classification, and preferred loading strategies for the use of the MDT system in the extremities has also been published; demonstrating that 30 participating therapists were able to use the system to successfully classify all patients with an extremity problem.¹⁶ A more recent pilot RCT study conducted on patients with rotator cuff tendinopathy revealed comparable treatment outcomes in these patients using the MDT-based, self-managed, loaded exercise program versus the usual physiotherapy program.⁵⁵

The MDT classification system, when used on patients with spinal disorders, has demonstrated acceptable inter-examiner reliability among trained physiotherapists.^28–34 In the extremities, Kelly et al.⁵⁶ conducted a pilot study with 11 patient vignettes and three MDT-trained practitioners, including two credentialed and one diploma therapists. May and Ross¹⁹ continued with a follow-up study using 25 patient vignettes and 93 MDT diploma therapists. However, the inter-examiner reliability of the MDT classification system for the extremities has not been investigated in any samples comprised exclusively of patients with shoulder disorders. The previous two studies included patients with variety of extremity joint disorders, with no secondary analysis exploring inter-examiner reliability of the MDT system in any individual joint such as the shoulder. Only 7 out of 25 vignettes of the larger reliability study¹⁹ were shoulder cases (correspondence from study author). The aim of our study was to investigate the inter-examiner reliability of MDT-trained diploma therapists when classifying patients with shoulder disorders.

Methods

Design and procedure

This was a two-phase study. In phase 1, a convenience sample of 11 MDT diploma holders were recruited from a publicly available list of MDT practitioners registered with the McKenzie Institute International who practice in Canada or the United States. They were asked to create 54 anonymous written clinical vignettes based upon findings from the initial assessment of previously treated patients with shoulder disorders. They were directed to document the patients’ age in years, but ‘not transfer’ any identifying information regarding their patients including their name, address, telephone, and date of birth in order to maintain anonymity of the patients. The number of vignettes created for each sub-classification was 11 derangements, 11 articular dysfunctions, 11 contractile dysfunctions, 11 ‘spinal’ category, which represents patients with shoulder pain deemed to be originating from the cervical spine, and 10 ‘other’ MDT categories. Due to a very low incidence of ‘postural syndrome’ in patients with extremity disorders¹⁶ a ‘spinal’ category was used as the fifth MDT subgroup for this study and the ‘postural’ subgroup was assigned to the ‘other’ categories. The ‘spinal’ category included patients with complaints of shoulder pain who were determined to have pain originating from the neck; this is commonly seen clinically and has been extensively reported in the literature.^46,52

The standard McKenzie extremity assessment form routinely utilized by MDT practitioners was used to structure the clinical findings of the vignettes. In the event that a clinician did not have any recent patients that would fit one specific MDT sub-classification, the vignette was created based on the presentation of patients in that subgroup from the past. A representative group of five vignettes are uploaded to the JMMT website. Ethical approval for the study was obtained from the Health Sciences Research Ethics Board of Western University.

In phase two, the 54 vignettes from phase 1 were used to examine inter-rater reliability. These vignettes were sent to six MDT diploma holders who practice in Canada and the United States who had no involvement with the first phase of the study. They were also recruited from the publicly available list of MDT practitioners registered with the McKenzie Institute International. Following informed consent, an explanation of the study was provided and the clinicians were asked to review each vignette and identify the MDT classification for each vignette from the following five subgroups: derangement, articular dysfunction, contractile dysfunction, spinal and other. All six clinicians were blinded to the MDT classification represented by each vignette.

Sample size

A confidence interval (CI) approach for sample size estimation of kappa was used.⁵⁷ This method allows researchers to design their inter-examiner agreement study with any number of outcomes and any number of examiners using a pre-specified level of precision in the estimation of kappa.⁵⁷ Assuming a preliminary estimate of kappa = 0.7, with a 95% CI of 0.2, we determined that 54 vignettes were needed for six examiners (MDT diploma holders).

Analysis

The kappa coefficient, standard error (SE), and raw percentage of agreement were calculated across the six participating physiotherapists. Data were analyzed using the MAGREE macro in Statistical Analysis System (SAS) version 9.3 for Windows. Kappa values were interpreted using the traditional thresholds of: less than 0.40 = poor; 0.41–0.60 = moderate; 0.61–0.80 = good; and 0.81–1.00 = very good.⁵⁸

Results

Five physical therapists and one chiropractor who solely apply the MDT method when treating their patients with extremity disorders were recruited to classify the clinical vignettes. Demographic information provided by the participating practitioners is shown in Table 1. Distribution of the MDT classification ratings of the clinicians, in addition to the true classification of the vignettes is shown in Table 2.

Table 1. Demographic information of the participating practitioners.

Variables	Distribution
Number of raters	6
Age, mean (SD) (years)	51 (8.6)
Gender	Female: 2
	Male: 4
Years in practice, mean (SD)	25.7 (8)
Years since MDT diploma, mean (SD)	16 (4)
Proportion of extremity patients in caseload (n)	<25%: 2
	25%–50%: 4
Practice setting (n)	Private: 4
	Hospital outpatient: 1
	Specialty clinic: 1

MDT: Mechanical Diagnosis and Therapy, SD: standard deviation.

Table 2. Frequency (%) of vignette classification by rater.

MDT classification	Actual classification (%)	Rater
		1 (%)	2 (%)	3 (%)	4 (%)	5 (%)	6 (%)
Derangement	11 (20)	14 (26)	12 (22)	13 (24)	11 (20)	13 (24)	13 (24)
Articular dysfunction	11 (20)	11 (20)	9 (16)	10 (19)	11 (20)	10 (19)	10 (19)
Contractile dysfunction	11 (20)	11 (20)	11 (20)	11 (20)	10 (20)	12 (22)	11 (20)
Spinal	11 (20)	12 (22)	12 (22)	12 (22)	11 (20)	12 (22)	11 (20)
Other	10 (20)	6 (12)	10 (18)	8 (15)	11 (20)	7 (13)	9 (17)
Total	54 (100)	54 (100)	54 (100)	54 (100)	54 (100)	54 (100)	54 (100)

MDT: Mechanical Diagnosis and Therapy.

There was consensus among all six raters on the vignettes’ classification in 78% of the vignettes (42 out of 54). The raw overall level of multi-rater agreement among the six clinicians was 96%. The corresponding kappa value was 0.90 (SE = 0.018). The highest level of chance-adjusted agreement was for the spinal category with kappa = 0.96; the lowest level was for the ‘other’ category with kappa = 0.80. By factoring in the true diagnoses of the vignettes in our analysis, the raw agreement and kappa were 95% and 0.89, respectively. Values of agreement for each one of the MDT classifications are shown in Tables 3 and 4.

Table 3. Agreement findings by MDT classification across raters.

MDT classification	Raw agreement (%)	Kappa	SE
Derangement	95	0.90	0.035
Articular dysfunction	97	0.90	0.035
Contractile dysfunction	97	0.92	0.035
Spinal	97	0.96	0.035
Other	94	0.80	0.035
Overall agreement	96	0.90	0.018

MDT: Mechanical Diagnosis and Therapy; SE: standard error.

Table 4. Agreement by MDT classification across raters and the actual MDT vignette classification.

MDT classification	Raw agreement (%)	Kappa	SE
Derangement	93	0.88	0.030
Articular dysfunction	96	0.87	0.030
Contractile dysfunction	97	0.93	0.030
Spinal	96	0.96	0.030
Other	93	0.77	0.030
Overall agreement	95	0.89	0.015

MDT: Mechanical Diagnosis and Therapy; SE: standard error.

Discussion

To our knowledge, this study is the first to address inter-examiner reliability of the MDT system exclusively in patients with shoulder pain. The results support the findings of previous reliability studies on application of the MDT in extremities.^50,51 The principal findings of our study suggest that experienced McKenzie practitioners have a ‘very good’ level of inter-examiner agreement when classifying patients with shoulder pain using the MDT system. The highest level of agreement was for the ‘spinal’ category with kappa = 0.96, and the lowest level of agreement was for the ‘other’ MDT categories with kappa = 0.80. The relatively lower level of agreement for the ‘other’ category was anticipated because multiple subcategories are included in this MDT classification. This makes diagnosis more challenging particularly when the decision is solely based on information collected in the initial assessment. A relatively higher level of agreement for the ‘spinal’ category may be due to the presence of more identifying symptoms, such as paresthesia, reported in some of the vignettes, and also the presence of, in some cases, a relatively quick response in the shoulder pain level of these patients by addressing their cervical spine. By including the actual classification of the vignettes in our analysis, as shown in Table 4, there is only a slight decline in both percent agreement and the kappa value. This slight decline could be due to the presence of insufficient clinical information provided in the vignettes, as these were based only on the clinical information gathered in the initial assessment session.

The results of our study on the shoulder generally reinforce the findings of previous reliability studies in the spine and the extremities, suggesting that the MDT system is a reliable method to classify patients with musculoskeletal shoulder disorders. Multiple studies have been conducted on inter-examiner reliability of the MDT system in patients with spinal disorders demonstrating an acceptable level of reliability among MDT practitioners in classifying their patients.^28–34 For instance, Razmjou et al.²⁸ and Kilpikoski et al.³⁰ reported good inter-examiner reliability between two MDT-trained therapists in classifying patients with low back pain into MDT classifications (kappa = 0.7). In another type of study using video and written clinical vignettes, Werneke et al.³⁴ reported substantial to almost perfect inter-rater agreement in identifying treatment approaches for neck and low back disorders among MDT-trained therapists. There are only two studies addressing inter-examiner reliability of the MDT system for patients with extremity disorders.^19,56 These two studies included a pilot study with 11 clinical vignettes⁵⁶ and three therapists, and a follow-up study with 25 clinical vignettes and 93 MDT diploma holders.¹⁹ The pilot study showed ‘good’ agreement with kappa a value of 0.7, and the follow-up study revealed ‘very good’ agreement with kappa value of 0.83 (95% CI, 0.68–0.98). The clinical vignettes used for these studies were based on patients with both upper and lower extremity disorders. There was little difference between the reliability in upper (kappa = 0.85) and lower extremity (kappa = 0.80) cases.¹⁹

The major limitation of the current study was that only practitioners with an MDT diploma, the highest level of MDT training, were included. This limits the generalizability of the findings of this study, as the inter-rater agreement among clinicians without this level of training may not be as high. Therefore, this study is a first step toward evaluating the reliability of using the MDT system to classify patients with shoulder pain. Future studies should include practitioners with different levels of training and experience so that the agreement findings are generalizable to a broader group of practitioners. Another limitation of this study was using written vignettes instead of having actual patients. The major concern in this regard, as stated by Werneke et al,³⁴ is the purification of the intervention being expressed in the vignettes, which may not represent all aspects of clinical practice, making the diagnosis easier for the raters and inflating the calculated kappa value. One strength of using written vignettes is that this approach eliminates the potential error created by inconsistent patient presentations between raters. As an alternative, future studies could consider the use of real patients instead of written vignettes in order to further establish reliability of the MDT in extremities.

Appendix

Appendix: Alternative diagnostic sub-classifications in the Mechanical Diagnosis and Therapy system comprising the ‘Other’ category.

Category	Definitions	Criteria - Essential (Common)	Examples (where necessary)
Trauma/recovering trauma	Recent trauma associated with onset of symptoms	Recent trauma associated with onset of constant symptoms/recent trauma associated with onset of symptoms in previous 6 weeks now intermittent and improving
Red flags	Fracture to bone	History of significant traumaLoss of functionAll movement make worse
	Malignant tumor	(Age>55)(History of cancer)(Unexplained weight loss)Progressive, non-mechanical pain, not relieved by rest
Inflammatory	Inflammatory arthropathy	ConstantExcessive movements exacerbate symptoms	RA, some stages of OA
Chronic pain syndrome	Pain-generating mechanism influenced by psychosocial factors or neurophysiological changes	Persistent widespread painAggravation with all activityDisproportionate pain response to mechanical stimuliInappropriate beliefs and attitudes about pain	Regional pain syndromes
Post-surgery	Presentation relates to recent surgery	Recent surgery (Local post-surgery protocols may apply)
Mechanically inconclusive	Unknown joint pathology	Inconsistent response to loading strategies (Inconsistent pattern of obstruction to movement)
Peripheral nerve entrapment	Peripheral nerve entrapment	No spinal symptomsLocal paresthesia/anesthesia (local muscle weakness)	Carpal tunnel syndrome, meralgia paraesthetica
Articular structurally compromised	Soft tissue and/or bony changes compromising joint integrity	Mechanical symptoms (ROM restricted, clunking, locking, catching)(Sensation of instability)Long history of symptoms or history of traumaIrreversible with conservative care	Late stage OA, dislocation, labral tear, cruciate ligament rupture, irreducible meniscal tear
Soft tissue disease process	A fibroblastic or degenerative disease process affecting inert soft tissue with unknown or disputed etiology	Each disease process has a unique clinical presentation, natural history, and varying degrees of efficacy to a variety of interventions	Frozen shoulder, Dupuytren’s, plantar fascia syndrome
Vascular	Symptoms induced by poor blood supply due to pressure increase in a closed anatomical space	Poorly localized severe ache(Commonly induced by exercise or trauma)(Paresthesia in field of local cutaneous nerve)(Muscle feels tight or full)	Compartment syndrome

Source: May S, Rosedale R. A survey of the McKenzie classification system in the extremities: prevalence of mechanical syndromes and preferred loading strategies. Phys. Ther. 2012 Jul 26 [Epub ahead of print].