Reliability and accuracy of an expert physical therapist as a reference standard for a manual therapy joint mobilization trial

ABSTRACT

Objective: Previous studies on learning joint mobilization techniques have used expert practitioners as the reference standard as there is no current evidence on what ideal forces would be for effective mobilizations. However, none of these trials have documented the reliability or accuracy of the reference standard. Therefore, the purpose of this study was to report both the reliability and accuracy of an expert physical therapist (PT) acting as a reference standard for a manual therapy joint mobilization trial.

Methods: A secondary analysis was performed using data from a published randomized, controlled, crossover study. The mobilization technique studied was the central posterior to anterior (PA) joint mobilization of the L3 vertebra. Reliability and accuracy data for the reference standard were collected over four time periods spanning 16 weeks.

Results: Intrarater reliability of the expert PT for R1 and R2 joint forces was excellent (R1 Force ICC_3,3 0.95, 95%CI 0.76–0.99 and R2 Force ICC_3,3 0.90, 95%CI 0.49–0.99). Additionally, the expert PT was 92.3% accurate (mean % error±SD, 7.7 ± 5.5) when finding Grade III mean peak mobilization force and 85.1% accurate (mean % error±SD, 14.9 ± 8.3) when finding Grade IV mean peak mobilization force. Finally, correlations between actual applied forces and computed ideal forces were excellent (Pearson r 0.79–0.92, n = 24, P < 0.01 for all correlations).

Discussion: The expert PT in this manual therapy joint mobilization trial showed excellent reliability and accuracy as the reference standard. The study supports the use of implementing quantitative feedback devices into the teaching of joint mobilization when a reliable and accurate reference standard has been identified.

Level of Evidence: 2b.

Introduction

Manual therapy techniques, such as joint mobilization and manipulation, are widely used for the evaluation and treatment of various musculoskeletal impairments [1,2]. In the Australian approach to manual physical therapy, four main grades of mobilization have been defined and are based on the amount of resistance and magnitude of movement applied [3–6]. Grades I and II, used primarily for pain modulation, are designated as movements before the first onset of resistance to movement, or R1. Grades III and IV are used primarily to increase joint motion respecting pain and are movements into resistance between the first onset of resistance and the end of the joint resistance movement, or R2. While Grade III mobilizations utilize large amplitude motions, Grade IV uses small amplitude motions. These grades have been further defined to include pluses and minuses. Minuses are used to indicate forces less than 50% of R1-R2 (e.g. Grade III- or IV- would represent 25% of the resistance range) and pluses are used to indicate forces above 50% of R1-R2 (e.g. Grade III+ or IV+ would represent 75% of the resistance range). When no plus or minus is used (e.g. Grade III or IV), it indicates 50% of R1-R2 [6–8].

While these parameters have been defined and utilized in practice for years, there is no current evidence on what ideal forces would be for effective joint mobilization. These techniques are commonly taught in both entry-level and post-professional physical therapy education [9–13]. Traditional teaching methods for learning manual therapy skills have consisted of an expert instructor demonstrating techniques followed by student practice sessions [12,14,15]. This method places the expert in a de facto role as a reference standard in the classroom without providing evidence that their skills are either accurate or reliable. In addition, previous studies on joint mobilization have used expert practitioners as the reference standard [16–24]. However, none of these trials have documented the reliability or accuracy of the reference standard. Therefore, the purpose of this study was to report both the reliability and accuracy of an expert physical therapist (PT) acting as a reference standard for a manual therapy joint mobilization trial.

Methods

A secondary analysis was performed using data from a published randomized, controlled, crossover study [25]. A board-certified, fellowship-trained manual physical therapist with 20 years of experience served as the reference standard for the study. The expert PT was blinded to all force measurements throughout the duration of the study. The mobilization technique studied was the central posterior to anterior (PA) joint mobilization of the L3 vertebra using the pisiform grip (See Figure 1). This spinal level was chosen due to the orientation of the L3 vertebra being such that direct vertical forces can be applied. A loadpad force monitoring device was used to record all force production data (See Figure 2). The loadpad was placed between the therapist’s pisiform grip and the patient’s skin over the L3 spinous process. Twenty-four mock patients were recruited from faculty, staff, and doctoral students of an accredited physical therapy program. Demographic information on all mock patients can be found in the original crossover study. In that study, 24 participants were divided into two groups and were tested on 8 force parameters of the central PA lumbar spine mobilization technique: R1 force, R2 force, Grade III and Grade IV mean peak force (MPF), amplitude, and frequency. One group then practiced using a device that provided feedback and the other group practiced without device feedback. Participants were tested on all force parameters at four time periods: baseline, 5 weeks, 11 weeks, and 16 weeks. The crossover period occurred after week 5 and a washout period of 5 weeks was given. The type of feedback each group received (with or without the device) was then switched. Student performance for each force variable was used to compare groups and was defined as the difference between the force parameter applied by a student and that applied by an expert physical therapist. The Institutional Review Board of the University of the Incarnate Word, San Antonio, Texas, USA approved the original study and all participants gave informed consent, and their privacy and rights were protected.

Figure 1.

Central PA lumbar mobilization technique using the pisiform grip

Figure 2.

Loadpad force monitoring device (10.5 × 5 cm; from Novel Electronics Inc., St. Paul, MN)

Data analysis

Reliability for finding R1 and R2 was calculated using the Intraclass correlation coefficient (ICC) model 3, form 3. The ICC_3,3 is used when each subject is assessed by the same rater and the average of 3 measurements is used. Response stability of all ICC scores was calculated using the standard error of measurement at the 95% level of confidence. Accuracy was computed as the complement of the mean percent error comparing actual applied mean peak force by the therapist to the computed ideal mean peak force by the device for Grade III and IV joint mobilizations. For the study, Grade III was defined as large amplitude movements between 25% and 50% into the resistance range. Grade IV was defined as small amplitude movements close to 50% into the resistance range. Validity coefficients were also calculated using Pearson product moment correlations comparing the computed ideal mean peak force for Grade III and IV (mean force between R1 and R2) with the actual applied mean peak force performed by the therapist for Grade III and Grade IV, respectively. Statistical analyses were performed using SPSS version 26.0 (SPSS Inc, Chicago, IL).

Results

The intrarater reliability of the expert PT finding both R1 and R2 joint forces for the central PA lumbar spine mobilization technique was excellent (R1 Force ICC_3,3 0.95, 95%CI 0.76–0.99 and R2 Force ICC_3,3 0.90, 95%CI 0.49–0.99). Additionally, the expert PT was 92.3% accurate (mean % error±SD, 7.7 ± 5.5) when finding Grade III mean peak mobilization forces and 85.1% accurate (mean % error±SD, 14.9 ± 8.3) when finding Grade IV mean peak mobilization forces. Tables 1 and 2 contain the percent error for Grade III and Grade IV mean peak forces, respectively, at all four measurement times. Finally, the validity coefficients demonstrated excellent correlations between actual applied forces and computed ideal forces (Pearson r 0.79–0.92, n = 24, P < 0.01 for all correlations). Figures 3 and 4 contain scatterplots and validity coefficients for each time frame and each mobilization grade.

Table 1.

Percent Error for Grade III Lumbar Spine Central PA Mobilization Technique

Data Collection Timeframe	Mean±SD*	SEM	Minimum*	Maximum*
Baseline	9.28 ± 6.41	1.33	0.07	21.9
Week 5	6.62 ± 3.91	0.82	0.32	14.6
Week 11	6.62 ± 4.63	0.97	0.38	15.3
Week 16	8.32 ± 6.57	1.37	0.42	26.7

Abbreviations: PA, Posterior to Anterior; SD, Standard Deviation; SEM, Standard Error of the Mean.

*Values are % Error between Mean R1R2 Force and Grade III Mean Peak Force [N].

Table 2.

Percent Error of Grade IV Lumbar Spine Central PA Mobilization Technique

Data Collection Timeframe	Mean±SD*	SEM	Minimum*	Maximum*
Baseline	18.4 ± 8.24	1.72	0.48	33.9
Week 5	12.2 ± 5.87	1.22	0.61	22.1
Week 11	12.0 ± 6.47	1.35	2.61	26.2
Week 16	17.1 ± 10.3	2.14	1.44	37.5

Abbreviations: PA, Posterior to Anterior; SD, Standard Deviation; SEM, Standard Error of the Mean.

*Values are % Error between Mean R1R2 Force and Grade IV Mean Peak Force [N].

Figure 3.

Scatterplot and validity coefficients for Grade III ideal vs. actual mean peak mobilization forces

Figure 4.

Scatterplot and validity coefficients for Grade IV ideal vs. actual mean peak mobilization forces

Discussion

In traditional physical therapy education, students observe an expert instructor perform required manual therapy skills followed by extensive practice on peers, faculty and eventually patients [12,14,20,26,27]. In this model of teaching, limited objective feedback may be given to students learning joint mobilization skills. Instead, more subjective feedback about applied mobilization forces may be given due to time contraints or large faculty-student ratios. This type of feedback is less accurate when compared with quantitative, objective force measurements performed in real-time [14,15,20,25,28]. Additionally, feedback given in settings with large faculty-student ratios and by instructors with high variability in force production can negatively impact motor skill mastery [22,29,30]. Fortunately, research has shown that when students are provided with objective feedback, they more consistently and accurately apply the desired manual therapy forces [18,19,21,25,31–37]. Joint force measurement devices that provide quantitative feedback allow an expert clinician to share objective force parameters with students in a classroom setting. Flexible force monitoring devices that allow for direct force measurement between the therapist’s hands and the actual joint being mobilized have been shown to produce reliable and accurate measurements [25,32,38–40]. When students are able to directly compare their performance on a joint mobilization skill to an expert standard, they can apply more accurate forces during manual therapy training [25]. However, the expert should be able to demonstrate consistent and accurate application of forces while demonstrating techniques in order to serve as a reference standard. The expert PT in this manual therapy joint mobilization trial showed excellent reliability and accuracy acting as the reference standard.

Limitations

There are several limitations to this study. First, only one therapist at a single institution was studied for reliability and accuracy. Additionally, the therapist was board-certified and fellowshipped-trained with 20 years of experience in the Australian manual therapy approach. Therefore, these results may not be generalizable to other therapists with different training and experience. Second, there was no external validation of R1 and R2 forces. Clinically, these forces are subjectively determined by the therapist based on a theorectical definition of tissue resistance to movement. This tissue resistance is also affected by multiple factors such as temperature, time of day, joint position, stage of healing or health, previous actvitiy level, and any mechanical stress from previous mobilization. These are challenges when working with human subjects in clinical or educational settings. Third, the order of force parameter testing was not randomized. This may have biased the results as Grade IV force parameters were always performed after Grade III. Finally, the device used in this study is able to capture force magnitude, frequency, amplitude, timing, and rate of force production. Other force quantification devices may not provide the same results as found in this study.

Future research

Future research exploring the inter- and intra-clinican reliability of joint mobilization forces at various joints could improve understanding of the discrepancies in force production between the expert PT and novice practitioners. Comparisons of reliability and accuracy between experimental groups and an expert should only be made once the reliability of the expert has been established. Future studies using expert practitioners as the reference standard should report reliability and accuracy statistics for each trial.

Conclusion

The expert PT in this manual therapy joint mobilization trial showed excellent reliability and accuracy as the reference standard. The study supports the use of implementing quantitative feedback devices into the teaching of joint mobilization when a reliable and accurate reference standard has been identified.

Biographies

Evan J. Petersen, PT, DSc, OCS, FAAOMPT is an Associate Professor at the University of the Incarnate Word, School of Physical Therapy where he teaches Patient/Client Management and Clinical Reasoning. Prior to his appointment at UIW, he was the Graduate Program Director of the Army-Baylor University Doctoral Fellowship in Orthopaedic Manual Physical Therapy. He has been a licensed physical therapist since 1999 and has practiced in outpatient orthopaedic clinics in the United States, Egypt, Germany, and Afghanistan. He is a board certified clinical specialist in Orthopaedics and a fellow in the American Academy of Orthopaedic Manual Physical Therapists.

Stephanie M. Thurmond, PT, DPT, ScD, COMT, CFMT is an Associate Professor for the School of Physical Therapy at Bowling Green State University. She serves as the Program Director for the developing Doctor of Physical Therapy program. Prior to her appointment at BGSU, she was the Director of Curriculum at the University of the Incarnate Word, School of Physical Therapy. She received her Doctorate of Physical Therapy from Hardin-Simmons University in 2002 and her Doctorate of Science from Texas Tech Health Sciences Center in 2011. She has primarily worked as a clinician in outpatient orthopaedics and entered academia in 2013.

Catherine A. Shaw, PT, DPT, graduated with a Doctor of Physical Therapy degree from the University of the Incarnate Word, School of Physical Therapy in San Antonio, Texas.

Kelly N. Miller, PT, DPT, graduated with a Doctor of Physical Therapy degree from the University of the Incarnate Word, School of Physical Therapy in San Antonio, Texas. She practices in the outpatient orthopaedic setting at Peak Physical Therapy in Frisco, Texas.

Tommy W. Lee, PT, DPT, graduated with a Doctor of Physical Therapy degree from the University of the Incarnate Word, School of Physical Therapy in San Antonio, Texas. He practices in both acute care and outpatient orthopaedic settings at John Peter Smith Hospital in Ft. Worth Texas.

Jonathan A. Koborsi, PT, DPT, graduated with a Doctor of Physical Therapy degree from the University of the Incarnate Word, School of Physical Therapy in San Antonio, Texas. He practices in the outpatient orthopaedic setting at Memorial Village Physical Therapy in Houston, Texas.

Funding Statement

This study was funded in part by the University of the Incarnate Word (UIW), Office of Research and Graduate Studies. Additional donations in kind were provided by the UIW School of Physical Therapy to complete the study. This study was reviewed and accepted by the Institutional Review Board at the University of the Incarnate Word, San Antonio, TX.

Disclosure of interest

The authors report no conflict of interest.

Supplementary material

Supplemental data for this article can be accessed here.

VIDEO – JMMT 749-Petersen

https://www.youtube.com/watch?v=kTARHihKyL4

Slide 1

Our study is entitled “Reliability and Accuracy of an Expert Physical Therapist as a Reference Standard for a Manual Therapy Joint Mobilization Trial.” Faculty members and students from the University of the Incarnate Word and Bowling Green State University conducted this study.

Slide 2

Previous studies on learning joint mobilization techniques have used expert practitioners as the reference standard as there is no current evidence on what ideal forces would be for effective mobilizations. However, none of these studies have documented the reliability or accuracy of the reference standard. Therefore, the purpose of this study was to report both the reliability and accuracy of an expert physical therapist (PT) acting as a reference standard for a manual therapy joint mobilization trial.

Slide 3

The manual therapy joint mobilization trial was a randomized, controlled, crossover study published last year in the Journal of Manual and Manipulative Therapy. In that study, twenty-four 1st Year Doctor of Physical Therapy students were randomized into two groups. One group practiced a common lumbar spine mobilization technique with a device that provided quantitative feedback on the mobilization forces, while the other group practiced without it. Data were collected over 4 times periods spanning 16 weeks. The study found that real-time, objective feedback using a direct force measurement device improved learning for some aspects of lumbar spine joint mobilization by entry-level physical therapy students.

Slide 4

During that study, a board-certified, fellowship-trained manual physical therapist with 20 years of experience served as the reference standard. Student performance on all aspects of central posterior to anterior (PA) joint mobilization were compared against the expert’s forces. Data were collected on the expert’s reliability and accuracy for performing the joint mobilization, however, the full results were not published in the original study. Therefore, the purpose of the current study was to report these data and provide recommendations for future manual therapy studies using experts as the reference standard.

Slide 5

This video shows the expert PT demonstrating finding R1, or the first onset of resistance to joint movement; and R2, the end of the joint resistance to movement. Then the PT demonstrates performing a Grade III joint mobilization, or large amplitude movement between 25% and 50% into the resistance range. Then the PT demonstrates a Grade IV joint mobilization, or small amplitude movement right around 50% into the resistance range.

Slide 6

The output screen shows a visual representation of the forces produced by the therapist who was blinded to this feedback during the study. For each technique performed, the device recorded the R1 force; the R2 force; the Grade III mean peak force, frequency and amplitude; and the Grade IV mean peak force, frequency, and amplitude. These forces were used to calculate the reliability of finding R1 and R2 force and the accuracy for finding Grade III and Grade IV mean peak forces.

Slide 7

Reliability for finding R1 and R2 was calculated using the Intraclass correlation coefficient (ICC) model 3, form 3. Accuracy was computed as the complement of the mean percent error comparing actual applied mean peak force by the therapist to the computed ideal mean peak force by the device for Grade III and IV joint mobilizations.

Slide 8

The intrarater reliability of the expert PT finding both R1 and R2 joint forces for the central PA lumbar spine mobilization technique was excellent. Additionally, the expert PT was 92.3% accurate when finding Grade III mean peak mobilization forces and 85.1% accurate when finding Grade IV mean peak mobilization forces.

Slide 9

Validity coefficients were also calculated using Pearson product moment correlations comparing the computed ideal mean peak force for Grade III and IV (mean force between R1 and R2) with the actual applied mean peak force performed by the therapist for Grade III and Grade IV, respectively. Correlations between actual applied forces by the expert PT and computed ideal forces by the device itself were excellent. Pearson r correlations were between 0.79–0.92 for all 24 correlations across the 4 different timeframes for both Grade III and Grade IV mean peak force.

Slide 10

In conclusion, the expert PT in this manual therapy joint mobilization trial showed excellent reliability and accuracy as the reference standard. The study supports the use of implementing quantitative feedback devices into the teaching of joint mobilization when a reliable and accurate reference standard has been identified.