Clinical Prediction Rules for Physical Therapy Interventions: A Systematic Review

Abstract

Background and Purpose: Clinical prediction rules (CPRs) involving physical therapy interventions have been published recently. The quality of the studies used to develop the CPRs was not previously considered, a fact that has potential implications for clinical applications and future research. The purpose of this systematic review was to determine the quality of published CPRs developed for physical therapy interventions.

Methods: Relevant databases were searched up to June 2008. Studies were included in this review if the explicit purpose was to develop a CPR for conditions commonly treated by physical therapists. Validated CPRs were excluded from this review. Study quality was independently determined by 3 reviewers using standard 18-item criteria for assessing the methodological quality of prognostic studies. Percentage of agreement was calculated for each criterion, and the intraclass correlation coefficient (ICC) was determined for overall quality scores.

Results: Ten studies met the inclusion criteria and were included in this review. Percentage of agreement for individual criteria ranged from 90% to 100%, and the ICC for the overall quality score was .73 (95% confidence interval=.27–.92). Criteria commonly not met were adequate description of inclusion or exclusion criteria, inclusion of an inception cohort, adequate follow-up, masked assessments, sufficient sample sizes, and assessments of potential psychosocial factors. Quality scores for individual studies ranged from 48.2% to 74.0%.

Discussion and Conclusion: Validation studies are rarely reported in the literature; therefore, CPRs derived from high-quality studies may have the best potential for use in clinical settings. Investigators planning future studies of physical therapy CPRs should consider including inception cohorts, using longer follow-up times, performing masked assessments, recruiting larger sample sizes, and incorporating psychological and psychosocial assessments.

Clinical prediction rules (CPRs) (or clinical decision rules) have become popular in the physical therapy literature. The intent of CPRs is to assist clinicians in making a diagnosis, establishing a prognosis, or implementing an intervention.¹ Although it has been suggested that well-constructed CPRs can improve clinical decision making and practice,² there is a lack of consensus as to what constitutes a methodologically sound CPR, especially in the derivation stage.

McGinn and colleagues³ recommended a 3-step process in the development and testing of a CPR. The first step involves the derivation of the CPR (derivation studies), and the second step involves the validation of the CPR (validation studies). The third step involves assessment of the impact of the rule on clinical behavior, also referred to as an “impact analysis.” For CPRs for physical therapy interventions, steps 2 and 3 are not routinely performed. Although it has been suggested that a validated CPR can be applied in various settings with confidence in its accuracy,⁴ our impression is that most CPRs reported in the physical therapy literature are derivation studies. Furthermore, the lapse in time before validation occurs can be extensive. This situation presents clinicians with the dilemma of whether they should incorporate the results of a derivation study into their clinical practice.

Our opinion is that the quality of a derivation study is one factor that should be considered before a CPR is implemented into clinical practice. This interpretation can be difficult, however, because the quality of CPR derivation studies pertaining to interventions has not been reported. Assessing the quality of derivation studies has potential advantages for physical therapy practice and research. First, a quality assessment will assist clinicians in deciding whether a given CPR is appropriate for implementation into clinical practice. Second, a quality assessment will assist future researchers in the design of high-quality studies for the development of new CPRs.

Therefore, the purpose of this systematic review was to determine the quality of CPRs developed for interventions used by physical therapists. Studies were included in this review if the explicit purpose was to develop a CPR related to a specific intervention approach for conditions commonly treated by physical therapists. Previously validated CPRs were excluded from this review because there is less debate over the clinical application of validated CPRs² and because methodological concerns about derivation studies are of less concern when a validation study has been reported.

Method

Data Sources and Searches

A systematic review of relevant databases (PubMed, CINAHL, ProQuest, and Academic Search Premier) from their inception up to June 2008 resulted in the retrieval of 49 potential publications (Figure). The search strategy began with the filter “predict$ OR clinical$ OR outcome$ OR risk$.”⁵ In comparison with a gold standard,⁶ this filter has a sensitivity of 98.4% for retrieving CPRs from the literature.⁵ A second search strategy consisted of the key words “clinical prediction rule.” The first author examined reference lists from all selected publications to verify that no pertinent publications were missed during the above-described searches. Studies were included in this review if the explicit purpose was to develop a CPR related to a specific intervention approach for conditions commonly managed by physical therapists. Previously validated CPRs were excluded from this review.

Figure.

Flow chart depicting search and selection process for clinical prediction rule (CPR).

Data Extraction and Quality Assessment

Quality scores were independently assigned to eligible studies by 3 reviewers using a modified version of a list of criteria, reported by Kuijpers et al,⁷ for assessing the methodological quality of prognostic studies. These criteria were selected because, in our experience, the physical therapy literature has followed a model that uses cohort designs for CPR derivation studies. Therefore, CPR derivation studies involving physical therapy interventions are appropriate for quality assessment tools that are sensitive to methodological issues that affect prognostic studies.⁸ Another reason why these criteria were appropriate for our purpose was that they were developed by authors aware of issues related to CPRs. Specifically, Kuijpers et al⁹ later reported a CPR for determining the prognosis for patients with shoulder pain in general practice settings.

The original list of criteria⁷ was altered slightly by removing the criterion related to the rate of response of potential study participants because this item is not commonly reported in the physical therapy literature. Additionally, we added an important criterion by including masking of outcome assessors and treating clinicians.^6,8 In our opinion, the resulting criteria are similar to those that have been suggested for evaluating the quality of prognostic studies for patients receiving physical therapy care⁸ and are consistent with the process of evaluating prognostic variables.¹⁰

The 18 criteria used to assess quality in this systematic review represented 8 categories: study population, response information, follow-up, intervention, outcome, masking, prognostic factors, and data presentation. A description of each criterion is provided in the Appendix. The criteria could be scored as positive, negative, or unclear. A positive score indicated that the criterion was identified in the study and met specific requirements consistent with a high-quality prognostic study. A negative score indicated that the criterion was identified in the study but did not meet specific requirements. A score of “unclear” meant that the study provided insufficient information regarding that criterion. To obtain a conservative estimate of quality, negative and unclear ratings were collapsed when study quality was rated. A total quality score was determined by adding positive scores, providing a potential high score of 18 (100%).

Ratings of individual studies to determine quality scores were independently assigned by the 3 reviewers before a meeting on the interpretation of the 18-item list of criteria (time 1). The meeting provided an opportunity for reviewers to assess agreement and discuss criteria that resulted in high disagreement. In addition, components for a given criterion that may have been overlooked by reviewers were clarified. As appropriate, a quality criterion was updated to reflect an updated interpretation. For example, there were differences in opinion about what constituted a prospective study (criterion E). During the meeting, an operational definition for a prospective study was approved. After the meeting, the 3 reviewers again independently assigned ratings (time 2), this time using the final guidelines provided in the Appendix.

Data Synthesis and Analysis

Statistical pooling of results was not performed because of the obvious heterogeneity among studies in populations used, interventions applied, and outcome measures administered. Reliability analyses were performed with SPSS 15.0 for Windows^* and Excel.^† Percentage of agreement was calculated for individual items. Negative and unclear ratings were collapsed into one variable so that ratings could be dichotomized into 2 categories. Interrater reliability was reported for the total quality score by use of the intraclass correlation coefficient (ICC [2,1]) and respective 95% confidence intervals (CI).¹¹

To determine a single overall quality score for an individual study (Tab. 1), each of the 3 reviewers’ scores for a particular study were averaged to account for the possibility that interrater agreement was less than 100% after time 2. Therefore, the overall quality scores solely reflect the results after time 2 ratings. As in other reviews using this scoring system, high-quality studies were operationally defined as those that had average quality scores of greater than 60%.⁷

Table 1.

Individual Clinical Prediction Rule (CPR) Studies and Mean Overall Quality Scores

^aMean of scores from all 3 reviewers after time 2.

Results

The initial search strategy yielded a total of 4,080 publications. A majority of the publications did not meet the criteria for review because they did not pertain to physical therapy interventions described in the Guide to Physical Therapist Practice.¹² After titles were reviewed and duplicates were determined, 25 potential publications were further reviewed. Of these, 17 publications were excluded on the basis of the following criteria: not intervention based (n=12),^9,13–23 nonspecific intervention (n=3),^24–26 and eventually validated CPR (n=2).^27,28Publications were excluded on the basis of the “nonspecific intervention” criterion if they did not report on the results of a specific physical therapy intervention approach or used multidisciplinary interventions. For example, methods consisting of various combinations of physical therapy interventions that were not specifically described, such as “exercises aimed at restoring neuromuscular control at the shoulder,”26(pp1231–1232) or not described in general, such as “physiotherapy for shoulder pain,”24(p486) were grounds for exclusion. Gross and Battié²⁵ used a multidisciplinary approach that consisted of physical therapy, occupational therapy, exercise therapy (kinesiology), medicine, and psychology but that was not specific to physical therapy alone; therefore, their study was also excluded. Additionally, studies that potentially included interventions that were not within the scope of physical therapy practice were excluded.²⁶

The remaining 8 studies were included in this review. Two additional publications were included after a review of reference lists and related articles, resulting in the analysis of 10 publications in this review. Five studies involved CPRs for responses to spinal manipulation.^29–33 The other studies predicted responses to lumbar stabilization,³⁴ hip mobilization,³⁵ patellar taping,³⁶ multimodal interventions for cervical radiculopathy,³⁷ and trigger point therapy for headache³⁸ (Tab. 1).

Methodological Criteria

Percentage of agreement on ratings of individual items ranged from 70% to 100%; items B, E, H, K, L, and P had the lowest levels of agreement (70%–86.7%) (Tabs. 2 and 3). After a meeting on the interpretation of the 18-item list of criteria, percentage of agreement on ratings of individual items ranged from 90% to 100% (Tabs. 2 and 3). Individual items commonly rated as low quality (ie, not meeting the criteria in greater than 50% of the studies) were items A, B, F, K, M, and R. Among these items, the inclusion of an inception cohort (item A), description of inclusion and exclusion criteria (item B), and follow-up of ≥6 months (item F) were met in ≤10% of the studies. The results indicated that for 6 items, all 3 reviewers were in absolute agreement that a given criterion was met across all studies reviewed (eg, all 3 reviewers were in absolute agreement that item B was met in 3% of all studies reviewed) (Tab. 4).

Table 2.

Interrater Reliability

^aOverall quality score converted to percentage.

Table 3.

Individual Item Percentage of Agreement

Table 4.

Methodological Criteria Commonly Receiving Low Ratings

^aItems for which all 3 reviewers agreed (absolute agreement) that a given criterion was met across the 10 studies reviewed (eg, all 3 reviewers were in absolute agreement that item B was met in 3% of all studies reviewed).

Quality Score

Absolute agreement among the 3 reviewers on overall quality scores was calculated at time 1 (ICC=.67, 95% CI=.10–.91) and after a meeting on the interpretation of the 18-item list of criteria (time 2) (ICC=.73, 95% CI=.27–.92) (Tabs. 2 and 3). Next, a mean quality score was calculated for each reviewer (Tab. 1). At time 2, mean quality scores for individual studies (X̄=11.13; 61.8%) ranged from 8.67 to 13.33 (48.2%–74.0%). Five studies^{29,30,34,37,38} were rated at greater than 60% (range=61.1%–74.0%). Four studies^31–33,36 were rated at 50% to 60% (range=53.7%–59.3%), and one study³⁵ was rated at less than 50% (48.2%).

Discussion

The intent of this review was to rate the methodological quality of CPR derivation studies reported in the physical therapy literature. This is an important issue because the quality of derivation studies for physical therapy interventions has not been systematically considered. The lack of consideration of derivation studies has important clinical and research implications. For example, CPR validation studies often are not performed, a fact that makes the decision to incorporate CPRs into clinical practice a potentially difficult one. One CPR has been validated in clinical settings with samples of patients similar to those included in the derivation study.²⁸ However, this CPR has not been validated in different settings with new groups of patients and different intervention parameters.³⁹ On the basis of the results of this review, several CPRs may be appropriate for clinical applications involving patients and clinical environments similar to those used in the CPR derivation process. We acknowledge that the quality of a derivation study is only one aspect of clinical decision making for the use of a CPR. However, it is information that was not previously reported and therefore might enhance clinical decision making. We used a quality score of greater than 60% as a cutoff score for a high-quality study, because this threshold was used in a previous review of prognostic studies.⁷

Studies that met this quality index included CPRs for determining factors associated with responses to stabilization exercises,³⁴ responses to muscle trigger point therapy for tension-type headaches,³⁸ the inability of patients with low back pain to show improvement with spinal manipulation,²⁹ manipulation of the thoracic spine in patients diagnosed with mechanical neck pain,³⁰ and a multimodal intervention approach for cervical radiculopathy.³⁷ The lower-quality studies included CPRs for predicting favorable responses to cervical manipulation in patients with neck pain,³¹ the management of cervicogenic headache,³² patellar taping,³⁶ lumbopelvic manipulation in patients with patellofemoral pain syndrome,³³ and hip mobilization for knee pain indicative of osteoarthritis.³⁵ Quality scores can assist clinicians in deciding whether to use these nonvalidated CPRs. However, quality scores are not a substitute for CPR validation studies. Validation studies provide more-definitive information for clinical applications because they are independent studies of new subjects and involve a variety of clinicians and patients.^1,3,6,40

An important factor to consider for a methodologically sound CPR derivation is the risk-to-benefit ratio associated with its application. Risk was not empirically assessed in the CPRs considered in this review, a fact that is not surprising given the status of the physical therapy literature.⁴¹ Clinical prediction rules were originally developed in the medical profession for decisions involving higher associated risks, such as those associated with traumatic injuries.^42–45 The risk associated with the interventions used in the CPRs considered in this review is believed to be minimal in comparison with the risk associated with emergency medicine. For example, consider the risk associated with the use of stabilization exercises for low back pain³⁴ in comparison with the risk of not ordering radiographs for traumatic injuries.^42–44 Failure to detect a fracture is associated with a risk higher than that associated with the use of stabilization exercises when those are not indicated. Unfortunately, the current physical therapy literature does not allow a quantitative consideration of the risk-to-benefit ratio, so clinical decisions must be based on qualitative factors.

Individual items that received low-quality ratings were similar to previously suggested areas of methodological concern for CPR studies.^2,6,40 Specifically, masking of outcome assessors and treating clinicians (item K) did not occur in a majority of the studies reviewed. Masking of outcome assessors and treating clinicians is important for limiting the measurement bias of potential predictor variables.^1,3,8 Additional areas of concern identified in this review included the use of an inception cohort and definition of the duration of symptoms in eligibility criteria. To limit potential error in establishing a prognosis, subjects should be enrolled in a common time frame with regard to their current condition.⁸ This criterion was not a component in a majority of the studies used to develop CPRs. Therefore, samples used to develop CPRs may lack homogeneity, thereby increasing the potential for bias in predictor variables and outcomes. Another area lacking in CPR derivation studies published to date was the follow-up period, which was suggested to be at least 6 months. Immediate effects were commonly reported; such immediate effects might be beneficial only in demonstrating evidence of responsiveness.⁸ Longer follow-up times are needed to demonstrate valid clinical implications for the use of a given intervention. Finally, an assessment of potential psychosocial prognostic factors (item M) was commonly not included. Psychological factors, such as kinesophobia, catastrophizing, anxiety, and depression might have strong influences on outcomes related to musculoskeletal conditions.^46–49 Incorporating these factors into the development process has important clinical implications for future CPRs.

Limited sample sizes have been reported to be common methodological flaws in CPR studies.² It has been suggested that 10 to 15 subjects are required for each prospective predictor variable in CPR studies.⁵⁰ Not meeting this requirement may lead to inaccurate statistical results because of overfitting of regression models.⁵⁰ It is important that our sample size determination was based on the final CPR model and not on initial prospective variables. The result was that only 40% of studies had an adequate sample size, and this was a liberal estimate. If we had elected to use initial prospective variables, then no studies would have met the criterion for sample size, a result suggesting that previously noted concerns about small sample sizes used in CPR derivation studies are legitimate. We suggest that future studies include larger sample sizes to account for derivation regression models in addition to the final, more parsimonious models.

Several limitations of this systematic review should be considered in the interpretation of the results. Although the results may be relevant to the decision-making process for implementing a CPR in practice, our findings should not be viewed as definitive. Our data provide complementary information on which CPRs to use in clinical practice, but the ultimate decision must be made in the context of a clinician's experience and factors specific to the encounter with a patient. These factors include, but are not limited to, whether patients seen in clinical practice are similar to those enrolled in the respective CPR study and whether a quantitative assessment of the risk-to-benefit ratio is available.

Another limitation is that the quality criteria used in this review were developed for prognostic studies, not specifically for CPR derivation studies involving interventions. Although these study designs are similar, there may be subtle differences with regard to quality determinations. The quality criteria used in this review did not include certain statistical elements that may have important implications for CPRs. For example, the criteria did not include the consideration of a quantitative risk-to-benefit analysis,⁴¹ reporting of potential predictor variable reliability,^2,6 or reporting of CIs^2,6 and effect sizes.⁵¹ Furthermore, the quality scores were equally weighted so that the “methodological importance” of a category was equally distributed among all of the criteria. This decision was made because we did not have clear evidence to follow for weighting decisions. Another relevant issue is that randomized designs have been suggested to be appropriate for CPR studies involving intervention selection.^6,40 Although this may be true, it appears that the use of cohort studies is much more common in the physical therapy literature, because only one study included in this review used a randomized design.³² Therefore, future assessments of the quality of CPR derivation studies should include the development of a standardized rating system with a more-specific statistical criterion, consideration of weighting of quality scores on the basis of the methodological importance of particular categories, and the development of a criterion that is sensitive enough to determine the overall quality of a study design (such as distinguishing between cohort and randomized designs).

There was substantial agreement among the raters on individual items and overall quality scores; however, it is clear that agreement can be improved. Improvement can be accomplished by providing quality criteria more explicit than previously reported criteria, especially with regard to inception cohort, responders versus nonresponders, frequency of outcome measures, and sample size determination. The reliability estimates were also imprecise (large 95% CIs); we believe that this result may have been attributable to the relatively small number of studies included in this review. Additionally, we opted to collapse negative and unclear ratings, a strategy that may have influenced the percentage of agreement among the reviewers. However, this decision to collapse the data was made a priori and, even if the data had not been collapsed, the overall quality scores would not have been affected (Tab. 1). These scores considered only positive ratings because negative and unclear ratings were equally weighted as “0” when the overall quality scores were determined.⁷

Conclusion

This systematic review provides impetus for future research. First, it reinforces the importance and priority of performing validation studies. There are currently 10 CPR derivation studies in the physical therapy literature that have not been validated, and these studies vary greatly in overall quality. A glut of CPR derivation studies with various degrees of quality may only further confuse clinical practice, a result that is contrary to the original intent of CPRs. It is clear that follow-up validation studies are a high priority for advancing clinical practice. Second, the results of this review provide a clear direction for investigators wishing to conduct high-quality CPR derivation studies. In our opinion, the areas that should be a high priority for future derivation studies aimed at CPR development are recruiting inception cohorts, performing masked assessments, including longer follow-up times, collecting larger sample sizes, and incorporating psychological or psychosocial assessments.

Appendix.

Appendix.

Eighteen-Item List of Criteria for Assessing the Methodological Quality of Studies^a

^aCriteria B, E, H, and R required consensus agreement or clarification before the second rating process, as follows: B—clarification regarding the inclusion of duration of symptoms; E—consensus agreement on the operational definition of a prospective design; H—clarification regarding information provided for dropouts/lost to follow-up; and R—clarification regarding the number of cases in the multivariate analysis being at least 10 times the number of independent variables in the multivariate analysis (on the basis of the final clinical prediction rule model).