Potential Pitfalls of Clinical Prediction Rules

What Are Clinical Prediction Rules?

A clinical prediction rule (CPR) is a combination of clinical findings that have statistically demonstrated meaningful predictability in determining a selected condition or prognosis of a patient who has been provided with a specific treatment^1,2. CPRs are created using multivariate statistical methods, are designed to examine the predictive ability of selected groupings of clinical variables^3,4, and are intended to help clinicians make quick decisions that may normally be subject to underlying biases⁵. The rules are algorithmic in nature and involve condensed information that identifies the smallest number of indicators that are statistically diagnostic to the targeted condition⁶. The number of derived or validated CPRs is increasing⁶, specifically in rehabilitation medicine where prescriptive studies have been developed for musculoskeletal interventions for low back pain^7,8, cervical pain^9,10, and knee dysfunction^11,12.

Clinical prediction rules may best be classified into three distinct groups: 1) diagnostic, 2) prognostic, and 3) prescriptive^1,13. Studies that focus on predictive factors related to a specific diagnosis are known as diagnosticCPRs. Clinical prediction rules that are designed to predict an outcome such as success or failure are considered prognostic. Clinical prediction rules designed to target the most effective interventions are identified as prescriptive, and these require prospective, longitudinal, randomized controlled trials that compare outcomes after selected interventions for subjects who meet a similar score on the CPR¹.

Clinical prediction rules are generally developed using a 3-step method¹⁴. First, CPRs are derived prospectively using multivariate statistical methods to examine the predictive ability of selected groupings of clinical variables³. The second step involves validating the CPR in a randomized controlled trial to reduce the risk that the predictive factors developed during the derivation phase were selected by chance¹⁴. The third step involves conducting an impact analysis to determine the extent that the CPR improves care, reduces costs, and accurately defines the targeted objective¹⁴.

Although there is little debate that carefully constructed CPRs can improve clinical practice, to my knowledge, there are no guidelines that specify methodological requirements for CPRs for infusion into all clinical practice environments. Guidelines are created to improve the rigor of study design and reporting. The following editorial outlines potential methodological pitfalls in CPRs that may significantly weaken the transferability of the algorithm. Within the field of rehabilitation, most CPRs have been prescriptive; thus, my comments here are reflective of prescriptive CPRs.

Methodological Pitfalls

CPRs are designed to specify a homogenous set of characteristics from a heterogeneous population of prospectively selected consecutive patients^5,15. Typically, the resulting applicable population is a small subset of a larger sample and may only represent a small percentage of the clinician's actual daily caseload. The setting and location of the larger sample should be generalizable^15,16, and subsequent validity studies require assessment of the CPR in different patient groups, in different environments, and with a typical patient group seen by most clinicians¹⁶. Because many CPRs are developed based on a very distinct group, that may or may not be reflective of a typical population of patients, the spectrum transportability¹⁷ of many current CPR algorithms may be limited.

Clinical prediction rules use outcome measures to determine the effectiveness of the intervention. Outcome measures must have a single operational definition⁵and require enough responsiveness to truly capture appropriate change in the condition¹⁴; in addition, these measures should have a well constructed cut-off score^16,18 and be collected by a blinded administrator¹⁵. The selection of an appropriate anchor score for measurement of actual change is currently debated¹⁹,²⁰. Most outcome measures use a patient recall-based questionnaire such as a global rating of change score (GRoC), which is appropriate when used in the short term but suffers from recall bias when used in long-term analyses¹⁹,²⁰,²¹. Other studies may use minimally detectable change scores that were originally validated using the GROC and also may be affected by both recall bias and differences in sample severity or pathology. Lastly, outcome measures that use scores that are influenced by administrative factors (discharge date, length of stay, patient charges), socio-demographic factors, or internal behavioral characteristics (changes in fear avoidance or attitude) are not consistent among populations⁵.

A potential drawback for CPRs is the failure to maintain the quality of the tests and measures used as predictors in the algorithm. The prospective test and measures should be independent of one another during modeling¹⁶; each should be performed in a meaningful, acceptable manner⁴; and clinicians or data administrators should be blinded to the patient's outcomes measures and condition²². Furthermore, the tests should demonstrate acceptable reliability (≥ 0.60)¹⁵and require administration within an acceptable timeframe of the outcome measure²²; equivocal or indeterminable results necessitate reporting²². Recognizing the likelihood of a true positive finding in the absence of any information will avoid the representative heuristic pitfall that may compel us toward identifying a clinical test as positive simply because the result fits the pattern of other findings²³. CPRs that use tests and measures with reliability or agreement below 0.60 may result in variable findings depending on the clinician who performs the examination and depending on the findings of other tests and measures.

It is my impression that the most frequent current pitfall of CPRs is associated with the failure to meet statistical assumptions during regression modeling. CPRs are typically underpowered falling below the suggested requirements of 10 to 15 subjects for each prospective predictor variable²⁴. Validation cohorts require sampling sizes of 100 or greater with use of logistic regression (used as a standard for CPR assessment)²⁵. Rarely is the statistical significance of the model reported in the rehabilitation-based CPRs, nor is the R² or R²-equivalent of the model identified⁵. An R² or R²-equivalent outlines the strength of association of the predictor variables (both independently and as a group) in explaining the variance of the outcome measure. Low R² or R²-equivalents may suggest that other variables more accurately predict the outcome of the study⁵ and generally suggest a low effect size of the independent variables identified and retained in the analyses²⁶. Most CPRs do report confidence intervals, and when reported, wide confidence intervals imply poor precision or too small of a sample size¹⁵.

Once a CPR is developed, it is important to recognize the true benefit of the tool. It has been suggested that for true impact on clinical practice, CPRs should provide a LR+ of 5 or greater²⁷. CPR derivations performed on high-risk groups, where failure to provide the appropriate intervention is highly undesirable, should have sensitivity values that are greater than specificity values²⁸. This indicates that the final algorithm will accurately provide all of the best treatment(s) possible versus assuring that only those specific to the problem are used²⁸.

CPRs should have clinical sensibility. Clinical sensibility implies that the tool makes inherent clinical sense, that it's easy to use, that the tests and measures are truly related to the outcome, and that clinician perception does not overly alter the findings of the tool¹⁵. Consequently, tests and measures that vary in clinical interpretation (e.g., spring tests of the spine) or that are potentially explained by factors beyond the original scope of the examination (e.g., hip osteoarthritis when addressing hip procedures that affect the knee) may not be as useful as factors that are more explicit during clinical assessment.

Lastly, most rehabilitation-related CPRs are derivation studies, which are the initial steps in the development of clinical decision rules. Derivation studies lack validation and require follow-up studies in diverse centers with different populations of patients and different clinicians. Whether the findings from a derivation study stand up to the scrutiny of further assessment is unknown¹⁵. In essence, adoption of a derivation-only CPR runs the risk of improper treatment. Careful attention should be made before blindly adopting derivation studies or basing treatment pathways on these tools.

Summary

Is this editorial an attack on clinical prediction rules? Actually, it's quite the contrary. Prescriptive CPRs are useful tools for a select and discrete population of patients. As manually oriented clinicians, we have long realized that sub-sets of the population benefit from manual therapy more so than others. CPRs allow us to isolate a sub-set of desired patient characteristics and to define which techniques are most useful for that population. The current rehabilitation-based CPRs have opened the door for additional research to improve our accuracy as clinicians. Unfortunately, many of the present rehabilitation-based CPRs may have methodological weaknesses that may allow questioning of the utility of the instrument. Although there is no such thing as a “perfect” study, better and more rigorous designs should provide additional, profound and clinically applicable findings. As a clinician and a researcher, I am an advocate of CPRs.