Development and validation of a screening tool to predict the risk of chronic low back pain in patients presenting with acute low back pain: a study protocol

Abstract

Introduction

Around 40% of people presenting to primary care with an episode of acute low back pain develop chronic low back pain. In order to reduce the risk of developing chronic low back pain, effective secondary prevention strategies are needed. Early identification of at-risk patients allows clinicians to make informed decisions based on prognostic profile, and researchers to select appropriate participants for secondary prevention trials. The aim of this study is to develop and validate a prognostic screening tool that identifies patients with acute low back pain in primary care who are at risk of developing chronic low back pain. This paper describes the methods and analysis plan for the development and validation of the tool.

Methods/analysis

The prognostic screening tool will be developed using methods recommended by the Prognosis Research Strategy (PROGRESS) Group and reported using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. In the development stage, we will use data from 1248 patients recruited for a prospective cohort study of acute low back pain in primary care. We will construct 3 logistic regression models to predict chronic low back pain according to 3 definitions: any pain, high pain and disability at 3 months. In the validation stage, we will use data from a separate sample of 1643 patients with acute low back pain to assess the performance of each prognostic model. We will produce validation plots showing Nagelkerke R² and Brier score (overall performance), area under the curve statistic (discrimination) and the calibration slope and intercept (calibration).

Ethics and dissemination

Ethical approval from the University of Sydney Ethics Committee was obtained for both of the original studies that we plan to analyse using the methods outlined in this protocol (Henschke et al, ref 11-2002/3/3144; Williams et al, ref 11638).

Strengths and limitations of this study.

• First prognostic tool in low back pain to follow, a priori, the Prognosis Research Strategy (PROGRESS) framework, and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) reporting guidelines for prognostic research.

• Prespecifies statistical analysis plan and informative levels of tool performance to increase transparency of results and of the final report.

• Restricted to only the use of predictor variables measured in previous data sets.

• Minor differences in the way variables are measured between the development and the validation data sets.

Introduction

Acute non-specific low back pain (LBP) is widely reported to have a favourable prognosis;¹ pain intensity reduces rapidly in the first few weeks,² and around 60% have fully recovered (return to work, no disability, and no pain) by 3 months.³ However, for the 40% of patients who continue to report pain at 3 months, or ‘chronic’ LBP, the prognosis is much poorer.⁴ Despite this difference in prognosis, many randomised trials apply the same treatment to all patients with acute LBP, which is an inefficient approach. An alternative approach is to target specific interventions to those at higher risk of developing chronic LBP. Research evaluating this ‘stratified’ approach to acute LBP management has been identified as a priority.⁵

Secondary prevention in LBP refers to preventing patients with acute LBP from developing chronic LBP.⁶ The first step in secondary prevention is to identify factors that are associated with poor outcome, or prognostic factors.⁷ Recently, the Prognosis Research Strategy (PROGRESS) group proposed a framework for conducting prognosis studies that included standards for identifying prognostic factors.⁸ Once prognostic factors have been identified they can be combined to produce prognostic models^{9 10} to assist clinicians in management decisions and researchers in trial design. For example, prognostic models applied to patients with LBP have predicted outcome better than chance or clinical judgment alone.^{11 12} Models that are used to screen patients into risk groups are known as prognostic screening tools.

The prognostic screening tools that are currently available for LBP are of limited use in secondary prevention.¹³ For example, the majority of existing tools were developed in secondary care,^{14 15} and in groups of patients with acute LBP and chronic LBP,^{16 17} making their routine application in primary care problematic without further testing. In addition, the predictive validity of these tools is invariably poorer when used outside of the sample in which they were developed.¹⁸

These limitations demonstrate the importance of external validation,¹⁹ which is rarely reported.^{20 21} To assess the external validity of a prognostic model, discrimination (the probability that patients who develop the health outcome of interest are allocated higher risk scores) and calibration (how closely predicted outcomes match actual outcomes) should ideally be assessed in a separate sample from that used to initially derive the model. For the few tools that have been externally validated in acute LBP,^22–24 discrimination ranged from poor to moderate, and calibration was only reported in one study.²³

There is also variability in the outcomes that prognostic tools have been developed to predict. Although these tools should predict reliable and clinically important outcomes,²⁵ in LBP there is no consensus on which outcomes are the most important.²⁶ Most published tools predict disability or return to work outcomes, rather than pain, which is the most common outcome used to define chronic LBP.^{21 27} Von Korff and Saunders²⁸ have argued that pain, in particular high pain intensity and/or disability, is the most important outcome to assess at 3 months. Three months is also the time at which a marked change in prognosis occurs.³ A recently formed research task force²⁷ agreed on a uniform definition of chronic LBP: 3 months’ worth of pain days in the past 6 months. The task force also emphasised the importance of grading the impact of chronic pain and disability, though validated cut-offs were not available. The recent progress made to define chronic LBP highlights the need to identify not only the patients at risk of ongoing pain for 3 months, but also the patients at risk of developing high-impact chronic pain and disability. Such patients are logical targets for early secondary prevention strategies. We are not aware of any tool that was developed to predict the onset of chronic pain.

The aim of this protocol is to describe the method and analysis plan for the development and validation of a prognostic screening tool for acute LBP that is suitable for secondary prevention.

Methods

Our study design is informed by the PROGRESS framework and specific recommendations for statistical approaches to prognostic research (table 1). We will report the study in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement for prediction studies.³⁴

Table 1.

Recommendations for prognostic research

PROGRESS type of clinical research	Types 1 and 27 29	Type 38 9 30–33			Type 410
Research theme	Original prognosis research and prognostic factor research	Development of a prognostic model	Validation of a prognostic model	Refinement of a prognostic model for clinical use	Impact study
Current recommendations (PROGRESS recommendations in parentheses)	Register study and publish a protocol (Recommendation 10) Recruit a representative and well-defined sample at a common, early time point (Recommendation 14) Ensure complete follow-up of sufficient length Choose prognostic factors measured based on sound theory Blind outcome assessors Account for covariates statistically Ensure sample size is large enough to assess multiple prognostic factors (10 outcome-events-per-predictor-rule) Validate the model Report all results explicitly and transparently (Recommendation 15)	Select candidate predictors that are clinically relevant Evaluate data quality (Recommendation 20) Describe data handling decisions for example, continuous variables should be analysed on their continuous scale (Recommendation 13) Select variables to be included in the final model using a prespecified strategy Assess the performance of the model (internal validation), ie, overall performance, discrimination and calibration	Prespecify acceptable performance of the model Assess overall performance, discrimination, and calibration in the validation sample Include a validation plot Update and recalibrate the model	Adjust the tool for clinical use Use a simple interface Do not refer to the tool as a ‘rule’ Make sure all aspects of the tool are clear and unambiguous Include uncertainty interval (95%CI) around posterior probability estimates	Take care with underpowered statistical analyses that are not pre-planned Report all subgroup findings Subgroup analyses should be replicated in new data Analyse continuous outcomes on their continuous scale Design RCTs to be 4 arm, ie, intervention and control in groups +ve and –ve on rule (Recommendation 22) Studies should compare ‘stratified’ vs ‘all-comer’ approaches (Recommendation 23)

PROGRESS, Prognosis Research Strategy.

The PROGRESS framework outlines four types of prognostic research: (1) fundamental prognosis research; (2) prognostic factor research; (3) prognostic model research; (4) stratified medicine research. The proposed study is Type 3, prognostic model research (table 1). To develop the prognostic screening tool, we will consider the recommendations of Kamper et al²⁹ and Royston et al.⁹ To validate the prognostic screening tool, we will use the method suggested by Altman et al³⁰ and Steyerberg et al.³⁵

Development of the tool

Development sample

The development sample data for the proposed study is from a cohort study that assessed the prognosis of acute LBP in primary care. Details of this study have been published elsewhere.³ In short, consecutive patients were recruited between November 2003 and July 2005 from primary care practices in the Sydney Metropolitan area, Australia. In the Australian healthcare system, ‘primary care’ includes first contact care provided by general practitioners, allied health practitioners (physiotherapists, chiropractors), and pharmacists. The original study recruited an inception cohort of 1248 patients with acute LBP (<4 weeks’ duration), 973 of which had pain of less than 2 weeks’ duration. All patients were managed according to the Australian National Health and Medical Research Council guidelines for acute LBP.³⁶ Outcomes were measured via telephone at 6 weeks, 3 months and 12 months. Key design features of the study are described in table 2 below.

Table 2.

Prognosis study by Henschke et al³ adherence to PROGRESS recommendations

Recommendation		Notes
Study was registered and protocol published (Recommendation 10)	Y	Protocol published and registered prior to original statistical analysis57
Sample was representative and recruited at a common, early time point (Recommendation 14)	Y	Patients were eligible if presenting to primary care with acute non-specific low back pain37
Follow-up was complete and of sufficient length	Y	98.4% follow-up at 3 months
Prognostic factors were based on sound theory	Y	6 groups of putative prognostic factors were measured: current history, past LBP history, sociodemographic characteristics, general health, psychological factors and work38–40
Outcome assessors were blinded	N	Unlikely to have introduced bias because this study did not originally have a hypothesis regarding prediction
Covariates were accounted for	NA	Not relevant for tool development.
Sample size was large enough to assess multiple prognostic factors (10 outcome-events-per-predictor-rule)	Y	1248 patients were recruited for a total of 21 predictors
Model was validated	NA	See ‘Validation of the tool’ below
Results were explicitly and transparently reported (Recommendation 15)	Y	Full protocol and results are published and available

LBP, low back pain; N, not achieved; NA, not applicable to the present study; Y, achieved.

Measures

Outcomes

The primary outcome will be the presence of chronic LBP. Chronic LBP will be defined as having greater than ‘very mild’ pain 3 months after the initial assessment. In the development sample, this was classified as >2 on a 6-point Likert scale (How much back pain have you had in the past week? 1=‘none’, 2=‘very mild’, 3=’mild’, 4=‘moderate’, 5=‘severe’, or 6=‘very severe’).⁴¹ The validation sample used a different pain rating scale, so we have defined greater than ‘very mild’ pain on this numeric rating scale as ≥3/10. We chose these cut-offs because a large proportion of patients with ‘ very mild’ pain (1–2/10) pain consider themselves to be recovered.²⁶

We will also develop two additional prognostic screening tools that consider the impact of pain and disability when defining chronic LBP.²⁷ For the tool predicting high pain, patients will be classified as having ‘chronic LBP (high pain)’ if they reported ‘moderate’, ‘severe’ or ‘very severe’ pain intensity on a 6-point scale, or ≥5 on the 11-point scale in the validation sample, at 3 month follow-up.⁴² Because there is still no consensus on what constitutes ‘high impact’ or ‘severe’ chronic LBP, we have selected our own cut-off in line with Von Korff and Saunders'²⁸ recommendation of ‘moderate’ or greater pain intensity at 3 months. For the tool predicting disability, patients will be classified as having ‘chronic LBP (disability)’ if they reported ≥2/5 on a 5-point disability scale (During the past week, how much has LBP interfered with your normal work (including both work outside the home and housework)? 1=‘not at all’, 2=‘a little bit’, 3=‘moderately’, 4=‘quite a bit’, 5=‘extremely’)⁴¹ at 3 month follow-up. Once again, because no established cut-offs exist,²⁷ we selected our own cut-off for what might constitute clinically important disability. We chose a disability score of ≥2/5 on the 5-point scale used in the development sample,⁴¹ or ≥7/24 on the 24-point Roland Morris Disability scale used in the validation sample. When both are converted to a 0–10 scale, these values approximate each other (7/24×10=3/10 or 1.5/5).² Choosing this disability cut-off will allow comparison to two recently published prognostic screening tools, which also selected a cut-off of 7 on the Roland scale.^{16 24}

Predictors

Candidate predictors will be selected from those measured at baseline in the cohort study if they are: (1) simple and reliable to measure in practice and (2) have a theoretical association with the development of chronic LBP. Candidate predictors are listed in online supplementary appendix A.

Statistical analysis

Cases with missing values will be removed from the dataset if follow-up rates are higher than 95%. If missing data exceeds 5%, ‘single imputation’⁴³ will be used. As per the recommendation from the PROGRESS group (table 1), we will ensure at least 10 cases per candidate predictor variable, to adequately power the regression analysis.^{9 44}

Variable selection

A logistic regression analysis will be used to investigate the relationship between the prognostic variables at baseline and the measures of chronic LBP. Age, sex and duration of the pain episode will be entered into block 1 of a multivariate analysis to reduce the complexity of the final model for clinical use, and ensure maximum sensitivity of the final models. In block 2, each potential prognostic factor will be added stepwise to the model by using an automated forward selection procedure. We will set a liberal significance level (p<0.10) to select variables that remain in the model. The predicted probability of chronic LBP will be modelled using the logistic regression equation

Continuous predictor variables will be treated as linear in the first multivariable regression model. After the initial predicted probabilities of chronic LBP are calculated, the linearity of continuous predictor variables will be examined with the predicted probability of chronic LBP as the dependent variable using scatter plots and the Box-Tidwell transformation.⁴⁵ The Box-Tidwell transformation involves log transforming each continuous predictor variable, producing an interaction term between the original variable and its log (eg, pain×pain LN), and including this term in the regression analysis. If the interaction term is significant, this indicates significant non-linearity. Continuous predictor variables that demonstrate a non-linear relationship with the dependent variable (predicted probability of chronic LBP) will be transformed by using fractional polynomial procedure.⁴⁶ Regression analyses will be performed using SPSS.⁴⁶

Scoring

Each individual will be allocated a risk score. The risk score will be calculated by a sum of the products of individual values of each predictor variable and its regression coefficient.⁴⁷ The full algorithm will be used to produce a score in the first instance to maximise predictive capacity. For the purpose of examining the performance of the predictive tool, patients will be classified as low, medium and high risks, based on their quartile of risk. Those in the highest quartile will be classified as high risk and those in the lowest quartile as low risk. The middle two quartiles will be classified as medium risk.

Performance

To examine the apparent performance (internal validity) of the prognostic screening tools, we will assess measures of overall performance, calibration and discrimination. Overall performance will be assessed using the Nagelkerke R² and Brier score. The Brier score is a method of quantifying differences between actual binary outcomes and their predictions, that is, average prediction error.³⁵ The Brier score ranges from 0 to 0.25, values close to 0 represent a useful model and values close to 0.25 a non-informative model. Calibration, that is, the agreement between observed and predicted frequencies of a given outcome, will be determined by plotting the mean predicted versus observed cases of chronic LBP for 10 risk stratification levels. The calibration slope and calibration-in-the-large statistic (intercept) will be calculated by constructing calibration plots. Discrimination, that is, the ability of the tool to discriminate between patients who did (+ve case) or did not (−ve case) develop chronic LBP, will be determined by using a Receiver Operator Characteristic Curve analysis, by calculating discrimination slope (box plots) and by examining risk-stratified likelihood ratios. Performance indices and plots will be calculated using R software.⁴⁸ Rules for interpretation of these statistics are presented in the Discussion.

After the performance indices have been calculated, we will internally validate the model using bootstrapping techniques suggested by Moons et al³³ (see online supplementary appendix B, Table B). Bootstrapping will be performed in SPSS using syntax available at http://gjyp.nl/marta/BootstrapValidationOfLogisticRegression.SPS. To assess model fit and optimism, bootstrapped estimates of the Nagelkerke R² and its SE will be compared with the original model estimates. We will conduct a sensitivity analysis to assess performance of the tool for patients in different settings (general practice, physiotherapy, chiropractic).

Validation of the tool

Validation sample

The validation sample consists of 1643 participants from a randomised trial conducted over 235 primary care centres in Greater Metropolitan Sydney, Australia. The trial, published elsewhere,^49–51 found no effect of paracetamol on recovery from acute LBP compared with placebo. In short, all participants were adults with acute non-specific LBP (<6 weeks duration) who had presented to primary care (GP, physiotherapist or pharmacist) between November 2009 and March 2013. Participants received up to 4 weeks of regular paracetamol, as-needed paracetamol, or placebo, and were followed up for 3 months on measures of pain and recovery.

Measures

Outcomes

Data on chronic LBP incidence (pain and disability at 3 months) will be extracted from the validation sample. Where necessary, all outcomes extracted from the validation sample will be transformed to match the format of the development sample. Patients will be classified as having chronic LBP, chronic LBP (high pain) and chronic LBP (disability), using the same definitions as were used in the development stage.

Statistical analysis

Each prognostic model will be assessed in the validation sample using the same statistical procedures as in the development stage. We will calculate estimates of overall performance (Nagelkerke R², Brier Score) discrimination (area under the curve (AUC), likelihood ratios, discrimination slope), and calibration (Calibration plot, Hosmer-Lemeshow test).

Posterior probability

We will calculate posterior probability along with 95% confidence limits according to the method recommended by Haskins et al.⁵² The 95% confidence limits will be calculated in Microsoft Excel using the following β distribution equations:

Validation plots

Validation plots of predicted versus observed risks showing the intercept, slope, AUC, scaled Brier score, and R² will be produced to summarise and compare performance of the tool in the development and validation samples.

Updating the tool

Because the validation sample is more recent, we will consider updating and extending the model in the validation sample. We will update the model using a recalibration method described by Steyerberg et al,⁵³ which involves multiplying the regression coefficients and intercept from the original logistic equation by the calibration slope (β) and the calibration intercept (α or ‘calibration in the large statistic’) from the validation data, to produce a new logistic equation. To extend the model, we will add new, potentially useful predictors from the validation sample for example, sleep quality.

If, based on our prespecified criteria below, we find the predictive validity of the prognostic model to be informative, we will consider simplifying it for clinical use (table 1). This process may involve steps such as refining and specifying measurement of the predictors, simplifying and clearly describing calculation of the prediction score/strata, and producing an electronic or paper-based form designed for clinical application.

Discussion

We have described the methods and statistical analysis plan to develop and validate a prognostic screening tool for acute LBP. To our knowledge, this tool will be the first of its kind in LBP to follow, a priori, the PROGRESS framework and TRIPOD reporting guidelines for prognostic research. Importantly for secondary prevention, the tool will be developed specifically to predict the onset chronic LBP at its inception at 3 months. The study is limited to using predictors measured in previously collected data sets.

We will use contemporary statistical methods to assess the calibration and discrimination of the screening tool in the two samples. The relative importance of calibration and discrimination ultimately depends on the purpose of the screening tool. For example, if the purpose of the tool is to aid clinical decision-making and provide accurate estimates of risk to patients, then calibration is an important consideration. If a clinician were to inform their patient that they had a 10% chance of chronic LBP, this estimate would be misleading if the tool was not well calibrated and, for example, 40% of patients with the same level of risk actually developed chronic LBP. If, on the other hand, the purpose of the tool is to select appropriate patients to include in a randomised trial, for example, in a trial to prevent chronic LBP, then adequate discrimination is important. In this example, a poorly discriminating tool would misclassify a large proportion of patients, including a number of inappropriate (low risk) patients and excluding appropriate (high risk) patients.

Ideally, a useful screening tool should have discrimination and calibration that is considered informative for its purpose. In general, Hosmer and Lemeshow suggest that for a logistic regression model, an AUC statistic of <0.7 represents poor discrimination, 0.7–0.8 acceptable discrimination, 0.8–0.9 excellent discrimination and 0.9–1.0 outstanding discrimination.⁵⁴ However, Steyerberg points out that for clinical decisions that are close to a ‘toss-up’, a tool with an AUC of 0.6 can be informative.³⁵ Published LBP tools report AUC values that range between 0.6²³ and 0.75⁵⁵ for pain, and between 0.68⁵⁶ and 0.83⁵⁵ for disability at 3–6 months of follow-up. On the basis of the reference standard values and those of previous work, we will consider AUC values of less than 0.6 to be non-informative. We also plan to use our additional measures of discrimination (likelihood ratios, discrimination plots) to determine whether or not the tool is informative. For example, overlapping likelihood ratio estimates among low-risk, medium-risk and high-risk groups would indicate poor discrimination. If the posterior probability CI includes the prevalence rate, we will consider the tool to not be informative, and not likely to be clinically useful.

Acceptable calibration of the tool will be based on the results of the calibration plots. If observed frequencies of the chronic LBP in the validation sample fall within 5% of predicted frequencies, the tool will be considered to have acceptable calibration. Calibration in the large statistic (intercept) should be close to 0 and the slope close to 1. With such a large sample, the p value of the Hosmer-Lemeshow test will be interpreted with caution.

Conclusion

This protocol outlines the design of development and validation studies for a prognostic screening tool in acute LBP. Results coming from this study will be interpreted for both clinical and research purposes.