Table 1.
Stages in the development and evaluation of clinical prediction rules
| Stage of development | Methodological standards |
|---|---|
| Stage 1. Identifying the need for a CPR | • Consider conducting qualitative research with clinicians to determine clinical relevance and credibility of CPR |
| • Conduct a systematic review of the literature to identify and evaluate existing CPRs developed for the same purpose | |
| • Consider updating, validating or testing the impact of existing CPRs | |
| Stage 2. Derivation of a CPR according to methodological standards | Study design for the derivation of a CPR |
| • Consider registering the study and publishing a protocol | |
| • Ensure the dataset is representative of the population for whom the CPR is intended | |
| • Conduct a prospective multicentre cohort study | |
| Statistical analysis | |
|
• Conduct multivariable regression analysis (logistic for binary outcomes, Cox for long-term prognostic outcomes) • Identify the model to be used, plus rationale if other methods used | |
| Missing data | |
| • Use multiple imputation | |
| Selection of candidate predictors for inclusion in a multivariable model | |
| • Only include relevant predictors based on evidence in the literature/clinical experience | |
| • Aim for a sample size with a minimum of ten events per predictor, preferably more | |
| • Avoid selection based on univariable significance testing | |
| • Avoid categorising continuous predictors | |
| Selection of predictors during multivariable modelling | |
| • Backward elimination of predictors is preferred | |
| • Avoid data-driven selection and incorporate subject-matter knowledge into the selection process | |
| Definition and assessment of predictor and outcome variables | |
| • Define predictor and outcome variables clearly | |
| • Consider inter-rater reliability of predictor measurement and potential measurement error | |
| • Aim for blind assessment of predictor and outcome variables | |
| Internal validation | |
| • Use cross-validation or bootstrapping and adjust for optimism | |
| • Ensure to repeat each step of model development if using bootstrapping | |
| CPR performance measures | |
| • Assess and report both calibration and discrimination | |
| • Consider decision curve analysis to estimate the clinical utility of the CPR | |
| Presentation of a CPR | |
| • Report the regression coefficients of the final model, including the intercept or baseline hazard | |
| • Consider a clinical calculator if the CPR is complex | |
| Reporting the derivation of a CPR | |
| • Adhere to the TRIPOD guidelines [36] | |
| Stage 3. External validation and refinement of a CPR | Study design for the external validation of a CPR |
| • Conduct a prospective multicentre cohort study | |
| • Aim for a sample size with a minimum of 100 outcome events, preferably 200 | |
| • Consider using a framework of generalisability to enhance the interpretation of the findings [34] | |
| Types of external validation | |
| • Conduct temporal, geographical and domain validation studies to ensure maximum generalisability | |
| • If multiple validations have been performed, conduct a meta-analysis to summarise the overall performance of the CPR, using a published framework [35] | |
| Refinement of a CPR: model updating or adjustment | |
| • Consider updating, adjusting or recalibrating the CPR if poor performance is found in an external validation study | |
| • Consider further external validation of updated CPRs | |
| Comparing the performance of CPRs | |
| • Compare the CPR with other existing CPRs for the same condition | |
| • Ensure the statistical procedures used for comparison are appropriate; consider a decision-analytic approach | |
| Reporting the external validation of a CPR | |
| • Adhere to the TRIPOD guidelines [36] | |
| Stage 4. Impact of a CPR on clinical practice | Study design for an impact analysis |
| • Consider whether the CPR is ready for implementation | |
| • Conduct a cluster randomised trial with centres as clusters, or a before–after study | |
| • Perform appropriate sample size calculations | |
| • Consider decision-analytic modelling as an intermediate step prior to a formal impact study | |
| Measures of impact of a CPR | |
| • Report the safety and efficacy of the CPR | |
| • Report the impact of the CPR on clinician behaviour if assessed | |
| Acceptability of a CPR | |
| • Evaluate the acceptability of the CPR using the validated OADRI [48], or using qualitative or vignette methods | |
| Comparison of a CPR with unstructured clinical judgement | |
| • Compare the sensitivity and specificity of the CPR with clinicians own predictions/decisions | |
| The four phases of impact analysis for CPRs | |
| • Follow the framework for the impact analysis of CPRs [33] | |
| • Ensure extensive preparatory and feasibility work is conducted prior to a formal impact study | |
| Reporting the impact analysis of a CPR | |
| • There are currently no published reporting guidelines for impact studies of CPRs; this is an area for future research | |
| Stage 5. Cost-effectiveness | • Conduct a formal economic evaluation, with sensitivity analyses to examine the uncertainty of the model projections |
| Stage 6. Long-term implementation and dissemination | • Devise and evaluate targeted implementation strategies to ensure maximum uptake |
| Barriers and facilitators to the use of CPRs | |
| • Assess barriers to the use of the CPR and devise strategies to overcome these |
CPR clinical prediction rule, TRIPOD Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis, OADRI Ottawa Acceptability of Decision Rules Instrument