Developing an International Register of Clinical Prediction Rules for Use in Primary Care: A Descriptive Analysis

Abstract

PURPOSE

We describe the methodology used to create a register of clinical prediction rules relevant to primary care. We also summarize the rules included in the register according to various characteristics.

METHODS

To identify relevant articles, we searched the MEDLINE database (PubMed) for the years 1980 to 2009 and supplemented the results with searches of secondary sources (books on clinical prediction rules) and personal resources (eg, experts in the field). The rules described in relevant articles were classified according to their clinical domain, the stage of development, and the clinical setting in which they were studied.

RESULTS

Our search identified clinical prediction rules reported between 1965 and 2009. The largest share of rules (37.2%) were retrieved from PubMed. The number of published rules increased substantially over the study decades. We included 745 articles in the register; many contained more than 1 clinical prediction rule study (eg, both a derivation study and a validation study), resulting in 989 individual studies. In all, 434 unique rules had gone through derivation; however, only 54.8% had been validated and merely 2.8% had undergone analysis of their impact on either the process or outcome of clinical care. The rules most commonly pertained to cardiovascular disease, respiratory, and musculoskeletal conditions. They had most often been studied in the primary care or emergency department settings.

CONCLUSIONS

Many clinical prediction rules have been derived, but only about half have been validated and few have been assessed for clinical impact. This lack of thorough evaluation for many rules makes it difficult to retrieve and identify those that are ready for use at the point of patient care. We plan to develop an international web-based register of clinical prediction rules and computer-based clinical decision support systems.

INTRODUCTION

Clinical prediction rules are tools that quantify the impact of multiple predictors from a patient’s history, physical examination, or laboratory results to inform a diagnosis, prognosis, or treatment response.¹ Different methods are used to develop these rules, including univariate and multivariate analysis, neural networks, predictive nomograms, and classification and regression tree (CART) analysis.^2,3 Examples of clinical prediction rules include the Centor score to predict streptococcal pharyngitis⁴ and the ABCD2 rule (age, blood pressure, clinical features, duration of symptoms, and diabetes history) to predict stroke.⁵

Before widespread clinical implementation, clinical prediction rules should pass through 3 stages of development. First, they should undergo derivation, whereby factors with predictive power are identified to develop the rule. Second, they should undergo validation, whereby the rule is tested in a new population for reliability and accuracy. This stage can be divided into 2 substages: narrow validation, in which the rule is tested in a similar population or clinical setting, and broad validation, in which the rule is tested in a different population or clinical setting. Third, the rules should undergo impact analysis, which assesses their impact on patient outcomes, physician behavior, or costs.¹ The majority of clinical prediction rules have been derived but not validated, and relatively few have undergone impact analysis.^6–8

Clinical prediction rules offer one way of implementing evidence-based medicine, especially if incorporated into clinical decision support systems (CDSSs), at the point of care. We are developing an international web-based register of clinical prediction rules for use in primary care, in collaboration with the Cochrane Primary Health Care field. We intend this register to be a resource for clinicians, available at the point of care, relevant to clinical practice, with rules summarized and categorized in terms of their derivation, validation, and impact analysis status. Although a number of clinical prediction rules are designed for use in primary care settings, articles describing these rules are particularly difficult to identify. This difficulty stems from the use of multiple terms to describe clinical prediction rules (eg, decision aid, score card) and primary care (eg, family medicine, general practice). Furthermore, no specific term is used to code clinical prediction rule articles in electronic search engines such as PubMed. To overcome this limitation, we developed an electronic search string to search the database for rules relevant to primary care.⁹ This search strategy was supplemented by searching additional relevant resources to develop a register of rules that are relevant to this setting.

The aims of this article are to describe the clinical prediction rules relevant to primary care that were retrieved for the creation of this register and to summarize them in terms of their clinical domain, stage of development, methodologic quality assessment, and the clinical setting in which they were studied.

METHODS

Search Strategy

We first searched the MEDLINE database (PubMed) using a search string specifically developed in house to retrieve clinical prediction rules relevant to primary care from 30 preselected medical journals between 1980 and 2009.⁹ No restriction was placed on language. See Supplemental Appendix 1 for details of the search string and included journals. Each article was screened on a hierarchical basis by title, abstract, and full text, if necessary. We next searched secondary sources of clinical prediction rules, including the Journal of the American Medical Association (JAMA) Rational Clinical Examination Series,¹⁰ and a handbook of rules.¹¹ Personal resources were also investigated. Key experts in the field were asked for relevant articles through announcements made at 2 international conferences. In-house researchers and librarians also made their personal libraries available. For every validation article retrieved, we obtained the original derivation article; if derivation articles had not been retrieved through other search methods, we searched for them manually.

Inclusion and Exclusion Criteria

Articles were eligible for inclusion in the register if they were considered to describe clinical prediction rules relevant to primary care. A clinical prediction rule was defined as “a clinical tool that quantifies the individual contributions that various components of the history, physical examination, and basic laboratory results make toward the diagnosis, prognosis, or likely response to treatment in a patient. [Clinical prediction rules] attempt to formally test, simplify, and increase the accuracy of clinicians’ diagnostic and prognostic assessments.”¹ We excluded health questionnaire screening tools. We included rules at all stages of development (derivation, validation [narrow or broad], or impact analysis). We used the term validation study to describe studies that attempted to validate a rule regardless of outcome.

Primary care was defined as “... normally the point of first medical contact within the health care system, providing open and unlimited access to its users, dealing with all health problems regardless of the age, sex, or any characteristic of the person concerned.”¹² Studies of clinical prediction rules conducted in non–primary care settings were eligible for inclusion if they were relevant to primary care. As the register is designed to be used internationally, we used broad inclusion criteria to acknowledge variation in the same-day diagnostic tests that are available across countries, with the understanding that not all countries will have access to the same services. The role of primary care clinicians also differs internationally.

Data Extraction

We screened all articles for inclusion according to title and abstract. Any queries were discussed with 2 authors, 1 methodologist, and 1 general practitioner. For each rule in an article, we extracted information on the following: (1) type of article: original study, systematic review, or review; (2) stages of development of the clinical prediction rule: derivation, validation, or impact analysis; (3) type of rule: prediction rule, decision rule, or both; (4) clinical domain to which the rule applies according to the International Classification of Primary Care, 2nd edition (ICPC-2); and (5) clinical setting in which the rule was studied.

We also collected several pieces of additional information. First, we assessed methodologic quality of each article using appropriate checklists; for derivation, there were 8 criteria assessing internal and external validity (Supplemental Appendix 2a), for validation studies there were 5 criteria assessing internal and external validity (Supplemental Appendix 2b), and for impact analysis, the choice of methodologic assessment varied according to the study type (Supplemental Appendix 2c). Second, we determined the country in which the study was conducted. Third, we ascertained the term used to describe clinical prediction rules, for example, clinical prediction rule, nomogram, or score card. These findings are not the main focus of this article, but details are presented in an online Appendix 3 (available at http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data).

RESULTS

Search Strategy

The process we used to identify and include articles for analysis is shown in Figure 1. The MEDLINE PubMed search (1980–2009) retrieved 72,837 articles; a total of 1,151 full-text articles were retrieved, of which 277 met all inclusion criteria. These articles were supplemented with 66 articles from secondary sources,^10,11 243 from experts, and 159 identified by searching for derivation articles from references of validation articles. We ultimately included a total of 745 unique articles in the register.

Figure 1.

Flow diagram of articles on clinical prediction rules in primary care.

Note: The 745 articles included in our review contained data on 895 original studies as many articles described more than 1 clinical prediction rule study. Analyses pertain to 434 unique clinical prediction rules.

^a Articles did not pertain to a clinical prediction rule, were not relevant to primary care, or both.

Overview of Retrieved Articles

Several of the articles included in the register described more than 1 stage of development of a clinical prediction rule or described multiple rules. For example, some authors derived and validated a rule in the same article,⁵ and others validated and compared a number of rules in the same article.¹³ We therefore conducted our analysis at the level of pieces of information about a single clinical prediction rule; for example, if an article both derived and validated a rule, it was considered as providing 2 pieces of information about that rule. This unit of measurement is henceforth referred to as a clinical prediction rule study. In all, the 745 articles contained 989 clinical prediction rule studies on 434 unique rules. Most, 895 (90.5%), were original studies; a few were systematic reviews (3.9%) or reviews or guidelines (5.6%). Our main analysis is based on the 895 original studies; the systematic reviews and reviews/guidelines are discussed in Appendix 4 (available online at http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data).

Publication of original clinical prediction rule studies spanned 1965 to 2009 and became more common in recent years (Figure 2). Of the 895 original studies included in the register, the majority were published in the 2000s (68.2%), markedly more than were published in the 1990s (20.2%), the 1980s (10.8%), and 1965–1979 (0.8%). The majority of new clinical prediction rules were derived and validated in the 2000s. Of the 461 validation studies, more were narrow (57.4%) than broad (42.5%). The overall number of impact analysis studies was relatively very low.

Figure 2.

Clinical prediction rule studies, split by decade reported and stage of development of the rule (N = 895).

Note: Few studies were reported before 1980; therefore, we grouped these studies into a broader time period (1965–1979).

Clinical Prediction Rule Characteristics

Stage of Development

Of the 434 unique clinical prediction rules included in the analysis, 238 (54.8%) were associated with at least 1 validation study, although some were associated with more. The stages of development of all included rules are presented in Appendix 5 (available online at http://hrbcentreprimarycare.ie/?q=cpr-register-paper-supplemental-data).

The validation studies predominantly concentrated on certain clinical prediction rules: 34 focused on the Alvarado score for appendicitis¹⁴; 15 on the CRB-65 (Confusion, Respiratory rate, Blood pressure, aged 65 years and older) for pneumonia¹⁵; 12 on the CURB-65 (Confusion, Urea nitrogen, Respiratory rate, Blood pressure, aged 65 years and older) for pneumonia¹⁵; 12 on the Framingham Coronary Heart Disease rule¹⁶; 11 on the ABCD2 rule for stroke⁵; 11 on the Centor score for streptococcal pharyngitis⁴; 10 on the Ottawa Ankle Rule for need for radiograph after ankle injury¹⁷; and 10 on the Pneumonia Severity Index (PSI).¹⁸

Of the 434 unique clinical prediction rules, only 2.8% had been evaluated in at least 1 impact analysis study, although some had been evaluated in more (Table 1).^19–34 The studies predominantly focused on a pair of rules: 4 focused on the Ottawa Ankle Rule and 2 on the McIsaac rule. The impact analysis studies included here had a mixture of study designs and outcome measures. Most studies reported improvement in primary outcomes with implementation of the rule.

Table 1.

Impact Analysis Studies of Clinical Prediction Rules (16 Studies Covering 12 Rules)

Clinical Domain	Clinical Prediction Rule	Author, Year	Clinical Setting	Study Design	Study Outcome
D (Digestive)
D14: Hematemesis/vomiting blood	Blatchford score	Stanley et al, 200919	Emergency department	Before-after	Positive
D88: Appendicitis	Alvarado score (MANTRELS)	Farahnak et al, 200720	Emergency department	RCT (pilot)	Positive
K (Cardiovascular)
K22: Risk factors for cardiovascular disease	UKPDS risk engine (patients with diabetes) with Dutch guidelines risk table	Koelewijnvan Loon et al, 200921	Primary care	Cluster RCT	Negative
	New Zealand risk guidelines for cardiovascular disease	Montgomery et al, 200022	Primary care	Cluster RCT	Negative
K74: Ischemic heart disease with angina	Pozen 1984 for admission in acute ischemic heart disease	Pozen et al, 198423	Emergency department	On-off	Positive
K93: Pulmonary embolism	Charlotte rule	Kline et al, 200424	Emergency department	Before-after (controlled)	Positive
	Wells rule for PE	Wells et al, 200325	Emergency department	Cluster RCT	Positive
L (Musculoskeletal)
L73: Fracture tibia/fibula	Ottawa ankle rule	Bessen et al, 200926	Emergency department	Before-after	Positive
	Ottawa ankle rule	Auleley et al, 199727	Emergency department	RCT	Positive
	Ottawa ankle rule	Stiell et al, 199528	Emergency department	Before-after (controlled)	Positive
	Ottawa ankle rule	Stiell et al, 199429	Emergency department	Before-after (controlled)	Positive
	Ottawa knee rule	Stiell et al, 199730	Emergency department	Before-after (controlled)	Positive
N (Neurologic)
N81: Injury nervous system other	Canadian cervical-spine rule	Stiell et al, 200931	Emergency department	Cluster RCT	Positive
R (Respiratory)
R72: Streptococcal pharyngitis	Centor score	Worrall et al, 200732	Primary care	RCT	Negative
	McIsaac rule	McIssac et al, 200233	Primary care	RCT	Negative
	McIsaac rule	McIssac and Goel, 199834	Primary care	RCT	Positive

MANTRELS = migration to the right iliac fossa, anorexia, nausea/vomiting, tenderness in the right iliac fossa, rebound pain, elevated temperature (fever), leukocytosis, and shift of leukocytes to the left; PE = pulmonary embolism; RCT = randomized controlled trial; UKPDS = UK Prospective Diabetes Study.

Clinical Domains

We classified the clinical prediction rule studies using ICPC-2 coding.³⁵ The studies included in the register spanned 17 broad clinical domains (Figure 3 and Supplemental Appendix 3). The majority pertained to cardiovascular, respiratory, and musculoskeletal areas.

Figure 3.

Broad clinical domains for the clinical prediction rule studies, split by stage of development of the rule (N = 895).

Note: Studies were classified according to the International Classification of Primary Care, 2nd Edition (ICPC-2).³⁵

Differences existed in terms of the stages of development of the rules across clinical domains (Figure 3). Some areas (eg, neurologic) were associated with more derivation than validation studies, whereas others (eg, digestive) were associated with more validation than derivation studies. Just 5 of the clinical domains (digestive, cardiovascular, musculoskeletal, neurologic, and respiratory) were associated with any impact analysis, and only a small subset of the rules within each of these domains had undergone impact analysis.

The clinical domains with the largest number of clinical prediction rules derived were stroke/cerebrovascular accident (K90, 29 rules), risk factors for cardiovascular disease (K22, 26 rules), pneumonia (R81, 24 rules), influenza (R80, 17 rules), pulmonary embolism (K93, 16 rules), and osteoporosis (L95, 16 rules). Many of these rules had not gone through validation or impact analysis, however. For example, for the K90 rule for stroke/cerebrovascular accident, there were more derivation studies than validation studies (29 vs 27), and no impact analysis studies had been done. The validation studies focused on 13 rules, with the ABCD2 being the most commonly validated rule (11 studies).⁵ The mixed pattern observed here is similar to that observed across the remaining clinical domains.

Clinical Setting

The clinical prediction articles included in the register came from a range of clinical settings (Table 2). The most common settings were primary care (28.2%) and the emergency department (28.0%). A large proportion also came from the outpatient setting (13.6%) or hospital setting (16.8%). Few articles described research among patients from 2 settings, for example, primary care and the emergency department (0.7%) and primary care and the inpatient setting (0.3%).

Table 2.

Clinical Setting of Clinical Prediction Rule Studies (N=895)

Clinical Setting	Studies, No. (%)
Primary care: general practice, community, physiotherapy clinic, nursing home, population studies, chiropractor clinic, residential clinic	252 (28.2)
Emergency department	251 (28.0)
Hospital: hospital inpatients, tertiary care, trauma center, stroke unit	150 (16.8)
Specialty clinics: specialty clinics including diabetes, cardiology, prostate, pediatric, arthritis, veteran affairs	122 (13.6)
Hospital inpatients and specialty clinics	24 (2.7)
Primary care and emergency department	6 (0.7)
Prehospital (emergency services)	6 (0.7)
Primary care and specialty clinics	6 (0.7)
Primary care and inpatients	3 (0.3)
Other
Clinical trial/study	27 (3.0)
Setting unclear	43 (4.8)
Guideline/opinion	5 (0.6)

DISCUSSION

Summary of Main Findings

The publication of clinical prediction rules has increased over time, particularly for cardiovascular, respiratory, and musculoskeletal conditions. Despite the large numbers of rules derived, however, only slightly more than one-half have been validated, and these validations are more likely to have been narrow than broad. Moreover, few rules have gone through impact analysis. Such limited evaluation makes it difficult to recommend clinical prediction rules uncritically for widespread clinical use. In this context, our development of an international register of clinical prediction rules relevant to primary care will help prioritize areas of research, as well as provide a platform for the development of CDSS tools that integrate clinical prediction rules into the point-of-care clinical decision support.

Comparison With the Existing Literature

Ours is the first review of clinical prediction rules that specifically focuses on primary care. Although similar reviews have been conducted, they have generally been on a smaller scale, searching up to 6 medical journals and retrieving up to 61 rules over a short time frame.^6,8,36,37 The exception is a review of clinical prediction rules for children, which retrieved 101 rules but searched only 2 resources.⁷ These studies have also indicated a pattern of limited validation and even more limited impact analysis. This lack of in-depth study of rules makes it difficult to determine the impact that they have on patient care, physician behavior, and health care costs.^1,3,6

Study Limitations

We took a broad approach to the clinical settings that were eligible for inclusion and included any clinical prediction rules that could be applied in a primary care setting. The majority of studies we included were conducted in primary care and emergency department settings. Some rules had been derived in hospital settings and subsequently validated in primary care (eg, the Centor score for streptococcal pharyngitis).⁴ The findings from these validation studies need to be considered in the context of the study setting, particularly with respect to the biases associated with patient selection in the different settings. Other rules had been derived for use in community settings but validated in hospital settings (eg, the CRB-65 for pneumonia).¹⁴ It is therefore necessary to adapt a broad approach to adequately quantify narrow and broad validation studies, and to account for differences between health care services and access to same-day technology across different countries. For this reason, not all included clinical prediction rules will apply to primary care in all countries.

Although we searched multiple resources to retrieve relevant articles for the register, the search was not designed to be a systematic review of any of the clinical prediction rules included here. Thirty primary care journals were purposively chosen from an extensive list of relevant journals as described in Supplemental Appendix 1. In addition, only 1 reviewer identified relevant articles during the article screening process using predefined inclusion criteria. As such, although the derivation article for every rule is included here, it is possible that some validation and impact analysis studies were not retrieved by our search strategy. It is also possible that we have overlooked clinical prediction rules relevant to primary care. Nevertheless, our review is substantially broader in scope than previous reviews.^6–8,36,37

Clinical Implications

The data we describe will be used to establish a web-based international register of clinical prediction rules relevant to primary care through the Cochrane Primary Health Care field, which we envisage will be available beginning in mid-2014. The coding of articles according to their clinical domain and stage of development will allow for easy navigation through the database.

The coefficients and algorithms for each of the clinical prediction rules have been extracted and will be used to develop a computer-based CDSS. This process will allow the pretest probability to be adjusted for any rules that were not derived or validated in a primary care setting. The development of a web-based CDSS should overcome some of the barriers associated with promoting the use of clinical prediction rules in routine clinical practice.³⁸ Future work will focus on integrating such clinical decision support fully into the electronic health record, which will assist in the knowledge transfer of up-to-date clinical evidence that will be available for use at the point of care.

Future Research

Our work highlights a number of areas for future research. The development of a register of clinical prediction rules will enable identification of clinical domains in which such rules have not yet been developed. Identifying these gaps may be particularly useful for determining where to focus research efforts to develop new rules, rather than simply adding to the mass of existing rules for a condition that is well covered. Where rules already exist, it is argued that these rules should be adapted to accommodate changes in clinical setting or to incorporate new evidence regarding predictor variables or changes in the management strategies of the population of interest.^8,39 From a primary care perspective, research efforts should be focused on developing or adapting clinical prediction rules that safely rule out serious illnesses, given the lower prevalence of these conditions in this setting.

Clinical prediction rules with limited validation and impact analysis can also be identified. The systematic reviews of rules identified here offer a way to determine the predictive accuracy of the rule, which may help prioritize which rules should undergo impact analysis. The methodologic quality analysis may also be useful in this regard. Previous reviews have identified quality components, such as clinical sensibility, as important for selecting rules for impact analysis.⁶ These research efforts should comply with relevant methodologic quality checklists, to overcome the problems outlined here in this regard.

It is clear that establishing an international register of clinical prediction rules relevant to primary care will require checking of multiple resources. The numerous terms used to describe clinical prediction rules in the literature makes it difficult to develop a search string/term for use in electronic resources such as PubMed. Our search string has good sensitivity and specificity but was designed to search 30 key journals relevant to primary care. The range of search terms identified in the current study provides a means to develop new search strings to identify clinical prediction rules across a range of journals and clinical domains. Furthermore, the contribution of both clinical and methodologic experts on clinical prediction rules has been identified as important to the ongoing development of the register. We will continue to make appeals to the research community to keep the register up to date, in addition to searching other resources.