External Validation of the Risk Assessment Model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for Medical Patients in a Tertiary Health System

Abstract

Background

Hospitalized medical patients are at risk for venous thromboembolism (VTE). Universal application of pharmacological thromboprophylaxis has the potential to place a large number of patients at increased bleeding risk. In this study, we aimed to externally validate the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) VTE risk assessment model in a hospitalized general medical population.

Methods and Results

We identified medical discharges that met the IMPROVE protocol. Cases were defined as hospital‐acquired VTE and confirmed by diagnostic study within 90 days of index hospitalization; matched controls were also identified. Risk factors for VTE were based on the IMPROVE risk assessment model (aged >60 years, prior VTE, intensive care unit or coronary care unit stay, lower limb paralysis, immobility, known thrombophilia, and cancer) and were measured and assessed. A total of 19 217 patients met the inclusion criteria. The overall VTE event rate was 0.7%. The IMPROVE risk assessment model identified 2 groups of the cohort by VTE incidence rate: The low‐risk group had a VTE event rate of 0.42 (95% CI 0.31 to 0.53), corresponding to a score of 0 to 2, and the at‐risk group had a VTE event rate of 1.29 (95% CI 1.01 to 1.57), corresponding to a score of ≥3. Low‐risk status for VTE encompassed 68% of the patient cohort. The area under the receiver operating characteristic curve was 0.702, which was in line with the derivation cohort findings.

Conclusions

The IMPROVE VTE risk assessment model validation cohort revealed good discrimination and calibration for both the overall VTE risk model and the identification of low‐risk and at‐risk medical patient groups, using a risk score of ≥3. More than two thirds of the entire cohort had a score ≤2.

Introduction

Venous thromboembolism (VTE) represents an important issue in the hospitalized, acutely ill medical patient population. Estimates from US databases reveal that up to 8 million hospitalized, acutely ill medical patients are at risk annually for developing VTE.¹ The incidence of deep vein thrombosis has been reported to range from 10% to 26% among general medical patients, and ≈75% of fatal pulmonary embolisms occur in nonsurgical patient populations.^2–3 Recent evidence suggests that higher rates of VTE prophylaxis in the absence of formal VTE risk stratification may not lead to reduced VTE rates.⁴

National quality organizations in the United States have opted for a group risk assessment and thromboprophylaxis strategy in the hospitalized medical patient^5–6; however, recent international guideline statements have stressed the need for individualized VTE risk assessment through the use of VTE risk assessment models (RAMs) in the acutely ill medical patient population.^5,7 This would allow proper identification of medical patients at risk of VTE and minimize potential harm from thromboprophylaxis for patients at low risk of VTE.⁷

Multiple VTE RAMs in the hospitalized, acutely ill medical patient population have been derived by expert consensus or by regression analysis to identify patient groups at risk for VTE^8–13; however, the VTE RAMs either have not undergone proper external validation or have been assessed only in management or impact analysis studies.^9,12 Proper external validation in settings and patient populations different from the populations from which the model was derived ensures the model's reproducible accuracy prior to widespread clinical use.¹⁴ The International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) VTE RAM was derived from a large international registry of 15 156 hospitalized, acutely ill medical patients.^10,15 The RAM consisted of 7 independent VTE risk factors that were given 1 to 3 points each, depending on their strength of association with VTE risk.¹⁰ These risk factors were then added to give a final composite VTE risk score that described the individual patient's VTE risk. Thresholds were then applied to define VTE risk groups. The objective of this study was to externally validate the IMPROVE VTE RAM in a hospitalized, general medical population as part of a large academic health system in the United States.

Methods

Medical patients at risk of VTE were identified as having been admitted to 1 of 2 tertiary care hospitals during the time period for which both billing and electronic health record data were available (December 2009 to April 2013.) Patients had be aged at least 18 years, have a length of stay of at least 3 days, and be discharged alive. Index admission required a primary medical diagnosis of acute infection, respiratory disease, heart failure, cancer, diabetes, pancreatitis, cholecystitis, or inflammatory bowel or rheumatic disease (Table S1). Patients with a secondary obstetric or mental health diagnosis were excluded (International Classification of Diseases, 9th revision [ICD‐9] codes 290 to 319 or 630 to 677). In an attempt to exclude patients with major surgical procedures, we included only patients whose billing diagnosis‐related group was suggestive of medical admission and excluded those suggestive of a surgical diagnosis. In addition, patients were excluded from the cohort if they had an international normalized ratio >1.5 or were receiving a therapeutic dose of anticoagulant therapy.

We identified VTE events using ICD‐9 codes (415.11, 415.13, 415.19, 451.11, 451.19, 451.81, 453.4, 453.40, 453.41, 453.87, 453.9). If the patient had a VTE ICD‐9 code as a secondary diagnosis present on admission for the index admission, the patient was excluded from the study. VTE events were those coded as not present on the index admission or, for those patients who did not have an event on the index admission, those coded as present on admission for their first readmission within 90 days following the index admission. Any readmission during the study period that did not identify a VTE event and that met the study criteria placed the patient in the at‐risk cohort again.

For a case–control approach to the individual risk factor analysis, we identified a sample of non‐VTE patients to match to the cases in a case–control format. Approximately 3 controls were identified for each case, matched by hospital, admission year, and patient sex. For quality control purposes, manual review of certain key data was performed. First, each VTE event identified by billing data was confirmed by review of objective testing result (Doppler compression ultrasound of lower extremity, chest computed tomography angiograms, and ventilation/perfusion scans). Cases were excluded from the study if no report could be found or if results were not consistent with VTE. Second, for all cases and controls, any operative reports associated with the index admission or 90 days prior were reviewed. Any report identifying a major surgical procedure resulted in exclusion of the case from the study. This review was not performed for the noncase population.

Patient characteristics considered as risk factors for VTE were those identified by the IMPROVE VTE RAM study.¹⁰ Points were assigned to each of these characteristics according to the 7‐factor IMPROVE model (previous VTE, 3 points; known thrombophilia, 2 points; lower limb paralysis, 2 points; cancer, 2 points; immobilization, 1 point; intensive care unit or coronary care unit stay, 1 point; aged >60 years, 1 point). For the risk factor of immobility, we used the proxy of bed rest for >7 days or a hospital length of stay >7 days. For the risk factor of lower limb paralysis, we used the proxy of hemiparesis, hemiplegia, paraplegia, or quadriplegia, as documented in the nursing‐assessment flow sheet of motor response (Table S2).

In the original derivation study of the IMPROVE VTE RAM, the American College of Chest Physicians (ACCP) threshold warranting pharmacological thromboprophylaxis was an IMPROVE score corresponding to a clinical VTE event rate of at least 1%.¹⁰ To perform our validation study of the IMPROVE VTE RAM, data were extracted from the electronic health records of close to 20 000 patients to reflect each patient's risk status during the index admission. Expected VTE event rates were calculated using the IMPROVE risk score and for individual risk factors along with their associated 95% CIs. Given the relatively low event rate, a case–control study was performed, and odds ratios were calculated for each IMPROVE VTE risk‐score category relative to the lowest risk group. A logistic regression model was calculated from the case–control sample to evaluate the predictive ability of the model. Receiver operating characteristic (ROC) curves were created to compare the overall diagnostic accuracy of the original derivation set¹⁰ and our validation set. All analyses were conducted using SAS version 9.3 (SAS Institute Inc).

The procedures used were reviewed and approved as being in compliance with ethical standards of the responsible institutional review committee at the home institution of the authors. All research activities were in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. This study was exempt from patient consent because the study was an analysis of a large data set.

Results

A total of 19 217 patients met the inclusion criteria, as shown in Figure 1. From this group, we identified 135 cases with a hospital‐acquired VTE event and 404 matched controls without a hospital‐acquired VTE event. Demographics for these groups are provided in Table 1. The cases were older (aged 70.78 versus 65.68 years) and had longer hospital length of stay (12.68 versus 7.48 days) than controls, and a higher percentage were female (62% versus 53%). A higher percentage of cases also had multiple VTE risk factors.

Figure 1.

Definition study population. DRG indicates diagnosis‐related group; ICD‐9, International Classification of Diseases, 9th revision; INR, international normalized ratio; LIJMC, Long Island Jewish Medical Center; LOS, length of stay; NSUH, North Shore University Hospital; VTE, venous thromboembolism.

Table 1.

Demographics of Study Population

	Noncases	VTE Event	Non‐VTE Event
	n=19 082	n=135	n=404
Age, y
Mean	65.7	70.8	66.1
Median	68	73	68
SD	18.3	14.1	17.7
Sex
Female	53%	62%	62%
Male	47%	38%	38%
Qualifying illness
Acute infection	39%	26%	39%
Respiratory	21%	19%	19%
Heart failure	14%	7%	15%
Cancer	14%	43%	12%
Diabetes, pancreatitis, cholecystitis	9%	2%	12%
Inflam bowel disease	3%	2%	3%
Rheumatic disease	0%	0%	0%
Index admission LOS
Mean	7.48	12.68	7.18
Median	6	9	6
VTE prophylaxis, pharmacological or order for mechanical	49%	49%	45%
Pharmacological VTE prophylaxis	43%	44%	40%

LOS indicates length of stay; VTE, venous thromboembolism.

Risk Factor Analysis

Of the 7 risk factors identified in the IMPROVE RAM study, 3 were found to be statistically associated with the risk of VTE in this population: aged >60 years, diagnosis of cancer, and prior VTE (Table 2). The results were essentially unchanged when the cases and controls were stratified into groups that received VTE prophylaxis, including pharmacological prophylaxis, during hospitalization and those that did not.

Table 2.

Risk Factors Associated With the IMPROVE RAM Score

	VTE Event (n=135)	Non‐VTE Event (n=404)	OR (95% CI)	P Value
Aged >60 years	108 (80.0%)	268 (66.34%)	2.03 (1.27 to 3.25)	0.0028
Prior cancer	76 (56.3%)	116 (28.71%)	3.20 (2.14 to 4.78)	<0.0001
Prior VTE	26 (19.26%)	28 (6.93%)	3.20 (1.80 to 5.69)	<0.0001
ICU/CCU stay	19 (14.07%)	41 (10.15%)	1.45 (0.81 to 2.60)	0.2093
Lower limb paralysis	1 (0.74%)	8 (1.98%)	0.37 (0.05 to 2.98)	0.4618
Immobility	42 (31.11%)	117 (28.96%)	1.11 (0.73 to 1.69)	0.6352
Known thrombophilic state	1 (0.74%)	1 (0.25%)	3.01 (0.19 to 48.41)	0.4385

ICU/CCU indicates intensive care unit or coronary care unit; IMPROVE, International Medical Prevention Registry on Venous Thromboembolism; OR, odds ratio; RAM, risk assessment model; VTE, venous thromboembolism.

Risk Score Analysis

The event rate and odds ratio at each level of IMPROVE risk score is shown in Table 3. The risk scores were collapsed into binary VTE risk categories of low risk and at risk, as shown in Table 4. This approach divides the 2 groups by their VTE incidence rates: The low‐risk group had a VTE event rate of 0.42 (95% CI 0.31 to 0.53), corresponding to an IMPROVE score of 0 to 2, and the at‐risk group had a VTE event rate of 1.29 (95% CI 1.01 to 1.57), corresponding to an IMPROVE score of ≥3. The odds ratio between the low‐risk and at‐risk groups was 3.07 (95% CI 2.17 to 4.33). Further calibration analysis revealed the ability of the model to distinguish between incidence rates based on the number of VTE events at each score level (score 0, 1, 2, 3, 4, 5 to 10) (Table S3).

Table 3.

Event Rate and Odds Ratio at Each Level of IMPROVE Risk Score

Score	VTE	No VTE	Event Rate	OR (95% CI)
	n=135	n=19 082
0	5	3917	0.13	Reference
1	27	5662	0.47	3.74 (1.44 to 9.71)
2	23	3368	0.68	5.35 (2.03 to 14.09)
3	41	3598	1.13	8.93 (3.52 to 22.62)
4	12	1393	0.85	6.75 (2.37 to 19.19)
5	11	596	1.81	14.46 (5.01 to 41.76)
6	11	369	2.89	23.35 (8.07 to 67.58)
7	4	140	2.78	22.38 (5.95 to 84.25)
≥8	1	39	2.5	20.09 (2.2933 to 175.94)

IMPROVE indicates International Medical Prevention Registry on Venous Thromboembolism; OR, odds ratio; VTE, venous thromboembolism.

Table 4.

Risk Scores Expressed as Binary VTE Risk

Score	VTE (n=135)	No VTE (n= 19 082)	Event Rate (95% CI)	OR (95% CI)
0 to 2 (low risk)	55	12 947	0.42 (0.31 to 0.53)	Reference
≥3 (high risk)	80	6135	1.29 (1.01 to 1.57)	3.07 (2.17 to 4.33)

OR indicates odds ratio; VTE, venous thromboembolism.

Model Discrimination

The ROC curve analysis is shown in Figure 2, in which the area under the ROC curve is 0.70 in our validation set. The negative and positive predictive values at each level of risk score are presented in Table 5. The negative predictive value was 99% across the IMPROVE RAM. The areas under the curve of the ROC curves did not show appreciable differences between the derivation and validation sets (derivation set 0.72; validation set 0.70) (Figure S1).

Figure 2.

Receiver operating characteristic curve for the validation cohort was 0.70.

Table 5.

Predictive Values at Each Threshold of IMPROVE RAM Score

Risk Score	Positive Predictive Value	Negative Predictive Value
0 vs ≥1	0.0085	0.9987
0 to 1 vs ≥2	0.0107	0.9967
0 to 2 vs ≥3	0.0129	0.9958
0 to 3 vs ≥4	0.0151	0.9942
0 to 4 vs ≥5	0.0231	0.9940
0 to 5 vs ≥6	0.0284	0.9936
0 to 6 vs ≥7	0.0272	0.9932
0 to 7 vs ≥8	0.0250	0.9930

IMPROVE indicates International Medical Prevention Registry on Venous Thromboembolism; RAM, risk assessment model.

Discussion

This study represents one of the largest, multicenter, external validation studies of a weighted and evidence‐derived VTE RAM in hospitalized, acutely ill medical patients, using the IMPROVE VTE risk score. The validation population revealed good discrimination characteristics of the risk score when compared with the original derivation population.¹⁰ The derivation and validation populations also exhibited concordance of VTE risk distribution across VTE risks scores, specifically, in low‐risk and at‐risk VTE risk score thresholds. Last, the results of this validation study used 7 evidence‐derived independent clinical factors in a weighted scoring system as part of the IMPROVE RAM that can be readily assessed during the course of hospital admission or hospital stay in the medically ill patient population.

This external validation study displayed an area under the ROC curve of 0.70, compared with the area under the ROC curve of 0.72 seen in the derivation cohort, which revealed good discrimination characteristics. In addition, as seen in the original derivation study, the validation study exhibited good calibration across increasing VTE risk scores, with symptomatic VTE event rates of ≈0.4% with IMPROVE scores of 0 to 2 (low VTE risk) and symptomatic VTE event rates of >1.0% with IMPROVE scores of ≥3 (at VTE risk). Importantly, if we assumed an IMPROVE score of ≥3 as a VTE risk threshold—as recommended by the ACCP guidelines for warranting pharmacological thromboprophylaxis, given a clinical VTE event rate of ≥1%—the present study suggested that ≈66% of the medically ill validation population would be at low risk of VTE. This finding is similar to the high proportion of patients with low VTE risk (66%) seen in the original derivation population.¹⁰ Consequently, the present validation study suggests an IMPROVE VTE risk score of ≥3 as a useful threshold in predicting an increased risk of VTE as a binary outcome of low risk versus at risk of VTE in this patient population. The model had a negative predictive value of ≈99% across all of the IMPROVE VTE scores, in line with previous VTE risk models in this patient population.¹⁶ Moreover, the use of thromboprophylaxis did not appreciably change either the relative weight or the rank order of the individual VTE risk factors, as was also seen in the original IMROVE VTE score derivation study.¹⁷ The 7 independent clinical VTE risk factors of the IMPROVE VTE risk score—previous VTE, known thrombophilia, lower limb paralysis, cancer, immobilization ≥7 days, intensive care unit or coronary care unit stay, and aged >60 years—have been well described in previous studies in this patient population and are simple to implement in a hospital setting.¹⁷

Multiple VTE RAMs in the medically ill population have been derived mostly by expert consensus that included subjective criteria.^9,12–13,16 These models have been assessed mostly in prospective management studies without proper external validation.^12,16 A recent validation study of 2 VTE RAMs in this patient population used non–evidence‐derived RAMs.¹⁶ The IMPROVE RAM was also found to have good calibration characteristics when applied to a large, global, phase 3 study of thromboprophylaxis in hospitalized, acutely ill medical patients, with an area under the ROC curve of ≈0.65.¹⁸ More recently, the IMPROVE VTE score has been externally validated in another large‐scale study in this patient population and revealed good discrimination and calibration characteristics.¹⁹ Based on a recent systematic review which was performed using validated quality criteria for risk modelling, only a few high quality evidence‐derived VTE RAMs exist which include the IMPROVE RAM and that of Woller et al.^10–11,20 The model by Woller et al incorporated a validation cohort as part of the same population as the derivation cohort, and this may have led to overly optimistic discrimination characteristics.²¹ Proper validation of VTE RAMs includes validation in external populations and additional impact analyses to demonstrate that the RAMs can be used with reproducible accuracy and confidence. The present validation study of the IMPROVE RAM represents one of the largest external, multicenter validation studies in this patient population and demonstrates level 2 validation, which suggest that the results can be used in various settings with confidence in their accuracy.¹⁴

Both the original derivation study and the present validation study of the IMPROVE RAM have important and consistent clinical implications. With ≈8 million acutely ill medical patients in the United States who are at risk of VTE,¹ proper VTE risk assessment is important to ensure accurate identification of both at‐risk patients and those patients who may not benefit from the bleeding risks associated with pharmacological prophylaxis, estimated to be between 0.4% and 1.7%.²² The present IMPROVE VTE validation study suggests that at least two thirds of medical patients are at low risk of VTE. Efforts to promote widespread pharmacological thromboprophylaxis, as is currently suggested by national quality organizations, without proper VTE risk assessment may expose a large number of these patients to unnecessary harm from bleeding and other adverse outcomes and to the costs of pharmacological and mechanical VTE prophylaxis.⁶

The strengths of the present study include a large database of electronic health records from 2 large acute‐care hospitals in a setting different from the original derivation cohort population. In addition, standardized abstraction instruments were used by trained personnel to extract the relevant independent clinical risk factors in the validation population. Moreover, the number of VTE events in both cases and controls were sufficiently robust to allow assessment of both the discrimination and the calibration of each VTE risk score of the model. A limitation of the present study is its retrospective design, which may have introduced bias in terms of practice patterns by admission year and hospital. This was minimized by ensuring that the control group was adjusted both for hospital and year of admission. Another limitation is the use of administrative database analysis based on ICD‐9 codes, which may lack specificity in identifying VTE events; however, all VTE events were confirmed by chart review. There is a small possibility that some small number of missing VTE events were overlooked, but given the strict methodology of the abstraction process, this is unlikely and would bias in favor of improved calibration over the original derivation cohort. In addition, difficulties abstracting based on administrative data may have resulted in unintended exclusion of VTE events, although these conservative estimates were based on strict methodology. This appears to be confirmed by the overall lower VTE risk, especially for those with higher IMPROVE VTE risk scores, compared with the derivation population; however this should not have influenced the relative weights and estimates between risk scores. Furthermore, we acknowledge the difficulty of establishing immobility criteria during the abstraction process, but we used electronic nursing notes and physical therapy notes to specify immobility status at a patient‐chart level. The present analysis excluded patients who died during hospitalization; the original analysis did not, which also may have affected overall study estimates. Finally, the stroke population was not included in the present validation study, compared with the derivation cohort, because it was felt this group was sufficiently different to warrant a separate validation study. This approach may have contributed to the lower overall VTE risk seen in the validation population compared with the derivation cohort.

To conclude, the present validation study of the IMPROVE VTE RAM represents one of the largest, multicenter, external validation studies to date of an evidence‐derived and weighted VTE RAM in the hospitalized, acutely ill medical patient population. The IMPROVE RAM is able to incorporate 7 well‐established and easy‐to‐implement clinical risk factors for VTE in this patient population either at admission or during hospital stay. The derivation and validation cohorts revealed good discrimination and calibration for both the overall VTE risk model and the identification of at‐risk patient groups. An IMPROVE VTE risk score of ≤2, which encompassed more than two thirds of the entire cohort, identified a population at low risk of VTE that likely would not benefit from pharmacological prophylaxis. This represents an important clinical advance in the field of VTE risk assessment in the hospitalized, acutely ill medical patient population by improving assessment of at‐risk VTE populations and avoiding unnecessary harm from overuse of prophylaxis. Additional prospective management studies and impact analyses of the IMPROVE VTE RAM should be undertaken in the acutely ill medical patient group.

Supplementary Material

Table S1. ICD-9 Codes for the Study Populations Inclusion Criteria

Table S2. IMPROVE VTE Risk Factors Weights and Definitions

Table S3. VTE by risk score

Table S3. ROC for Derivation Cohort = 0.72