External Validation of the Padua and IMPROVE-VTE Risk Assessment Models for Predicting Venous Thromboembolism in Hospitalized Adult Medical Patients: A Retrospective Single-Center Study in Japan

Abstract

Objectives: To assess the external validity of the Padua and International Medical Prevention Registry on Venous Thromboembolism (IMPROVE-VTE) risk assessment models (RAMs) for predicting venous thromboembolism (VTE) within 90 days of admission among hospitalized medical patients in Japan.

Materials and Methods: A university hospital cohort comprising 3876 consecutive patients ages ≥15 years admitted to a general internal medicine department between July 2016 and July 2021 was retrospectively analyzed using data extracted from their medical records.

Results: A total of 74 VTE events (1.9%), including six cases with pulmonary embolism (0.2%), were observed. Both RAMs had poor discriminative performance (C-index=0.64 for both) and generally underestimated VTE risks. However, recalibrating the IMPROVE-VTE RAM to update the baseline hazard improved the calibration (calibration slope=1.01). Decision curve analysis showed that a management strategy with no prediction model outperformed a clinical management strategy guided by the originally proposed RAMs.

Conclusions: Both RAMs require an update to function in this particular setting. Further studies with a larger-sized cohort, including re-estimation of the individual regression coefficients with additional, more context-specific predictors, are needed to create a useful model that would help advance risk-oriented VTE prevention programs.

Introduction

Venous thromboembolism (VTE) is potentially a fatal condition, with estimates suggesting that up to 30% mortality within a month of diagnosis.¹⁾ VTE commonly develops in both surgical patients without any preventive interventions and hospitalized medical patients. In North America and Europe, up to 3% of hospitalized medical patients develop symptomatic VTE,^2,3) whereas the typically reported incidence in Asia ranges 0.2%–0.9%.⁴⁾ Although incidence rates of VTE in hospitalized general medical patients in Japan have remained unclear, a bi-center prospective, routine survey using lower extremity venous ultrasound found VTE equates to 18% of hospitalized, high-risk, non-surgical patients regardless of symptoms.⁵⁾

To prevent VTE-associated deaths, risk-oriented and individualized pharmacologic and non-pharmacologic prophylaxis strategies have been proposed.⁶⁾ For instance, the American Society of Hematology (ASH) VTE guidelines⁷⁾ for medical patients recommend identifying high-risk patients through risk assessment models (RAMs) and prescribing anticoagulation as prophylaxis depending on the individual’s risks for both VTE and bleeding.⁷⁾ In the ASH guidelines, the Padua RAM²⁾ and the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE)-VTE RAM⁸⁾ have been identified as the two most widely externally validated RAMs. The Padua RAM is an empirically derived 11-item prediction score based on reduced mobility, history of VTE, existing thrombophilia conditions, recent history of trauma and/or surgery, ages ≥70 years, presence of heart and/or respiratory failure, presence of active cancer, acute myocardial infarction or ischemic stroke, acute infection and/or rheumatologic disorder, on-going use of hormonal treatment, and obesity.²⁾ In the Padua RAM derivation cohort, ≥4 scores indicate high VTE risk (incidence without prophylaxis: 11%), whereas zero to three scores are deemed to indicate low VTE risk (incidence 0.3%). In contrast, the IMPROVE-VTE RAM, which is formally derived from an international VTE registry involving over 15,000 patients from 52 hospitals in 12 countries, assesses seven items, namely previous history of VTE, existing thrombophilia conditions, existence of lower extremity paralysis, presence of active cancer, immobilization over a week, intensive care unit (ICU) admission, and ages ≥60 years.⁸⁾ In the derivation cohort, the RAM provides six risk categories based on the score, which can be further simplified into three-risk groups, namely low (scores of 0 and 1; incidence 0.5%), intermediate (scores of 2 and 3; incidence 1.5%), and high (scores ≥4; incidence 5.7%) risk for developing VTE.

Although non-pharmacologic prophylaxis has been generally recommended in Japan, the roles of RAMs and risk-adapted pharmacologic prophylactic interventions have yet to be established.⁶⁾ Additionally, the performance and calibration of the Padua and IMPROVE-VTE RAMs have never been assessed in a Japanese population despite being extensively validated worldwide, suggesting the need for a formal validation before their routinary use in clinical practice.⁹⁾ The current study therefore validated the ability of both RAMs to generally predict VTE in medical in-patients admitted at a tertiary-care university hospital in Japan.

Materials and Methods

Data source and participants

This retrospective, observational cohort study was conducted at Fujita Health University Hospital, a tertiary-care, academic hospital in Japan. The study protocol was approved by the ethics committee of Fujita Health University (approval number HM21-576) and conforms to the provisions of the Declaration of Helsinki. The requirement for informed consent was waived since this study was based on a retrospective review of the data extracted from the medical records.

We retrospectively identified patients who were admitted to our general internal medicine department between July 5, 2016, and July 5, 2021, and extracted data from their electronic medical records. The eligibility criteria were patients ages ≥15 years and were hospitalized for at least three days, whereas the exclusion criteria were patients with surgery and/or trauma, pregnant women, patients on anticoagulation therapy for any reasons at admission, and patients hospitalized for VTE and bleeding.

Predictors and interventions

One of the investigators (DA) extracted the baseline patient characteristics used in the Padua and IMPROVE-VTE RAMs and deep vein thrombosis (DVT) prophylaxis interventions, if any were administered during hospitalization, from the medical records. The baseline characteristics included the presence of lower extremity paralysis, intensive care unit admission, reduced mobility, history of VTE, existing thrombophilia condition, recent history of surgery and/or trauma, age, presence of heart and/or respiratory failure, active cancer, acute myocardial infarction or ischemic stroke, acute infection and/or rheumatologic disorder, on-going use of hormonal treatment, and obesity. Points assigned by each RAM are provided in Tables 1a and 1b. Data were gathered from the registered diagnosis procedure combination codes (i.e., equivalent to hospital admission codes) and the clinical information at admission were individually described. Detailed operational definitions of the presence or absence of specific items are provided in Table 2.

Table 1a Padua RAM used in medical in-patients^a.

Baseline features	Points
Active cancerb	3
Previous VTE	3
Reduced mobilityc	3
Already known thrombophilia conditiond	3
Trauma/surgery within a month	2
Age≥70 years	1
Heart or respiratory failure	1
Acute myocardial infarction or ischemic stroke	1
Acute infection or rheumatologic disorder	1
BMI≥30 kg/m2	1
Hormonal therapy	1

a. Predicted VTE event rates at 90 days are 11% (total score ≥4; note that event rate is 2.2% when receiving adequate in-hospital thromboprophylaxis), and 0.3% (total score<4).

b. Patients with local or distant metastases and/or in whom chemotherapy or radiotherapy had been performed in the previous 6 months.

c. Bedrest with bathroom privileges (due to either patients’ limitations or physician’s order) for at least 3 days.

d. Carriage of defects of antithrombin, protein C or S, factor V Leiden, G20210A prothrombin mutation, and antiphospholipid syndrome.

VTE: venous thromboembolism; RAM: risk assessment model; BMI: body mass index

Table 1b IMPROVE-VTE RAM used in medical inpatients^a.

Baseline features	Points
Previous VTEb	3
Known thrombophiliac	2
Currentd lower limb paralysis	2
Current cancere	2
Immobilized ≥7 daysf	1
ICU or CCU stay	1
Age >60 year	1

a. Expected VTE event rates at 92 days are 8.1% (total score ≥5); 4.8% (total score 4), 1.6% (total score 3), 1.5% (total score 2), 0.6% (total score 1), and 0.4% (total score 0). For the three-risk group model, expected VTE event rates are 5.7% (total score ≥4), 1.5% (total score 2–3), and 0.5% (total score 0–1).

b. Previous VTE and age were both known to have occurred 3 months prior to VTE; other patient factors were known to have been present at or during hospital admission.

c. Carriage of defects of antithrombin, protein C or S, factor V Leiden, G20210A prothrombin mutation, and antiphospholipid syndrome.

d. “Current” indicates present at or during current admission.

e. Patients with local or distant metastases and/or in whom chemotherapy or radiotherapy had been performed in the previous 6 months.

f. Days immobile immediately prior to and during hospital admission.

CCU: coronary care unit; ICU: intensive care unit; IMPROVE: International Medical Prevention Registry on Venous Thromboembolism; RAM: risk assessment model; VTE: venous thromboembolism

Table 2 Main demographic characteristics and distribution of patients in the previous studies and our study.

Category/Variable (%)a	Padua cohort	IMPROVE cohort	Validation cohort
			Total	Patients without VTE	Patients with VTE	P valueb
Study characteristics
No. of patients	1180	15156	3876	3802	74
Design of study	Prospective	Retrospective	Retrospective
Line of care	Tertiary	Secondary/ Tertiary	Tertiary
No. of centers	Single-center	Multicenter	Single-center
Setting	Internal medicine	Internal medicine	Internal medicine
Median duration of hospitalization [IQR], days	8.3 [±5.4]c	7 [5, 13]	14.0 [8.0, 27.0]	14.0 [8.0, 27.0]	25.0 [17.0, 66.0]	<0.001
ICU admission	ND	741 (4.9)	656 (16.9)	627 (16.5)	29 (39.2)	<0.001
Median ICU stay [IQR], days	ND	ND	0 [0, 0]	0 [0, 0]	0 [0, 3.8]	<0.001
Mortality	113 (9.6)	163 (11.0)	309 (8.0)	295 (7.8)	14 (18.9)
VTE	37 (3.1)	143 (0.9)	74 (1.9)
PE	15 (1.3)	76 (0.5)	6 (0.2)
Patient characteristics
Men	555 (47.0)	7578 (50.0)	2076 (53.6)	2043 (53.7)	33 (44.6)	0.13
Median age [IQR], years	75.09 [±12.7]c	68 [52, 79]	77 [63.0, 85.0]	77.00 [63.0, 85.0]	81.50 [73.0, 87.0]	0.002
Age≥70 years	779 (66.0)	5398 (35.6)d	2609 (67.3)	2546 (67.0)	63 (85.1)	0.001
Median BMI [IQR], kg/m2	ND	ND	20.1 [17.3, 23.0]	20.1 [17.3, 23.0]	19.9 [17.2, 22.7]	0.45
BMI ≥30 kg/m2	76 (6.4)	2,421 (16.0)e	147 (3.8)	146 (3.8)	1 (1.4)	0.53
Preventive intervention characteristics
Pharmacological and/or non-pharmacological therapy	ND	6062 (44.0)f	750 (19.3)	720 (18.9)	30 (40.5)	<0.001
Pharmacological intervention alone	303 (25.7)g	5,979 (39.4)	211 (5.4)	198 (5.2)	13 (17.6)	<0.001
Non-pharmacological intervention alone	ND	1,661 (11.0)	309(8.0)	302 (7.9)	7 (9.5)	0.662
Bleeding	5 (0.4)	ND	46 (1.2)	41 (1.1)	5 (6.8)	0.002
Active cancer	234 (19.8)	1,735 (11.4)	155 (4.0)j	148 (3.9)	7(9.5)	0.027
Previous VTE	46 (3.9)	551 (3.6)	68 (1.8)k	64 (1.7)	4(5.4)	0.04
Immobilization	272 (23)	1169 (7.7)	2884 (74.4)l	2820 (74.2)	64 (86.5)	0.015
Already known thrombophilic condition	3 (0.3)	21 (0.1)	4 (0.1)m	4 (0.1)	0 (0.0)	1.0
Trauma/surgery within a month	31 (2.6)	ND	26 (0.7)n	26 (0.7)	0 (0.0)	1.0
Heart or respiratory failure	254 (21.5)	4312 (28.5)h	1741 (44.9)o	1699 (44.7)	42 (56.8)	0.045
AMI or ischemic stroke	12 (1)	ND	21 (0.5)	19 (0.5)	2 (2.7)	0.06
Acute infection or rheumatologic disorders	220 (18.6)	1883 (12.4)i	3129 (80.7)p	3064 (80.6)	65 (80.7)	0.14
Hormonal therapy	13 (1.1)	252 (1.7)	25 (0.6)q	24 (0.6)	1 (1.4)	0.38
Leg paralysis	ND	309 (2)	345 (8.9)r	338 (8.9)	7 (9.5)	0.84
Central venous catheter	ND	1228 (8.1)	565 (14.6)	536 (14.1)	29 (39.2)	<0.001

AMI: Acute myocardial infarction; BMI: body mass index; CCU: coronary care unit; ICU: intensive care unit; IMPROVE: International Medical Prevention Registry on Venous Thromboembolism; IQR: interquartile range; ND: no data; PE: pulmonary embolism; RAM: risk assessment model; VTE: venous thromboembolism

a. Data are presented as n (%) otherwise indicated.

b. Fisher’s exact test or chi-squared test was used to analyze the between-group differences.

c. Estimates from original data.

d. Elderly patients were defined as aged ≥75 years.

e. Obesity was not explicitly defined based on BMI.

f. In the original report of the parental cohort, 7,640 patients (50%) were reported to have received pharmacologic and/or mechanical VTE prophylaxis.²³⁾

g. 238 (20.2%) received adequate pharmacological prophylaxis, whereas in 65 patients (5.5%) inadequate prophylaxis was performed.

h. Patients with a prior history of congestive heart failure or respiratory failure.

i. Patients with infection only.

j. Patients actively treated with anticancer drug or those with multiple metastases or under palliative care.

k. Patients with an explicit description of a history of “VTE,” “DVT,” or “deep vein thrombosis” in their medical record.

l. Patients with an explicit description of immobilization such as “difficulty in moving,” “inability to move,” “in need of full assistance,” or “bedridden,” etc. in their medical record.

m. Patients with a history of antiphospholipid syndrome, antithrombin III deficiency, protein C or S deficiency, etc.

n. Patients with a history of trauma or surgery in their medical record.

o. Patients with oxygen saturation of <90% on oxygen therapy or clinically diagnosed congestive heart failure.

p. Patients with an explicit description of acute infection, fever of unknown origin, or current or prior rheumatologic disorders.

q. Patients on estrogen-based hormonal therapy on admission.

r. Patients with an explicit description of leg paralysis, such as “hemiplegia,” “paraplegia,” or “lower extremity paralysis,” in their medical records.

The extracted data on DVT prophylaxis included both pharmacologic and non-pharmacologic interventions. Pharmacologic prophylaxis was based exclusively on unfractionated heparin, typically 10,000 units per day. We further determined whether a foot pump was used as a non-pharmacological prophylaxis.

Risk stratification

We initially reported the classification of patients into seven risk categories for developing VTE according to the total scores of the Padua RAM (0, 1, 2, 3, 4, 5, and ≥6 points) and six risk categories according to the total scores of the IMPROVE-VTE RAM (0, 1, 2, 3, 4, and ≥5 points). Thereafter, following the empirically defined risk-stratified groups reported in the derivation studies, patients were categorized into two groups, namely those having low (scores 0–3) and high-risk (scores ≥4) according to the Padua RAM, and three groups, namely those having low (scores 0 and 1), intermediate (scores 2 and 3), or high-risk (scores ≥4) according to the IMPROVE-VTE RAM.

Outcomes

The outcome of interest was the cumulative incidence of VTE at 90 days post-admission. One of the investigators (DA) blinded to both the patient characteristics and assigned scores for both RAMs extracted the data from the radiology or ultrasound reports and defined VTE as any DVT and/or pulmonary embolism (PE) confirmed via contrast-enhanced computed tomography or lower extremity venous echography. Only suspected patients (e.g., symptoms or abnormal laboratory findings) were worked up for VTE by doppler ultrasonography and/or contrast-enhanced computed tomography; we did not routinely screen or perform surveillance for asymptomatic patients during hospitalization. For sensitivity analysis, the cumulative incidence of diagnosed VTE and sudden unexplained death within 90 days as a composite outcome was jointly analyzed. Sudden unexplained death was defined as an unexpected sudden death described as “unexpected sudden death” and/or “unknown cause of death” in the medical records. Autopsy was not performed in all such cases.

Sample size

Despite targeting a minimum sample size of 100 events as recommended by experts,¹⁰⁾ our available cohort, which covered a five-year period, had only 74 events, meagerly satisfying a smaller threshold of >50.

Statistical analysis

This study followed the recommended framework for the external validation of a prognostic model.¹¹⁾ Continuous variables were presented as medians and interquartile ranges, whereas categorical variables were presented as numbers and percentages. The Harrell’s C-statistic and Gönen and Heller’s K-statistic were used to measure concordance, whereas the Royston–Sauerbrei D statistic (R²_D)¹¹⁾ was used to measure variation. To visualize the extent of discrimination, each risk group stratified according to the RAMs were plotted using the Kaplan–Meier method.

To examine calibration, we estimated the regression coefficient of the linear predictor (i.e., linear combination of a set of coefficients and predictive variables) on the original RAMs in the Cox model as the calibration slope and assessed its model fit. To primarily visualize the calibration, the model-derived predicted mean event curves to the Kaplan–Meier curves for the observed events was visually compared; then, standard calibration plots were constructed.

The cumulative VTE incidence for the assigned individual scores and the originally estimated set of regression coefficients determined using the Cox model in the derivation cohort were available for the IMPROVE-VTE RAM only. Thus, the baseline hazard function was recalibrated using the Royston–Parmar flexible parametric survival models to update the original IMPROVE-VTE RAM (see Appendix Text for details).¹¹⁾ The calibration using the standard calibration plots was visualized and the updated predicted mean event curves to the Kaplan–Meier curves was visually compared.

Comparing the strategies based on the original RAM (updated RAM for the IMPROVE-VTE only) and among default management strategies (i.e., some interventions preventing VTEs for all and no intervention for all without RAM-based risk stratifications), decision curve analysis was conducted.¹²⁾ We estimated the difference between the proportion of “true positives” (i.e., appropriate implementation of interventions for patients who developed VTE) and the proportion of “false positives” (i.e., unnecessary implementation of interventions for patients who did not develop VTE) as the net benefit, weighted by the odds of provider and/or patient preference regarding whether to implement the interventions.

All statistical analyses were performed using Stata SE version 17.0 (Stata Corp, College Station, TX, USA) or R version 4.1.2 [R Core Team (2021)]. To obtain cluster-robust standard errors and address the repeated observations among multiple inclusions in the same patients, a clustered sandwich estimator was used. All analyses used two-tailed P-values with the level of significance set at <0.05.

Results

Patient characteristics

Among the 4654 patients admitted to our department during the study period, 3876 were eligible (Fig. 1). The characteristics of whom are shown in Table 2. Compared to the patients included in the Padua and IMPROVE-VTE deviation cohorts, our study population had more acutely and critically ill patients but had similarity in terms of average age, sex distribution, cumulative VTE incidence, and mortality rates. Particularly, our cohort had longer average hospital stays (medians of 14 days vs. 8 and 7 days, respectively), frequent ICU admissions (16.9% vs. frequency not reported and 4.9%, respectively), more patients with immobility (74.4% vs. 23.0% and 7.7%, respectively), cardiac or respiratory failure (44.9% vs. 21.5% and 28.5%, respectively), infectious or rheumatic diseases (80.7% vs. 18.6% and 12.4%, respectively), and paralytic lower extremities (8.9% vs. frequency not reported and 2.0%, respectively). In contrast, fewer patients in our cohort received any form of VTE prophylaxis (19.3% vs. 25.7% and 44.0%, respectively), had prior risk factors, such as active cancer (4.0% vs. 19.8% and 11.4%, respectively), history of VTE (1.8% vs. 3.9% and 3.6%, respectively), and had obesity defined as body mass index≥30 kg/m² (3.8% vs. 6.4% and 16.0%, respectively).

Fig. 1 Patient flow diagram.

a. 211, 509, and 750 patients used unfractionated heparin only, foot pump only, and unfractionated heparin and/or foot pump, respectively.

Outcomes

Overall, 74 of the 3876 (1.9%) patients developed VTE (68 with DVT alone [15 with proximal DVT and 53 with distal DVT] and 6 with PE with or without DVT [all with non-massive PE]) during a median follow-up period of 14 days (interquartile range [IQR]: 8 to 27 days; total 81,001 patient-days) with an overall incidence rate of 9.14/10,000 patient-days (Table 2). Compared with patients who did not develop VTE, those who developed VTE were more likely to stay longer in the hospital (median 25 vs. 14 days; P<0.001), to be admitted in the ICU [39.2% (29/74) vs. 16.5% (627/3802); P<0.001], to be older (median, 82 vs. 77 years; P=0.002), to have active cancers [9.5% (7/74) vs. 3.9% (148/3802); P=0.027], to have a prior history of VTE [5.4% (4/74) vs. 1.7% (64/3802); P=0.04], to be immobile [86.5% (64/74) vs. 74.2% (2820/3802); P=0.04], to have heart or respiratory failure [56.8% (42/74) vs. 44.7% (1699/3802); P=0.045], and to receive central venous catheters [39.2% (29/74) vs. 14.1% (536/3802); P<0.001]. Among the 750 patients (19.4% of the entire population) who received any form of prophylaxis, only 30 (4.0%) developed VTE, whereas among the 3126 without prophylaxis, only 44 (1.4%) developed VTE.

External validation of the RAMs

The score distributions of both RAMs overlapped substantially between the patients who did and did not develop VTE (Appendix Fig. 1). The median scores of the Padua RAM were six (IQR, 5–6) and five (IQR, 3–6) for the patients with and without VTE diagnosis, whereas the median scores of the IMPROVE-VTE RAM were two (IQR, 2–3) and two (IQR, 1–2) for those with and without VTE diagnosis, respectively.

Discrimination

Although the full seven-risk-group Padua RAM appeared to discriminate between the risk groups (Log-rank Chi-squared=17.29; P=0.008), the visual assessment of the Kaplan–Meier plots did not show a stepwise increase in the VTE rates as scores increased (Appendix Fig. 2). The C-statistic and K-statistic of the full seven-risk-group Padua RAM were 0.64 (95% confidence interval [CI]: 0.58–0.69) and 0.67 (95%CI: 0.61–0.73), respectively, with an explained variation statistic R²_PM of 0.17 (95%CI: 0.06–0.30). The originally proposed two-risk-group Padua RAM also had excellent discriminative performance (Log-rank Chi-squared=14.38; P<0.001), although the two-risk groups yielded similar VTE incidences beyond 60 days according to the Kaplan–Meier plots (Fig. 2).

Fig. 2 Kaplan–Meier plots for the two-risk group Padua RAM (left panel) and the three-risk group IMPROVE-VTE RAM (right panel).

IMPROVE: International Medical Prevention Registry on Venous Thromboembolism; RAM: risk assessment model; VTE: venous thromboembolism

Similarly, the full six-risk-group IMPROVE-VTE RAM appeared to discriminate between the risk groups (Log-rank Chi-squared=38.81; P<0.001); however, the visual assessment of the Kaplan–Meier plots did not continually show a stepwise increase as scores increased in the VTE rates (Appendix Fig. 2). The C-statistic and K-statistic of the full six-risk-group IMPROVE-VTE RAM were 0.64 (95%CI: 0.59–0.69) and 0.60 (95%CI: 0.56–0.64), respectively, with an explained variation statistic R²_PM of 0.15 (95%CI: 0.06–0.25). Similarly, the originally proposed three-group IMPROVE-VTE RAM had excellent discriminative performance (Log-rank Chi-squared=16.59; P<0.001); however, two of the three-risk groups (high- and intermediate-risk groups) showed overlapping cumulative incidence curves according to the Kaplan–Meier plots (Fig. 2).

Calibration

Predicted risks according to the full seven-risk-group Padua RAM were systematically limited and had insufficient variety (calibration slope=1.54; 95%CI, 1.26–1.86; P<0.001 and Chi-squared of model misspecification=53.17; P<0.001). Visual assessment of the limited observed and predicted event plots based on the originally proposed two-risk-group Padua RAM showed that the observed event rates were generally higher than that of the corresponding predicted event rates (Fig. 3).

Fig. 3 Calibration plots for the two-risk group Padua RAM (left panel) and the three-risk group IMPROVE-VTE RAM (right panel).

IMPROVE: International Medical Prevention Registry on Venous Thromboembolism; RAM: risk assessment model; VTE: venous thromboembolism

Similarly, predicted risks according to the full six-risk-group IMPROVE-VTE RAM were too low and had insufficient variety as well (calibration slope=1.44; 95% CI, 1.24–1.68; P<0.001 and Chi-squared of model misspecification=49.23; P<0.001). Visual assessment of the observed and predicted event curves for the full six-risk-group IMPROVE-VTE RAM, except for the single group with the score of four, showed that the observed event rates were higher than that of the corresponding predicted event rates (Appendix Fig. 3). The results were generally congruent after stratifying patients into the originally proposed three-risk groups (Fig. 3).

Update of the IMPROVE-VTE RAM

Recalibrating the baseline survival function derived from the original model increased the predicted mean event rates, which constructed the updated mean event curves that were generally higher than that of the original predicted mean event curves and visually comparable to the Kaplan–Meier survival curves of the observed events in our cohort (Appendix Fig. 3). This update appeared to correct the poor calibration of the full six-risk-group IMPROVE-VTE RAM (updated calibration slope=1.02; 95%CI, 0.04–1.99; P=0.051). The calibration slopes of the updated RAM also confirmed the corrected calibration (Appendix Fig. 4).

Decision curve analysis

Decision curve analysis based on the two-risk-model Padua RAM (i.e., classifying patients into high- and low-risk groups) showed that the net benefit obtained from the Padua RAM-based management outperformed the “intervention for none” strategy (without RAM) but was inferior to the “intervention for all” strategy (without RAM) over nearly the entire theoretically conceivable range of threshold probability (i.e., a range of 0.3–7.5% risk of developing VTE) (Appendix Fig. 5). Similarly, the strategies based on the full six-risk-group and originally proposed three-risk-group IMPROVE-VTE RAM outperformed the “intervention for none” strategy but was inferior to the “intervention for all” strategy. In contrast, the risk-stratified strategy based on the updated six-risk-group IMPROVE-VTE RAM outperformed all other strategies when the threshold probability was ≥6.6%. Nevertheless, the maximal net benefit was only 0.008 (i.e., 8 more cases appropriately managed per 1000 cases) over the “intervention for all” or “intervention for none” strategy when the threshold probability was 8.1%.

Sensitivity analysis

After additionally considering 20 sudden unexplained death cases as the composite outcome events, the overall cumulative incidence increased to 2.4% (94 events out of 3876 patients) with an overall incidence rate of 11.60/10,000 patient-days. Both the Padua and IMPROVE-VTE RAMs still had limited discriminative performance and underestimated the event risk, which were consistent with the results of the main analysis (Appendix Table 1 and Appendix Figs. 6–8).

Discussion

This retrospective, single-center, external validation study aimed to assess the predictive ability of the Padua and IMPROVE-VTE RAMs on an individual’s risk to develop VTE in general medical setting of a tertiary-care university hospital in Japan. Accordingly, our results comparably showed that both RAMs had poor discriminative performance and similarly underestimated the risk of VTE. However, the discriminative performance of the IMPROVE-VTE RAM improved after recalibrating the model to have a revised baseline hazard function. Decision curve analysis showed that the two-risk-group (high- vs. low-risk) Padua RAM and the three-risk-group (high- vs. intermediate- vs. low-risk) IMPROVE-VTE RAM, wherein these RAMs were originally designed for application in clinical practice, were outperformed by the reference strategies without RAM. The updated full six-risk-group IMPROVE-VTE RAM outperformed the reference strategies without RAM, albeit with only marginal incremental net benefit.

The case-mix of our cohort explicitly differed from those of the derivation cohorts.^2,8) Our cohort was averagely sedentary and frequently included critically ill patients based on clinical characteristics (i.e., higher frequencies of immobility and hemiparesis and higher rates of cardiac or respiratory failure and infectious or rheumatic diseases). This series of distributions was expected considering that acute infections, such as aspiration pneumonia, urinary tract infections, and sepsis with exacerbated heart failure in bedridden elderly patients, typically nursing home residents with a prior history of stroke, were common causes of admissions to the general internal medicine department of our hospital.

Our study has two strengths: first, this has been the first external validation study of the Padua and IMPROVE-VTE RAMs using a full cohort design conducted in East Asia. Despite its retrospective design wherein the data were derived from clinical practice, our study provides more accurate insights into how these RAMs perform in general medical in-patients across East Asia than previous studies with a same case-control design,^13,14) where discrimination and calibration are formally undeterminable. Additionally, the standard approach for recalibrating the baseline hazard function to update the IMPROVE-VTE RAM, which were partially adapted for our local setting, was utilized.

Several limitations of the current study warrant discussion. First, this study was a retrospective analysis based on a single-center experience, which could limit generalizability of our findings. Reliable and accurate data extractions could only be achieved with an a priori formulated and standardized set of operational definitions for the baseline features. Second, even with a total of 3876 patients, the sample size with only 72 events was still small. Notably, a minimum of 100 events is recommended as the target number for the external validation of a RAM with a time-to-event outcome.¹⁵⁾ Although the 1.9% incidence of symptomatic VTE in our study was almost comparable to that reported studies on general medical in-patients in Western countries (1%–3%),^2,3) our failure to follow-up patients after discharge, as these studies did, may have resulted to missed cases of post-discharge VTE given that late-onset post-discharge VTE events were quite common.²⁾ Moreover, censorship due to discharge and mortality from causes other than VTE can affect the accurate estimation of the cumulative incidence of VTE.¹⁶⁾ Additionally, the few events precluded a full model update, including re-estimation of the individual regression coefficients or extension of the model by adding new prognostic factors in addition to recalibrating the model.¹¹⁾ Third, we failed to account for the effects stemming from VTE prophylaxis, such as unfractionated heparin administration and non-pharmacological interventions. However, experts suggest that treatment effects are negligible in prognostic model research given that such effects are typically insignificant.¹⁷⁾ Indeed, the derivation study of the original IMPROVE RAM⁸⁾ completely ignored these preventive effects, as we did in the current study. Similarly, in the original Padua study, majority of cases were ignored due to under-dosed pharmacological prophylaxis, which was defined as <15,000 units per day of unfractionated heparin.²⁾

The poor discriminative performance and limited net benefit estimated for both RAMs in this study is common considering that previous studies with a cohort design reported a comparable C-index of 0.58–0.64 and 0.63–0.77 for the Padua RAM^18,19) and IMPROVE-VTE RAM,^8,19,20) respectively. Given that no other cohort-type studies have formally assessed the calibration measures or performed decision curve analysis to assess the net benefit as our study did, a cross-study comparisons of these metrics are impossible. As mentioned, external validation based on case-control studies,^13,14,21) though economical and efficient, cannot accurately estimate discrimination and calibration measures.

Conclusion

Considering the poor performance and limited net benefit observed herein, the current studied RAMs are insufficient for patient stratification and determination of VTE prophylaxis. Future research needs to substantially update these RAMs. A candidate approach would incorporate additional baseline parameters into the currently available RAMs as the baseline model,¹¹⁾ which would include already well-assessed prognostic factors, such as C-reactive protein, D-dimer, heart rate, platelet and white blood cell counts, body temperature, and leg edema.⁹⁾ Conducting a prospective, multi-institutional study with comparable levels of care and similar risk profiles, including a sufficiently large number of patients to achieve the target number of VTE events,²²⁾ would advance VTE prevention based on evidence and risk stratification.

Disclosure Statement

All authors have no conflict of interest.

Author Contributions

Study conception: all authors

Data collection: DA

Analysis: DA, TT

Investigation: DA, TT

Manuscript preparation: DA, TT

Critical review and revision: all authors

Final approval of the article: all authors

Accountability for all aspects of the work: all authors

Supplementary Information

Supplementary materials are available at the online article sites on J-STAGE and PMC.