A systematic review and meta-analysis of risk prediction models for post-thrombotic syndrome in patients with deep vein thrombosis

Abstract

Objective

This systematic review and meta-analysis aimed to systematically evaluate the prediction models for the risk of post-thrombotic syndrome (PTS) in deep vein thrombosis (DVT) patients.

Methods

This systematic review and meta-analysis was guided by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA). A systematic search on the following electronic database: PubMed/MEDLINE, EMBASE, and Cochrane Library, and Chinese databases such as WANFANG and CNKI was conducted to look for relevant articles based on the research question. The risk of bias for each studies included was carried out based on Prediction Model Risk of Bias Assessment Tool (PROBAST).

Results

We identified 10 studies that developed a total of 13 clinical prediction models for PTS risk in DVT patients, 3 models were externally validated, 2 models were temporally validated. The top 5 predictors were: BMI (N = 9), Varicose vein (N = 6), Baseline Villalta Score (N = 6), Iliofemoral thrombosis (N = 5), and Age (N = 4). The high risk of bias was from the analysis domain, which the number of participants and selection of predictors often did not meet the requirements of PROBAST. A random-effects meta-analysis of C-statistics was conducted, the pooled discrimination was C-statistic 0.75, 95%CI (0.69, 0.81).

Conclusion

Among the 13 PTS risk prediction models reported in this study, no prediction model has been applied to clinical practice due to the lack of external validation. In the development of prediction models, most models were not standardized in data analysis. It is recommended that future studies on the design and implementation of prediction models refer to Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) and PROBAST.

Graphical abstract

1. Introduction

Post-thrombotic syndrome (PTS) is the most common comorbidity of Deep vein thrombosis (DVT), the main clinical manifestation of PTS is chronic venous insufficiency. Studies have reported that even with standardized anticoagulation therapy, the proportion of PTS after proximal DVT is still as high as 20%–50 %, and about 5–10 % of them can develop into severe PTS [1]. PTS, recurrent venous thromboembolism (VTE) and chronic thromboembolic pulmonary hypertension (CTEPH) are the three important long-term adverse outcomes of VTE. Although PTS is not fatal, it can cause serious medical, social and economic consequences [2,3]. There are many risk factors of PTS, but there is a lack of simple and effective treatment, and its prevention mainly depends on long-term management after DVT. Prevention is the key to the comprehensive management of PTS; however, the accurate identification of high-risk patients is still a challenge [4,5].

Risk prediction model refers to the use of multivariate models to estimate the probability of having a disease or the probability of having a future outcome [6]. An accurate risk prediction model can facilitate the provision of comprehensive health education and optimize anticoagulation therapy for high-risk PTS patients, thereby effectively preventing severe PTS and reducing medical expenses. Currently, several studies have reported prediction models for PTS; however, these models exhibit limited accuracy and specificity in their predictions. Furthermore, the absence of external validation has hindered the application of any prediction model in clinical practice. External validation is necessary to assess the reproducibility and generalization of a model and is necessary before any prediction model can be applied, as prediction models often perform worse in external validation than in development [7,8].

In this context, we aimed to conduct a systematic review and meta-analysis by comprehensively searching for studies on PTS risk prediction models in patients with DVT, so as to assess the predictive performance of the models and provide a basis for the clinical application of PTS risk models.

2. Methods

Systematic searches, data collection and reporting for this systematic review and meta-analysis were guided by Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) [9], the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) [10]. The study protocol has been registered on the International Prospective Register of Systematic Review (PROSPERO). The registration number is CRD42023412881. The study was conducted based on existing literature, hence patients and public were not involved in this study.

2.1. Search strategy

We searched the following electronic database: PubMed/MEDLINE, EMBASE, and Cochrane Library, and Chinese databases such as WANFANG and CNKI were systematically searched to include Chinese literature (last search conducted on March 30, 2023). The search strategy and protocol were designed in consultation with clinician. No restriction was imposed on language, publication date and status, and duration of follow-up. The search strategies are shown in the supplementary file 1.

2.2. Study selection

Studies were eligible if they: (1) Investigated the risk of PTS in patients with DVT; (2) A prediction model for the risk of PTS in DVT patients was constructed; (3) Externally validate or update the prediction model for the risk of PTS in DVT patients. Exclusion criteria were as follows: (1) Prediction models of other diseases; (2) Only the risk factors of PTS in DVT patients were analyzed, and no prediction model was developed; (3) The prediction model was not correctly constructed, such as incorrect statistical methods or missing key steps; (4) Review articles.

After all searches were completed, a first round of selection was made by two reviewers based on titles and abstracts to eliminate duplicates, and then carried out the full-text screening to select the studies that meet the inclusion criteria.

2.3. Data extraction and risk of bias assessment

Data extraction was conducted independently by the two reviewers, and literature with different opinions wad discussed or the data will be extracted again by the third reviewer. The Excel software was used to prepare and fill in the data extraction form. The following contents were extracted: Author, Publication year, Study design, Country or region, Recruitment period, Diagnostic scale, Follow-up time, Development sample size, Validation sample size, Best performing model, Predictors, Outcome, Internal validation methods, External validation methods, The incidence of PTS. To assess the performance of the predictive model, measures of discrimination, calibration, and classification were extracted. The risk of bias for each study included was carried out based on Prediction Model Risk of Bias Assessment Tool (PROBAST) [11].

2.4. Statistical analysis

The meta-analysis was conducted using RevMan 5.3 software and STATA 16.0. Cochran's Q test and I² statistic will be used to assess heterogeneity levels of studies. The random effects model was adopted to compare the studies.

3. Result

A total of 104 relevant articles were identified by a systematic search of the database. After reading the title, abstract and full text, 90 articles were eliminated, and 2 English articles and 2 Chinese articles were added by manual search. Finally, a total of 10 articles were included in this study (The flow chart shown in Fig. 1).

Fig. 1.

The flow chart.

Among the 10 studies, 2 studies used machine learning to develop prediction models [12,13], 1 study used COX regression [14], and the remaining 7 studies used Logistic regression to develop models [[15], [16], [17], [18], [19], [20], [21]]. The Anat Rabinovich, 2020 [17] study was the external validation of the prediction model developed by Anat Rabinovich, 2018 [18], while Anat Rabinovich, 2020 [17]study also updated the original model to form a new prediction model. And the Tao Yu, 2022 [12] developed 4 prediction models by machine learning methods, so a total of 13 PTS risk models were developed in 10 studies. Among all prediction models, 3 models were externally validated, 2 models were temporally validated, and all of them were internally validated except Hao Huang, 2018 [14]. Five studies were based on Chinese populations, and the remaining five studies were based on European/Caucasian populations (Netherlands, Switzerland, Canada and the United States). Detailed characteristics of the studies were shown in Table 1.

Table 1.

General characteristics of the included studies in the systematic review of PTS prediction models.

Study	Study designs	Population of Development (Sample size)	Study period	Predictors	Outcome	Internal validation	External validation, population (Sample size)
Tao Yu, 2022 [12]	Prospective Cohort	America (ATTRACT database) (550)	December 2009–December 2014	Extreme gradient boosting (XGBoost)： Diabetes mellitus, Baseline villalta Score, BMI, Previous VTE, High cholesterol, Weight, Treatment type.	Developed and external validated four prediction model for PTS risk by machine learning.	10 fold cross-validation	External validation, Chinese cohort (117).
				Logistic regression: Baseline villalta Score, Diabetes mellitus, BMI, Previous VTE, COPD, Treatment Type, High Cholesterol.
				Random forest: Weight，Baseline villalta Score, BMI, Diabetes mellitus, Inpatient qualify DVT, DVT leg, treatment type
				Gradient boosting decision tree (GBDT): Baseline villalta Score, Previous VTE, Diabetes mellitus, BMI, Weight, High Cholesterol, Treatment Type,
Lijun Zhu, 2022 [13]	Retrospective Cohort	China (518)	December 2018–December 2019	Proximal DVT, Recurrent DVT, Age, Male sex, History of varicose veins.	Developed a prediction model for PTS after DVT.	5 fold cross -validation.	None
Hao Huang, 2018 [14]	Retrospective Cohort	China (209)	January 2013–December 2014	Iliac Vein Compression Syndrome, Occlusion, Residual Iliac-femoral vein thrombosis, Residual Femoral-Popliteal vein thrombosis, Insufficient Anticoagulation.	Developed of APTSD score prediction model for PTS risk in DVT patients.	Not reported	Temporal validation, Chinese cohort (102).
Jiantao Zhang, 2022 [15]	Prospective cohort	China (540)	June 2014–December 2016	Ilio-femoral DVT, Active cancer, History of chronic venous insufficiency, Previous venous thromboembolism, Chronic kidney disease, Duration of compression therapy <6 months.	Developed a prediction model for PTS risk in DVT patients	Bootstrap	Temporal validation, Chinese cohort (268).
Peng Qiu, 2021 [16]	Retrospective, case-control study	China (210)	June 2016–June 2018	The number of signs and symptoms, Male sex, Varicose vein history, BMI, Chronic DVT.	Developed a prediction model for PTS risk in DVT patients. Externally validated the SOX-PTS predictive model, and the SWITCO-PTS predictive model in their set.	Not reported.	Temporal validation, Chinese cohort (90).
Anat Rabinovich，2020 [17]	Prospective Cohort	America (ATTRACT database) (691)	December 2009–December 2014	More extensive, BMI≥35, Baseline villalta score, Age.	Externally validated the SOX-PTS score for estimating the risk of developing PTS, moderate to severe PTS, and severe PTS, in patients with proximal DVT.	Bootstrap	This was an external validation study with model updates and the addition of an age variable.
Anat Rabinovich，2018 [18]	Prospective Cohort	Canada/America (SOX Trial database) (762)	June 2004–February 2010	Iliac DVT, BMI≥35，Baseline villalta score.	Developed a prediction model for PTS after DVT.	Bootstrap	None
Marie Méan, 2018 [19]	Prospective Cohort	Switzerland (SWITCO65+ database) (276)	September 2009–December 2013	Age>75 y, Concomitant antiplatelet/NSAID therapy, Multi-level thrombosis, Prior varicose vein surgery, Number of leg signs and symptoms.	Developed of prediction model for PTS risk in >65 y DVT patients. Externally validated the SOX-PTS predictive model in their set.	Bootstrap	None
Elham E. Amin, 2018 [20]	Prospective Cohort	Netherlands (451)	June 2003–June 2013	Baseline model: Age>56, BMI>30, Varicose veins, Smoking, Female sex, Iliofemoral thrombosis, History of DVT. Secondary model: Age>56, BMI>30, Varicose vein, Smoking, Residual vein obstruction.	Developed a two-step model consisting of a model to be applied at baseline to predict the probability of developing PTS at 6 months, and a model to be applied at 6 months to predict the probability of PTS 24 months after initial thrombosis for those patients who did not develop PTS till then.	Bootstrap	External validation, Italy cohort (1107).
Tian'an Huang， 2022 [21]	Retrospective Cohort	China (204)	June 2016–June 2018	BMI＞24, Duration of disease＞14 days, History of varicose veins, Iliac DVT, Thrombus removal of level Ⅲ.	Developed a prediction model for PTS after DVT.	Bootstrap	None

4. Predictors

Most models included between 5 and 7 predictor (N = 11), one model included 4 predictors, and one model included 3 predictors, which was the least included predictor among all models. Among the 31 predictors involved in this study, the top 5 were: BMI (N = 9), varicose vein (N = 6), Baseline Villalta Score (N = 6), iliofemoral thrombosis (N = 5), and age (N = 4). The specific conditions of the included predictors are presented in Table 1.

4.1. Model performance

From the review, not all studies used the same evaluation metrics to report model performance, so we only reported the discrimination, which was the Area under the receiver operating curve (AUC) or C-statistics of the model. Calibration plots were used to evaluate the Calibration of most models. Table 2 showed that the discrimination of most prediction models was between 0.7 and 0.8, the worst performing model had a discrimination of 0.63, and the best performing model used COX regression, and its discrimination was 0.825. Tao Yu, 2022 [12] and Lijun Zhu, 2022 [13] developed prediction models by machine-learning. Tao Yu, 2022 [12] reported four machine learning algorithm models, and the best performance was XGBOOST and GBDT. Lijun Zhu, 2022 [13] used Random forest (Table 2).

Table 2.

The performance of prediction model.

Study	Best performing model	Area under the curve (95%CI)/C-value (95%CI)	Incidence of PTS in the development cohort
Tao Yu, 2022 [12]	Extreme gradient boosting (XGBOOST)	0.77 (0.74,0.80)	58.90 %
	Gradient boosting decision tree (GBDT)	0.77 (0.74,0.80)
Lijun Zhu, 2022 [13]	Random forest	0.722	21.81 %
Hao Huang, 2018 [14]	Cox regression analysis	0.825 (0.747,0.903)	47.70 %
Jiantao Zhang, 2022 [15]	Logistic regression	0.773 (0.699–0.848)	14.07 %
Peng Qiu, 2021 [16]	Logistic regression	0.724	42.00 %
Anat Rabinovich, 2020 [17]	Logistic regression	0.63 (0.59, 0.67)	47.00 %
Anat Rabinovich, 2018 [18]	Logistic regression	0.65 (0.64, 0.67)	12.53 %
Marie Méan, 2018 [19]	Logistic regression	0.77 (0.71–0.82)	58.30 %
Elham E. Amin, 2018 [20]	Baseline model: Logistic regression	0.67 (0.61, 0.73)	45.70 %
	Secondary model: Logistic regression
Tian'an Huang, 2022 [21]	Logistic regression	0.825 (0.759, 0.892)	31.90 %

4.2. Risk of bias assessment

Of all the included studies, except for two studies with low risk, the remaining nine studies had high risk (Fig. 2 (a)). The elevated risk was from the analysis domain, which the number of participants and selection of predictors often did not meet the requirements. Most prediction models were developed based on existing clinical research databases or retrospective data, so there was no sample size calculation process and the sample size did not meet the PROBAST requirement that the number of participants with the outcome relative to the number of candidate predictor parameters is ≥ 20. In the screening of predictors, most studies used univariate analysis and did not mention the handling methods of missing data.

Fig. 2.

Prediction Model Risk of Bias Assessment Tool (PROBAST) for the studies included in this review. a The risk of bias of the 10 included studies. b The applicability of the 10 included studies.

In terms of applicability, high risks were from the outcome domain, mainly because the timing of the primary outcome did not match the review question (Fig. 2 (b)). PTS should be diagnosed at least 6 months after the DVT, and studies determining the risk of PTS after DVT generally followed up to 2 years, whereas among the 10 included studies with risk prediction models for PTS, one study followed up to 6 months and one study, Elham E. Amin,2018 [20], the prediction model was a two-step model consisting of a model to be applied at baseline to predict the probability of developing PTS at 6 months, and a model to be applied at 6 months to predict the probability of PTS 24 months.

4.3. Meta-analysis results

Tao Yu, 2022 [12] and Lijun Zhu [13], 2022 used machine learning and Peng Qiu, 2021 [16] didn't report 95 % CI, so meta-analysis was performed on the model discrimination (AUC/C-statistic) of the remaining seven studies. The pooled discrimination was 0.75, 95%CI (0.69,0.81) (Fig. 3). A meta-analysis of the incidence of PTS in the development cohorts reported by each study resulted in a pooled incidence of 38 % (Forest plot and the subgroup analysis are shown in the supplementary file2).

Fig. 3.

Forest plot of the C-statistic/AUC of prediction models.

5. Discussion

This systematic review and meta-analysis included 10 studies of prediction models for the risk of PTS in patients with DVT, and these studies developed a total of 13 prediction models, of which 2 were high-quality models. The AUC/C-statistic of all models was greater than 0.63, and the most common predictors in the models were BMI, Varicose vein, Iliofemoral thrombosis, Baseline Villalta Score, and Age.

The majority of prediction models included in this study employed traditional statistical methods, such as Logistic regression and COX regression, while only two studies utilized machine learning methods. Compared with traditional statistical methods, machine learning can accommodate more predictors than only statistically significant variables, so machine learning is more suitable for analyzing large data sets and multivariate data [22]. However, the disadvantages of machine learning are that its construction is complex, many studies are under-reported, and machine-learning model is not well interpretable, making it difficult for researchers to figure out what is driving outcomes [23]. In general, there is still great potential for machine learning methods in PTS risk prediction models in the future.

In terms of predictor, although BMI was the most common variable in the models, the cut-off value of BMI was different in different models, which was caused by different populations for model development. There were three BMI cutoffs: BMI≥35(Anat Rabinovich, 2018 [18]), BMI≥30 (Elham E. Amin, 2018 [20]), and BMI≥24(Tian'an Huang, 2022 [21]). For Caucasians, BMI was more likely to be included in the model, and among the 9 prediction models included in BMI, 7 prediction models were developed based on Caucasians and only 2 models were based on Asians. Therefore, the effect of obesity on PTS in the Asian population needs to be further explored.

Age has the same problem as BMI. In the included prediction model, age has the following three cut-off values: Age≥65 (Anat Rabinovich, 2020 [17]), Age>75 (Marie Mean, 2018 [19]), and Age>56 (Elham E. Amin, 2018 [20]). At present, many studies have found that advanced age is a risk factor for PTS in patients with DVT [24,25], but what age belongs to advanced age, because the study population is different, the results are not the same. A large sample study is needed to confirm this.

In this study, six models included Varicose veins, although the form of inclusion was inconsistent (Varicose veins, Prior varicose vein surgery, Varicose vein history, History of chronic venous insufficiency). Varicose veins or chronic venous insufficiency is a non-negligible factor in the development of PTS, which will aggravate the degree of venous hypertension and further increase the possibility of DVT patients developing into PTS [26]. Therefore, this variable should be taken into account in the subsequent external validation of the model or model development.

In terms of model performance, the discrimination of most prediction models was between 0.7 and 0.8, and the pooled discrimination of meta-analysis was 0.75, which was moderate accuracy. Michelle Pradier conducted a study to assess the performance of SOX-PTS score (Anat Rabinovich，2018 [18]), Méan model (Marie Méan, 2018 [19]) and Amin model (Elham E. Amin, 2018 [20]) in SAVER database [27]. The result showed that, the performance of the SOX-PTS score and the Méan model were acceptable with an AUC of 0.72 and 0.74 respectively. Conversely, the Amin model was a poor predictor for PTS (AUC = 0.58). However, SAVER solely examined PTS 6 months post-DVT and lacks validation of the models' performance at the 24-month mark after DVT. Therefore, further studies are needed to confirm the accuracy of these models and how to use them in clinical practice.

This systematic review and meta-analysis identified some major problems in PTS prediction models. Firstly, prediction models were developed on existing databases of randomized trials or multicenter clinical trials, such as SOX-Trail database, ATTRACT Trail database, SWITCO65+ database. These databases have the advantages of prospective and complete follow-up data. However, since these databases were not established for the purpose of studying the risk of PTS after DVT, some candidate predictors and information of predictors would be missing in the process of model development. In view of this, a specificity database of PTS should be established in the future.

Secondly, there was a lack of external validation. External validation is a key step for models to be applied to clinical practice, but most of the existing models lack external validation. The models with external validation did not perform well, potentially attributed to the limited number of predictors incorporated in these models (only 3 predictors).

Thirdly, there were problems with data analysis. For instance, in order to better display the model, many models often convert continuous variables like age, BMI, and D-dimer into binary variables. However this approach inevitably leads to information loss and introduces potential changes in higher-order risks before and after the cut-off value.

Fourthly, there was no uniform standard for the number of predictors included in the model. As observed in the 13 models analyzed, the minimum number of predictors included was three, while the maximum reached seven. For the prediction model, the more predictors and the wider the scope (biology, demographic and imaging variables, etc.) involved, the prediction result will certainly be more accurate and more exact. Nevertheless, excessive predictors will affect the usefulness of the model in clinical practice, and if predictor variables are notreadily accessible, it will also affect the application of the model.

Finally, lack of biology predictors. Study has reported that PTS is associated with growth factors and chemokines, especially interleukin (IL) 6, IL-8, and IL-10, but not necessarily causally [28]. The BioSOX study reported that, Only ICAM-1 levels showed consistent, statistically significant differences across time points between patients who developed PTS and patients who did not, following adjustment by a logistic regression model [29]. The study of PTS biomarker has always been the focus of PTS research. The research and discovery of biomarker can provide new ideas and new targets for the treatment of PTS, such as the application of intravenous active drugs in the treatment of PTS in recent years [30]. Future PTS prediction models can incorporate biology predictors to predict the risk of PTS more comprehensively and accurately.

This study has certain limitations. Firstly, like all reviews, this work is limited by the differences in the original studies. Due to the inconsistent research methods, predictors and their cut-off values (such as BMI and age), it is not possible to conduct meta-analysis on some high-weight predictors. Limitations of this study include the lack of a head-to-head comparison and the recommendation of a model which would best improve clinical care, which is due to heterogeneity between studies and lack of external validation of the model.

6. Conclusion

In this review and meta-analysis, we identified 10 studies that predicted the risk of PTS after DVT. These prediction models lack external validation, which is a key step to ensure the model's clinical application. In the development of prediction models, the lack of a specificity database is the main reason for the lack of predictors in the models, or the inconsistent cut-off value of predictor variables. In addition, most of the prediction model research has problems in data processing, and it is recommended that future studies on the design and implementation of prediction models refer to Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) and PROBAST.

Funding statement

This study was funded by Shanxi Province “136 Revitalization Medical Project Construction Funds”.

Data availability statement

Data included in article/supplementary material/referenced in article.

Ethics approval and consent to participate

Review and/or approval by an ethics committee and informed consent was not needed for this study because the study was conducted based on previous published studies.

CRediT authorship contribution statement

Xiaorong Guo: Writing – original draft, Data curation, Conceptualization. Huimin Xu: Writing – review & editing, Data curation. Jiantao Zhang: Methodology, Data curation. Bin Hao: Writing – review & editing, Conceptualization. Tao Yang: Writing – review & editing, Methodology, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the supplementary data to this article.