Development and External Validation of a Text-Matching Algorithm to Identify Patients with Lower Extremity Deep Venous Thrombosis from Radiology Reports at Five Medical Centers

Abstract

Background:

Venous ultrasound studies evaluating for deep venous thrombosis (DVT) are often only available as unstructured radiology reports. To identify the presence or absence of DVT in a cohort of patients often requires manual review. Existing automated methods (i.e., natural language processing) for identifying the presence of DVT from free text are computationally complex and difficult to integrate into the electronic health record (EHR) and electronic data warehouses as structured data. We sought to develop and validate a simple regular expression algorithm-based approach in structured query language (SQL) to extract the presence or absence of DVT from unstructured ultrasound (US) reports.

Methods:

We extracted the radiology reports for venous US obtained from the ED at five academic medical centers to create the Regular Expression Aided Determination of Deep Venous Thrombosis (READ-DVT) algorithm. All reports were reviewed for the presence or absence of DVT by trained adjudicators. We split data from one medical center into test and training sets and used the data from four remaining centers for external validation. We created the algorithm through an iterative process within the training set and applied these results to the test and validation sets. We then evaluated performance, and based on these results, re-iterated and conducted multiple rounds of testing and validation including validation on new, unseen data when indicated. For each data set we calculated the sensitivity, specificity, positive predictive value, negative predictive value, and F1 score.

Results:

We extracted 9,249 radiology reports, of which 1,398 (15.1%) were adjudicated positive for DVT. Our algorithm achieved a sensitivity of 93.4%, a specificity of 99.8%, and an F1 score of 0.958 on the test set. On three external validation sets our algorithm achieved sensitivities ranging from 93.7% to 100.0%, specificities ranging from 98.9% to 99.5%, and F1 scores ranging from 0.991 to 0.957. Notably, on our fourth external validation set, our algorithm underperformed with sensitivity of 83.8%, specificity of 99.4%, and F1 score of 0.901. By using these data to iteratively update READ-DVT we found a simple addition to our algorithm vastly improved the model function with no change in performance in the original test, training, and initial three validation sets. We created a new external validation set from the same medical center as our fourth external validation set and achieved sensitivity of 98.0%, specificity of 99.5%, and F1 score of 0.980.

Conclusions:

READ-DVT is a robust, multi-center validated method for identifying the presence or absence of DVT in radiology reports. The algorithm performed similarly to the published results of more complex natural language processing tools designed for the same task. In combination with similar validated tools for pulmonary embolism, READ-DVT may enable accurate automated coding of venous thromboembolism.

Introduction:

Deep venous thrombosis (DVT) is a common condition, with a global incidence ranging from 52 to 162 per 100,000 person years.^1–6 The gold standard diagnostic test for DVT is comprehensive duplex ultrasonography.⁷ Typically, the results of such testing are stored in the electronic health record (EHR) as unstructured, free text information.^{8, 9} While easily accessible by clinicians, unstructured data presents several challenges when using automated systems to gather information from the EHR on patients evaluated for and diagnosed with DVT for research or other purposes. Traditionally, unstructured reports have required manual review, which is costly, time-consuming, and error-prone.^10–12 Other approaches to identify cases of venous thromboembolism (VTE), including DVT, using diagnostic codes,^{13, 14} anticoagulation prescriptions,¹⁵ and insurance claims^{16, 17} are inaccurate in over 25% of cases. Better automated, validated tools to identify cases of DVT are needed.

Within the last five years, there has been tremendous interest in using artificial intelligence to help extract clinical data from the EHR.^18–20 Several studies have effectively developed methods to identify VTE, including DVT, using natural language processing (NLP) and large language models (LLMs), an advanced deep learning implementation of NLP.^21–25 While computationally exciting, these methods face critical obstacles for broad-based clinical implementation. NLP and LLMs: (1) require large amounts of high-quality clinical training data which are not always available,^{26, 27} (2) are often considered “black box” models and lack end user interpretability which reduces the ability to confirm clinical validity,^{28, 29} (3) are prone to inheriting biases present in training data which limit generalizability,^{28, 30} and (4) are challenging to integrate within the EHR and transfer to other EHRs.³¹ A well-described, less computationally complex approach is the implementation of a regular expression algorithm (RegEx).^{32, 33} RegEx algorithms are simple, flexible, and powerful text processing and matching tools which allow users to easily describe and parse text with relatively few lines of code.³⁴ Unlike NLP tools which require multiple processing steps, RegEx algorithms only require a single coding command. The basic premise of a well-written RegEx algorithm is it can identify the text of interest in a free text field while allowing for variability in how that free text is written. For example, a RegEx algorithm looking to identify the derivatives of “thrombus” including “thrombosis,” “thrombotic,” “thrombi” would use search parameters looking for “thromb%” which would include anything with the root “thromb” but allow for variability in the suffix.³⁴

We have previously created and externally validated a RegEx algorithm to identify the presence or absence of PE within computed tomographic angiograms (CTAs) with 93% sensitivity and 99% specificity.³² We sought to develop and validate a similarly robust RegEx algorithm using radiology reports from venous ultrasounds to identify cases positive for DVT.

Materials and Methods:

Our retrospective, observational study protocol was approved and issued a waiver of informed consent by the institutional review boards of all five institutions given the minimal risk to human subjects. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.³⁵

Settings and Subjects:

This study was conducted using data from Vanderbilt University Medical Center (VUMC, Nashville, TN), Massachusetts General Hospital (MGH, Boston, MA), Barnes-Jewish Hospital (BJH, St. Louis, MO), University of Texas Southwestern Medical Center (UTSW, Dallas, TX), and University of Cincinnati Medical Center (UCMC, Cincinnati, OH). The study protocols and procedures were reviewed and approved by the Institutional Review Board (IRB) at each respective institution (Protocol numbers 251078 [VUMC], 2023P002843 [MGH], 202407204 [BJH], STU-2024–0870 [UTSW], and 2024–0559 [UCMC]). In alignment with existing data use agreements (DUAs), deidentified data from each site was extracted and transmitted to VUMC for review and analysis.

Our derivation cohort included radiology reports from all venous ultrasounds performed in the ED at VUMC between October 1, 2021 and September 30, 2022. Our four validation cohorts included all venous ultrasounds performed in the ED at MGH between October 1, 2021 and September 30, 2022, BJH between January 1, 2024 and December 31, 2024, UTSW between January 1, 2024 and December 31, 2024, and the UCMC between January 1, 2024 and December 31, 2025. UCMC data were split into an initial validation set, January 1, 2024 to December 31, 2024, and a revalidation set, January 1, 2025 to December 31, 2025. Data sets included radiologist-interpreted venous ultrasounds (duplex venous ultrasounds [VUMC, MGH, BJH, UTSW, and UCMC] and venous compression ultrasounds [MGH]) and ED provider-performed and interpreted venous compression ultrasounds (UCMC). All radiology reports were free text reports.

Venous ultrasound results were eligible for inclusion if they were performed on adult (age ≥ 18 years) emergency department (ED) patients. At all sites, we included venous ultrasounds with formal radiology interpretations and impressions. At one site, UCMC, we included ED provider-performed and interpreted bedside venous ultrasounds where formal reports signed by the attending ED physician were available. Reports signed by attending ED physicians were considered finalized results consistent with professional expectations and competencies from the American Board of Emergency Medicine (ABEM) and the American College of Emergency Physicians (ACEP).^{36, 37} All radiology reports were written in English. We excluded upper extremity ultrasounds in our analysis given that they are rarely performed, and arm thrombi account for only 4–10% of DVTs, which could affect statistical power for derivation and validation of our RegEx.^{38, 39} Additionally, a dedicated algorithm to identify upper extremity DVT presence and location is preferable.³⁸

Data Source:

Radiology reports of venous ultrasounds were extracted from data from the Automated Registry of CardioVascular Emergencies (ARCVE), one of the largest multi-center registries of patients evaluated for VTE in the world with nearly 400,000 included patient encounters (www.arcve.org).^{40, 41} Each ARCVE institution uses Epic^™ (Verona, WI) as their EHR. ARCVE data are extracted using ³ queries that cull reports of all venous ultrasounds performed on ED patients at each institution within the specified dates. All radiology-performed venous ultrasounds were read by radiologists, including trainees with attending oversight. All ED-performed bedside venous ultrasounds in the UCMC data set were read by ED physicians, including trainees with attending oversight. The use of dictation software, interactive templates, and direct text entry was at the discretion of the physician. Within the EHR, the impression and narrative sections are separate fields. We derived our algorithm using only impression sections, as they are typically clearer and less ambiguous than narrative sections. For simplicity, we hereafter refer to impression sections as the “radiology report.” We excluded duplicate studies and studies for which the radiology report was unavailable or incomplete.

We split our derivation cohort (n = 1,516) into a training set (n = 1,004, 66.2%) and a test set (n = 512, 33.8%). The external validation sets included 7,228 venous ultrasounds from the MGH (n = 2,684, 34.7%), BJH (n = 1,318, 17.0%), UTSW (n = 2,340, 30.3%), and two UCMC (n = 886 [11.5%] and n = 506 [6.5%]) validation sets.

Our unit of analysis was the radiology report. Depending on imaging performed, reports include either left-sided or right-sided venous ultrasounds. When bilateral venous ultrasounds were performed and were described in a single report, the results were considered as a single imaging study. Only one report was included in the event of report duplication. If patients had multiple different venous ultrasounds performed during the study period, these repeat studies were included as unique entities.

Creation of Reference Standard:

The venous ultrasound radiology reports were extracted and stored in site-specific secure repositories for manual review. When available, manual adjudication used REDCap^™ (Nashville, TN).^{42, 43} The results from manual review defined our reference standard for the training, test, and validation sets. Each ultrasound radiology report was read by at least two trained reviewers. In instances where these reviewers disagreed, a third attending emergency physician and expert reviewer provided final adjudication. Reviewers were not blinded to study objectives. Reviewers identified the presence or absence of DVT. The VUMC training and test sets were independently reviewed by two medical students (AA and SJO) with discrepancies reviewed by a practicing emergency physician (WBS). The MGH validation set was independently reviewed by two practicing emergency physicians (WBS and DAB) with discrepancies reviewed by a third practicing emergency physician (JOW). The BJH, UTSW, and two UCMC validation sets were independently reviewed by two practicing emergency physicians (WBS and JOW) with discrepancies reviewed by a third practicing emergency physician (DAB).

Algorithm Development, Validation, and Revalidation:

We developed a structured query language (SQL) algorithm, READ-DVT (Regular Expression Aided Determination of Deep Venous Thrombosis) that is a combination of two RegEx queries with simple logic to identify DVT from the radiology report of venous ultrasounds (eTable 1). READ-DVT was iteratively developed in two distinct phases. In the first phase, the derivation cohort was randomly split into a two-thirds training set and a one-third test set. Using the training set, the READ-DVT 1.0 algorithm was iteratively derived and adjusted using variations on inclusion and exclusion criteria until sensitivity and specificity could no longer be jointly improved without accruing losses in the other (Figure 2).

Figure 2:

Selection of study cohorts and analysis flowchart

Abbreviations: BJH, Barnes-Jewish Hospital; MGH, Massachusetts General Hospital; READ-DVT, Regular Expression Aided Determination of DVT; UCMC, University of Cincinnati Medical Center; UTSW, University of Texas Southwestern Medical Center; VUMC, Vanderbilt University Medical Center

Notes: a. Description of study cohorts including derivation cohort (VUMC only), validation cohorts (MGH, BJH, UTSW, and UCMC), and revalidation cohort (UCMC).

b. Derivation diagram for READ-DVT 1.0 demonstrating use of VUMC training data for iterative algorithm development followed by internal validation using VUMC test data and external validation data using validation cohorts (MGH, BJH, UTSW, and UCMC).

c. Derivation diagram for READ-DVT 2.0 after underperformance of READ-DVT 1.0 in the UCMC validation set. READ-DVT 2.0 was iteratively developed using the UCMC validation set, validated on the prior data including the original derivation cohort (VUMC) and remaining validation cohorts (MGH, BJH, and UTSW), and validated on new data using the UCMC revalidation cohort.

Venous ultrasound radiology reports were pre-processed by dividing the text into individual sentences. The two RegEx queries were then developed to identify phrases indicating the presence or absence of DVT. If any sentence in the report had a match for a positive phrase in either RegEx query without any negative phrases the report was considered positive for DVT.

During validation, we noted good performance of READ-DVT 1.0 on three of the four validation sets but underperformance on one (UCMC) validation set. This prompted the second post hoc phase of algorithm development for post-validation refinement due to noted poor model performance in one validation set. During this refinement, the UCMC validation set was used as a training set and READ-DVT 1.0 was iteratively developed on the UCMC validation set again until sensitivity and specificity could no longer be jointly improved without accruing losses in the other. The resulting algorithm, READ-DVT 2.0, is nearly identical to READ-DVT 1.0 but includes one additional positive phrase in the second RegEx query (signs%dvt). To ensure validity of this iterative process, a new UCMC revalidation set was adjudicated and used as an independent test set. READ-DVT 2.0 was then reapplied to every other data set and the results of READ-DVT 1.0 and 2.0 on each of these data sets were compared. Pseudocode for both READ-DVT 1.0 and 2.0 can be found in eTable 1.

Statistical Analysis:

Algorithm development, performance measures, and cohort descriptive statistics were completed using Microsoft SQL Server Management Studio (Redmond, WA) and interrater reliability was calculated using R version 4.3.1 (Vienna, Austria).⁴⁴ Patient characteristics for the VUMC cohort are presented using standard descriptive statistics. The interrater reliabilities for reference standard creation are described using Cohen’s Kappa.⁴⁵ READ-DVT 1.0 and 2.0 algorithm performance was assessed using sensitivity, specificity, positive predictive value, negative predictive value, and the F1 score.^{46, 47}

Results:

There were 1,534 venous ultrasounds included in the initial derivation set of which 1,516 were included in the final derivation cohort with 18 (1.2%) excluded due to either being an incomplete or a duplicate report. Of these, 1,004 (66.2%) were included in the training set and 512 (33.8%) in the test set. In the derivation set, the median age was 53.9 (IQR 39.7–66.3) years, 819 (54.0%) were female, 367 (24.2%) were Black or African American, 1,053 (69.5%) were White, 71 (4.7%) were Hispanic, and 38 (2.5%) received two or more venous ultrasounds during the study period (Table 1). In the validation sets, <3% of studies were excluded due to missing, incomplete, or duplicate reports at any site: MGH (total = 2,764, excluded = 80 [2.9%]), BJH (total = 1,318, excluded = 1 [0.08%]), UTSW (total = 2,344, excluded = 4 [0.2%]), UCMC validation (total = 889, excluded = 3 [0.3%]), UCMC revalidation (total = 508, excluded = 2 [0.4%]).

Table 1.

Characteristics of patients at time of venous ultrasound exam

Characteristic	VUMC Derivation Cohort (n = 1516 patients)	VUMC Training Set (n = 1004 patients)	VUMC Test Set (n = 512 patients)
Age, median (IQR), y	53.9 (39.7 – 66.3)	54.2 (39.9 – 66.2)	53.5 (39.5 – 66.6)
Sex, No. (%)
Female	819 (54.0)	533 (53.1)	286 (55.9)
Male	697 (46.0)	471 (46.9)	226 (44.1)
Race, No. (%)a
Asian	8 (0.5)	7 (0.7)	1 (0.2)
Black or African American	367 (24.2)	256 (25.5)	111 (21.7)
White	1053 (69.5)	689 (68.6)	364 (71.1)
Other Raceb	55 (3.6)	33 (3.3)	22 (4.3)
Unknown	33 (2.2)	19 (1.9)	14 (2.7)
Ethnicity, No. (%)a
Hispanic	71 (4.7)	38 (3.8)	33 (6.4)
Non-Hispanic	937 (61.8)	633 (63.0)	304 (59.4)
Unknown	508 (33.5)	333 (33.2)	175 (34.2)
Venous Ultrasound exams per patient, No. (%)c,d
One	1434 (94.6)	946 (94.2)	488 (95.3)
Two	32 (2.1)	23 (2.3)	9 (1.8)
Three	6 (0.4)	4 (0.4)	2 (0.4)

Abbreviations: IQR, interquartile range; VUMC, Vanderbilt University Medical Center

Notes:

^d.The VUMC cohort has a significant proportion of patients with missing race and ethnicity status because of a change made in 2022 to how race and ethnicity were ascertained and data from prior to this change were deleted before our study began.

^e.American Indian or Alaskan Native, Middle Eastern, Native Hawaiian or Pacific Islander, Multiracial not listed

^f.Total number of exams is greater than n patients in each cohort since patients could contribute multiple exams.

^g.The number of ultrasound exams per patient was the number of unique exams which occurred during the study period. If a patient had two exams at the same time (i.e., a DVT study of both lower extremities) this was only considered one exam.

The prevalence of DVT within the study cohort was 15.1% (12.7% at VUMC, 10.7% at MGH, 21.1% at BJH, 15.1% at UTSW, 20.9% in the UCMC validation set, and 20.2% in the UCMC revalidation set). Cohen’s Kappa was 0.965 in the VUMC derivation set, 0.988 in the MGH validation set, 0.991 in the BJH validation set, 0.995 in the UTSW validation set, 1.000 in the UCMC validation set, and 0.994 in the UCMC revalidation set (Table 2).

Table 2.

Data set descriptions and interrater reliability of radiology report

Cohorta	Period	Total (n)	Included (n, %)	DVT+ (%)	Cohen’s Kappa
VUMC combined	10/01/21–09/30/22	1,534	1,516 (98.3%)	193 (12.7%)	0.965
VUMC training	10/01/21–09/30/22		1,004 (66.2%)b	132 (13.1%)
VUMC test	10/01/21–09/30/22		512 (33.8%)b	61 (11.9%)
MGH validation	10/01/21–09/30/22	2,764	2,684 (97.1%)	286 (10.7%)	0.988
BJH validation	01/01/24–12/31/24	1,318	1,317 (99.9%)	278 (21.1%)	0.991
UTSW validation	01/01/24–12/31/24	2,344	2,340 (99.8%)	354 (15.1%)	0.995
UCMC validation	01/01/24–12/31/24	889	886 (99.7%)	185 (20.9%)	1.000
UCMC revalidation	01/01/25–12/31/25	508	506 (99.6%)	102 (20.2%)	0.994

Abbreviations: BJH, Barnes-Jewish Hospital; DVT, Deep Venous Thrombosis; MGH, Massachusetts General Hospital; UCMC, University of Cincinnati Medical Center; UTSW, University of Texas Southwestern Medical Center; VUMC, Vanderbilt University Medical Center

Notes:

^a.The VUMC data are presented as both a combined set as well as the random division of the training (2/3) and test (1/3) sets.

^b.Percentages for training and test set are presented as a proportion of the included VUMC combined set (n = 1,516).

Algorithm Performance and Description:

When applied to the VUMC training set, the READ-DVT 1.0 algorithm achieved sensitivity of 98.5%, specificity of 99.5%, and F1 score of 0.977. This performance was similar to the held-out VUMC test set with a sensitivity of 93.4%, specificity of 99.8%, and F1 score of 0.958. For the MGH, BJH, and UTSW validation sets, the performance was similar to the VUMC test set (Table 3).

Table 3.

READ-DVT performance in training, internal validation, and external validation

Cohorta	Algorithmb	Sensitivity	Specificity	PPV	NPV	F1 Score
VUMC training	1.0 & 2.0	98.5%	99.5%	97.0%	99.8%	0.977
VUMC test	1.0 & 2.0	93.4%	99.8%	98.3%	99.1%	0.958
MGH validation	1.0 & 2.0	93.7%	99.2%	93.1%	99.2%	0.934
BJH validation	1.0 & 2.0	100.0%	99.5%	98.2%	100.0%	0.991
UTSW validation	1.0 & 2.0	97.2%	98.9%	94.2%	99.5%	0.957
UCMC validation	1.0	83.8%	99.4%	97.5%	95.9%	0.901
UCMC validation	2.0	98.9%	99.4%	97.9%	99.7%	0.984
UCMC revalidation	2.0	98.0%	99.5%	98.0%	99.5%	0.980

Abbreviations: BJH, Barnes-Jewish Hospital; MGH, Massachusetts General Hospital; NPV, negative predictive value; PPV, positive predictive value; READ-DVT, Regular Expression Aided Determination of DVT; UCMC, University of Cincinnati Medical Center; UTSW, UT Southwestern Medical Center; VUMC, Vanderbilt University Medical Center

Notes:

^a.The VUMC derivation cohort was randomly split into training (2/3) and test (1/3) sets.

^b.READ-DVT 1.0 represents the algorithm trained on the VUMC training set and applied to the test and validation sets. READ-DVT 2.0 represents the algorithm which was iteratively re-trained on the UCMC validation set and applied to the VUMC training, test, and MGH, BJH, UTSW validation, and UCMC revalidation sets. Except where noted, performance of READ-DVT 1.0 and READ-DVT 2.0 was equivalent.

READ-DVT 1.0 underperformed on the UCMC validation set with a sensitivity of 83.8%, specificity of 99.4%, and F1 score of 0.901. This reduction in performance was due to a high proportion of false negatives in the UCMC validation set. We identified the cause of the increased false negative rate as the use of the term “signs of DVT” in the UCMC venous radiology reports which was not a term used in the radiology reports in the training, test, or other validation sets. READ-DVT 1.0 was iteratively redesigned using the UCMC validation set as a new training set (see eTable 1) to create READ-DVT 2.0. The performance of READ-DVT 2.0 on the UCMC validation set improved sensitivity to 98.9% with the number of false negatives decreasing from 30 to 2. There was no change in the reports identified as false positives, so the specificity on the UCMC validation set was 99.4% for both READ-DVT 1.0 and 2.0 algorithms. Given that the UCMC validation set could no longer be considered an external validation set, we applied READ-DVT 2.0 to a new UCMC validation set. This yielded sensitivity of 98.0%, specificity of 99.5%, and F1 score of 0.980. When READ-DVT 2.0 was reapplied to the VUMC training and test sets, and the MGH, BJH, and UTSW validation sets, the performance metrics were unchanged compared to READ-DVT 1.0 in these data sets (Table 3). Contingency tables comparing results of the READ-DVT 1.0 and 2.0 algorithms to manual review are shown in the Appendix (eTables 2–9).

Discussion:

READ-DVT is a simple, multi-center validated RegEx algorithm which identifies the presence or absence of DVT from unstructured radiology reports with excellent test characteristics. It is a valid algorithm to define the DVT phenotype among ED patients undergoing venous US and may serve as a model for the general process of extracting information from unstructured EHR text.

One of the primary benefits of using RegEx algorithms to identify features in free text is transportability. Modern machine learning-based NLP algorithms involve between 8 to 12 steps in the interpretation of free text.⁴⁸ These systems require resources and expertise to implement. In contrast, READ-DVT relies upon a RegEx algorithm with simple logic that can be implemented using nearly any data analysis software and few computational requirements.

One major limitation, as demonstrated here, is that site validation is critical and may limit transportability across institutions. READ-DVT 1.0 had exceptional test characteristics in the first three of our four validation sets but underperformed on one validation set. This was due to a local radiology practice involving a phrase that was not used at any of our other sites. There are significant practice pattern variations between individual radiologists and institutions.^{49, 50} Such linguistic practice variations are a limitation to RegEx algorithms and demonstrate the necessity of site-specific recalibration prior to implementation, a process which should be considered best practice for any clinical prediction model.⁵¹ While the post hoc adjustments made to READ-DVT 2.0 based on findings in one of the validation sets demonstrate limits to transportability and possibly broad-scale implementation, the overall findings of this iterative process made the algorithm more robust. While there is a risk of optimism bias due to the two-step revalidation process, we argue that when implementing a predictive algorithm at new sites that such a process of iterative refinement is critical to create broadly applicable algorithms. If this process is done carefully each iterative process may improve generalized model performance.

The development and performance of READ-DVT does have three further limitations. First, our reference standard was based on manual review. While our interrater reliability was high, some references suggest that error in medical record abstraction can be as high as 1–3% even in properly trained reviewers.⁵² A system that used radiologist-entered discrete data attesting to the presence or absence of DVT in venous ultrasound reports could address this, but this is not current practice and would likely have minimal effect on our results. Second, the data collected and evaluated by READ-DVT is limited only to determining the presence or absence of a DVT. It does not consider additional features such as the location of the clot, the size of the clot, or the presence of multiple clots. These data can be important as patients with more severe DVT can be candidates for catheter directed thrombolysis or mechanical thrombectomy.¹ Third, all of our data were collected from medical centers that use Epic^™ as their EHR. Our results have not been validated using other EHR databases, but the SQL-based algorithm may be translated to other platforms as almost all EHR systems use relational databases and SQL where the portability of the algorithm should be straightforward.^{53, 54}

Overall, our results suggest that a computationally straightforward RegEx algorithm can accurately identify the presence of DVT from radiology reports. Notably, when compared to previously published data using more computationally complex, machine learning-based algorithms including NLP algorithms, support vector machines (SVMs), and bidirectional encoder representations from transformers (BERTs), our more computationally straightforward RegEx algorithm had similar performance (Table 4). Two of the more robust algorithms for identifying DVTs are described in publications by Verma et al. (2022) and Jafari et al. (2025).^{23, 25} The weighted NLP algorithm described by Verma et al. (2022) achieved 94% sensitivity and 99% specificity on a test set of 445 reports.²³ The VTE-BERT algorithm described by Jafari et al. (2025) achieved 92% sensitivity and 98% specificity on an external validation set of 375 reports.²⁵ Notably, Verma et al. (2022) trained and validated their model on only 1,245 reports from five medical centers and Jafari et al. (2025) trained and validated their model on 1,466 reports from two medical systems. In training, testing, and validating READ-DVT, we used a total of 9,249 radiology reports. To our knowledge, this is the largest multi-center study evaluating an algorithm to identify DVT from radiology reports.^{23, 25}

Table 4.

READ-DVT 2.0 performance compared to previously described methods

Algorithm	Publication	Class	Sample (n)a	Result Set	Sensitivityb	Specificityb
N/Ac	Rochefort et al. 201521	SVM	2000	Cross-Validation	80%	98%
N/A	Tian et al. 201722	Symbolic NLP	4000	External Test	94%	96%
N/A	Verma et al. 202123	NLP	1245	Internal Test	94%	99%
ClotCatcherc	Wang et al. 202324	BERT	2273	Internal Testd	98.4%	99.2%
VTE-BERTc	Jafari et al. 202525	BERT	1515	Internal Test	91%	99%
READ-DVT 2.0		RegEx	9249	Internal Test	93.4%	99.8%

Abbreviations: BERT, Bidirectional Encoder Representations from Transformers; READ-DVT, Regular Expression Aided Determination of DVT; RegEx, Regular Expression Algorithm; SVM, Support Vector Machine; VTE-BERT, Venous Thromboembolism Bidirectional Encoder Representation from Transformers

Notes:

^a.Represents the total samples used, including train, test, and validation sets, where applicable. Includes only DVT studies except when authors did not differentiate between PE and DVT when reporting sample numbers (Jafari, Wang, Rochefort, Tian)

^b.Represents best model and best reported result by authors for test or validation sets. Cross-validation sets reported when no test or validation results reported.

^c.Model trained for detection of DVT and PE, reported results for combined VTE.

^d.Represents best reported results using one of two test sets. Test set from Grady Memorial Hospital

Conclusion:

READ-DVT is a simple, portable, multi-center validated, text-matching RegEx algorithm that can identify the presence or absence of DVT from radiology reports. READ-DVT and similar algorithms offer the ability to accurately identify clinical features where techniques such as diagnostic codes, anticoagulation prescriptions, insurance claims data, and manual review can be inaccurate or cumbersome. The development of tools like READ-DVT is critical to applications such as clinical predictive modeling. However, local validation of computer algorithms is necessary prior to use.

Figure 1:

Graphical Abstract for “Development and External Validation of a Text-Matching Algorithm to Identify Patients with Lower Extremity Deep Venous Thrombosis from Radiology Reports at Five Medical Centers”

Abbreviations: BJH, Barnes-Jewish Hospital; DVT, Deep Venous Thrombosis; MGH, Massachusetts General Hospital; READ-DVT, Regular Expression Aided Determination of DVT; UCMC, University of Cincinnati Medical Center; UTSW, University of Texas Southwestern Medical Center; VUMC, Vanderbilt University Medical Center

Highlights:

• Identification of deep venous thrombosis (DVT) from radiology reports is challenging.

• READ-DVT, a regular expression algorithm, identifies DVT in venous ultrasound reports.

• READ-DVT has been validated on data from five academic medical centers.

• Our analysis shows the importance of site-by-site validation and regular expression iteration.