Automated vs Semi-Automated Lab Value Extraction for the VA Cardiac Surgical Quality Improvement Program

Abstract

Background:

The Veterans Affairs Surgical Quality Improvement Program (VASQIP) trains surgical quality nurses (SQNs) at each VA hospital to extract or verify 187 variables from the medical record for all cardiac surgical cases. For ten preoperative laboratory values, VASQIP has a semi-automated system in which local lab values are automatically extracted, verified by SQNs, and lab values recorded at other VA facilities are manually extracted. The objective of this study was to develop and validate a method to automate the extraction of these ten preoperative laboratory values and compare results with the current semi-automated method.

Materials and methods:

We developed methods to extract ten preoperative laboratory values and measurement dates from the VA Corporate Data Warehouse (CDW) using Logical Observation Identifiers Names and Codes (LOINC). Automated vs. semi-automated information extraction was compared in terms of agreement, conformance to data definitions, proximity to surgery, and missingness.

Results:

For surgeries with both automated and semi-automated lab values, the intraclass correlation coefficients for the ten variables ranged from 0.90 to 0.98. For several variables, the automated method resulted in much lower rates of missing data (e.g., 2.4% vs. 22.5% missing data for high density lipoprotein) and eliminated out-of-date-range entries.

Conclusions:

Although SQN-extracted data are widely considered the gold standard within national surgical quality improvement programs, there may be advantages to fully automating extraction of lab values including high congruence with semi-automated SQN-extracted/verified values and lower rates of missingness and out-of-date-range data.

Introduction

The Veterans Affairs Surgical Quality Improvement Program (VASQIP) provides all VA cardiac surgical programs with institution-level, risk-adjusted outcome data for the purposes of quality improvement (QI).¹ VASQIP data abstraction is performed by trained surgical quality nurses (SQNs) at each VA hospital who extract or verify 187 variables from the medical record for every cardiac surgical case performed within the VA system.² Although manual data extraction and verification by SQNs is expensive and time consuming, it is currently considered the standard for ensuring high quality data in national surgical QI programs like VASQIP and similar programs such as the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP)³ and the Society of Thoracic Surgeons (STS) Cardiac Surgery Database⁴.

Although SQN-extracted (and/or verified) data is currently the standard process used by VASQIP, automation of data extraction could improve the timeliness, efficiency, and perhaps in some cases the accuracy of this process. Full or partial automation could also liberate local resources, presently invested in manual abstraction, to pursue more proactive and continuous engagement in other aspects of surgical QI. For the ten preoperative laboratory values captured by VASQIP, a semi-automated system is currently used in which local lab measurements are automatically extracted and verified by SQNs. Laboratory values recorded at other VA facilities are manually extracted from the electronic medical record by the SQNs. The definitions for each of these lab values are specific in terms of timing relative to surgery, the units and nature of allowable entries, and whether missing data is permissible (Table 1).

Table 1:

Cardiac VASQIP Laboratory Definitions and Extraction Method Details

Lab Value	VASQIP Definition	LOINC Codes	Method Details
High Density Lipoprotein	Indicates the high-density lipoprotein (HDL) result (in mg/dl) preoperatively evaluated closest to surgery. Entering “NS” for “No Study” is allowed.	2085–9, 49130–8, 27340–9, 18262–6	Search for results in the two-year period before surgery. Choose most proximal to surgery if multiple values found.
Hemoglobin	Indicates the patient’s Hemoglobin result (in g/dl) preoperatively evaluated closest to surgery but not greater than 30 days before surgery. Entering “NS” for “No Study” is not allowed.	718–7, 30313–1, 4635–9, 30350–3	Search for results in the 30-day period before surgery. Choose most proximal to surgery if multiple values found.
Hemoglobin A1c	Indicate the Hemoglobin A1c result (%) preoperatively evaluated closest to surgery. Entering “NS” for “No Study” is allowed.	4548–4, 17856–6, 59261–8	Search for results in the 1000-day period before surgery. Choose most recent one to surgery if multiple values found.
Low Density Lipoprotein	Indicates the low-density lipoprotein (LDL) result (in mg/dl) preoperatively evaluated closest to surgery. Entering “NS” for “No Study” is allowed.	2089–1, 13457–7, 18262–6	Search for results in the one-year period before surgery. Choose most proximal to surgery if multiple values found.
Serum Albumin	This is the serum albumin result (g/dl) most closely preceding surgery - not to exceed 30 days for Cardiac surgery. Data input must be 1 to 4 numeric characters in length which may include a prefix of a less than or greater than sign “<“ or “>“. Entering “NS” for “No Study” is not allowed.	1751–7	Search for results in the 30-day period before surgery. Choose most proximal to surgery if multiple values found.
Serum Bilirubin	This is the result of the preoperative total bilirubin test. Data input must be 1 to 5 numeric characters in length which may include a prefix of a less than or greater than sign “<“ or “>“. Entering “NS” for “No Study” is also allowed.	1975–2	Search for results in the 180-day period before surgery. Choose most recent one to surgery if multiple values found.
Serum Creatinine	This is the serum creatinine result (mg/dl) most closely preceding surgery - not to exceed 30 days for Cardiac surgery. Data input must be 1 to 4 numeric characters in length which may include a prefix of a less than or greater than	2160–0, 38483–4	Search for results in the 30-day period before surgery. Choose most recent one to surgery if multiple values found.
	sign “<“ or “>“. Entering “NS” for “No Study” is not allowed.
Serum Potassium	Indicates the Serum Potassium result (in mg/L) preoperatively evaluated closest to surgery but not greater than 90 days before surgery. Entering “NS” for “No Study” is allowed.	2823–3, 32713–0	Search for results in the 90-day period before surgery. Choose most recent one to surgery if multiple values found.
Serum Triglyceride	Indicates the patient’s Serum Triglyceride result (in mg/dl) preoperatively evaluated closest to surgery. Entering “NS” for “No Study” is allowed.	2571–8	Search for results in the 365-day period before surgery. Choose most recent one to surgery if multiple values found
Total Cholesterol	Indicates the Total Cholesterol result (in mg/dl) preoperatively evaluated closest to surgery. Entering “NS” for “No Study” is allowed.	2093–3, 21197–9, 14647–2	Search for results in the 720-day period before surgery. Choose most recent one to surgery if multiple values found

The objective of this study was to develop and validate a method to fully automate the extraction of cardiac VASQIP’s ten preoperative laboratory values and compare the results with the current semi-automated method. Only by comparing extraction methods can programmatic leaders and stakeholders in surgical QI ascertain the potential tradeoffs between cost, timeliness, and accuracy. Our hypothesis was that full automation of preoperative lab value extraction would provide higher congruence with established data definitions.

Material and Methods

VASQIP’S semi-automated, SQN abstraction (SA-VASQIP) was considered the gold standard (i.e., control) in this study—in other words, the actual VASQIP lab values in the dataset were used as the comparator for the automated extracts (A-VASQIP) from the VA Corporate Data Warehouse (CDW). We used the fiscal year 2018 cardiac VASQIP dataset, which contained data from 4,549 cardiac surgeries conducted in 41 VA facilities. Cardiac surgery for VA patients that were performed outside the VA system are not included in the VASQIP program. Python scripts to generate the values for specific surgeries from CDW are available on VA’s public facing Github repository (GitHub | Office of Information and Technology (va.gov) for general access; https://github.com/department-of-veterans-affairs/VASQIP_Values_Automated_Extraction/tree/main) or by request from the authors. These scripts could be adapted for other patients and contexts. This study was approved by the Stanford IRB and VA Palo Alto’s Research and Development Committee.

Based on cardiac VASQIP data definitions (Table 1), we developed methods to extract the ten preoperative laboratory values and dates from the src.Chem_PatientLabChem CDW table using Logical Observation Identifiers Names and Codes (LOINC) (Table 1). LOINC is a universal coding and reporting system for laboratory results and other clinical observations.⁵ When no preoperative timeframe was specified in the variable definition, we explored multiple options to minimize missing data. Preoperative laboratory values collected by cardiac VASQIP include High Density Lipoprotein, Low Density Lipoprotein, Hemoglobin, Hemoglobin A1c, Albumin, Bilirubin, Creatinine, Potassium, Triglyceride, and Total Cholesterol.

Automated VASQIP (A-VASQIP) vs. semi-automated VASQIP (SA-VASQIP) values and dates were compared in terms agreement, conformance to data definitions, proximity to surgery, and missingness. For surgeries with non-missing values for both semi-automated and automated methods, we used the intraclass correlation coefficient (ICC) as the measure of agreement. We also assessed conformance to data definitions, especially if the extracted values fell within the stated time frame (e.g., within the 30 preoperative days). As all variables specify that the value closest to surgery should be extracted, we compared the methods on how many days before surgery the extracted value was recorded. We also compared the percent of missing values and the degree of overlap of missing values.

Results

For High Density Lipoprotein, SA-VASQIP had 22.5% missing data compared to 2.4% for A-VASQIP (Table 2). For the 3,490 cases with values using both methods, 94.1% of the values were recorded on the same day, while A-VASQIP was more proximal for 86.8% (178 of 205) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.98).

Table 2:

Semi-Automated (SA) vs Automated (A) Extraction of Cardiac VASQIP Lab Values for 4549 Surgeries

Lab Value	Missing in SA-VASQIP	Missing in A-VASQIP	Matched Missing	Matched Non-missing	Proximity	Days Different (SA-MA)	ICC*
High Density Lipoprotein	1024 (22.5%)	110 (2.4%)	75 (1.6%)	3490	3285 (94.1%) same day. Of 205 recorded on different day, 178 (86.8%) A-VASQIP more proximal	Mean = −7.44 SD = 70.43 Range −-897, 729	.98
Hemoglobin	0% (64 (1.4%) >30 days before surgery)	61 (1.3%)	0 (0%)	4475	3858(86.2%) same day. Of 617 recorded on different day, 347 (56.2%) A-VASQIP more proximal	Mean = −1.21 SD = 2.29 Range −29, 25	.95
Hemoglobin A1c	632(13.9%)	228(5.0%)	179(3.9%)	3868	2803(72.5%) same day. Of 1,065 recorded on different day, 790 (74.2%) A-VASQIP more proximal	Mean = −6.24 SD = 89.25 Range −955, 991	.98
Low Density Lipoprotein	1309(28.8%)	925(20.3%)	241(5.3%)		2045(80.0%) same day Of 511 recorded on different day, 134 (26.2%) A-VASQIP more proximal	Mean = −11.35 SD = 75.46 Range −969, 364	.97
Serum Albumin	560(12.3%) (131(2.9%) >30 days before surgery)	619(13.6%)	603(13.3%)	3842	3749(97.6%) same day Of 93 recorded on different day, 42 (45.2%) A-VASQIP more proximal	Mean = −1.21 SD = 1.06 Range −22, 24	.99
Serum Bilirubin	420(9.2%)	184(4.0%)	149(3.3%)	4094	4012(98.0%) same day 74(1.8%) A-VASQIP More proximal Of 82 recorded on different day, 74 (90.2%) A-VASQIP more proximal	Mean = −1.34 SD = 9.63 Range −436, 52	.99
Serum Creatinine	1 (0%) (61(1.3%) >30 days	60(1.3%)	52(1.1%)	4478	4170(93.1%) same day Of 308 recorded on different day, 87 (28.2%) A-VASQIP more proximal	Mean = −0.98 SD = 1.36 Range −23, 24	.99
	before surgery)
Serum Potassium	37(0.8%)	10(0.2%)	1(0%)	4503	3940(87.5%) same day Of 563 recorded on different day, 264 (46.9%) A-VASQIP more proximal	Mean = −1.22 SD = 3.71 Range −111, 48	.90
Serum Triglyceride	853(18.8%)	383(8.4%)	194(4.3%)	3507	3334(95.1%) same day Of 173 recorded on different day, 153 (88.4%) A-VASQIP more proximal	Mean = −4.21 SD = 43.1 Range −-756, 285	.99
Total Cholesterol	1022(22.5%)	104(2.3%)	70(1.5%)	3493	3312(94.8%) same day Of 181 recorded on different day, 157 (86.7%) A-VASQIP more proximal	Mean = −3.96 SD = 45.2 Range −756, 364	.98

^*ICC3 – single rater fixed

For Hemoglobin, SA-VASQIP had 0% missing data but 1.4% had a date >30 days before surgery, inconsistent with VASQIP’s data definition. A-VASQIP had 1.6% missing data. For the 4,475 cases with values using both methods in the 30-day preoperative period, 86.2% of the values were recorded on the same day, while A-VASQIP was more proximal for 56.2% (347 of 617) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.95).

For Hemoglobin A1c, SA-VASQIP had 13.9% missing data compared to 5.0% for A-VASQIP. For the 3,868 cases with values using both methods, 72.5% of the values were recorded on the same day, while A-VASQIP was more proximal for 74.2% (790 of 1,065) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.98).

For Low Density Lipoprotein, SA-VASQIP had 28.8% missing data compared to 20.3% for A-VASQIP. For the 2,556 cases with values using both methods, 80.0% of the values were recorded on the same day, while A-VASQIP was more proximal for 26.2% (134 of 511) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.97).

For Serum Albumin, SA-VASQIP had 12.3% missing data and 2.9% with a date >30 days before surgery, incongruent with the data definition. A-VASQIP had 13.3% missing data. For the 3842 cases with values for both methods in the 30-day preoperative period, 97.6% of the values were recorded on the same day, and A-VASQIP was more proximal for 45.2% (42 of 93) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.99).

For Serum Bilirubin, SA-VASQIP had 9.2% missing data and. A-VASQIP had 4.0% missing data. For the 4094 cases with values for both methods, 98.0% of the values were recorded on the same day, and A-VASQIP was more proximal for 90.2% (74 of 82) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.99).

For Serum Creatinine, SA-VASQIP had 0.0% missing data and 1.3% (61) with a date >30 days before surgery, incongruent with the data definition. A-VASQIP had 1.3% missing data. For the 4478 cases with values for both methods in the 30-day preoperative period, 93.1% of the values were recorded on the same day, and A-VASQIP was more proximal for 45.2% (42 of 308) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.99).

For Serum Potassium, SA-VASQIP had 0.8% missing data and A-VASQIP had 0.2% missing data. For the 4503 cases with values for both methods, 87.5% of the values were recorded on the same day, and A-VASQIP was more proximal for 46.3% (264 of 563) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.90).

For Serum Triglyceride, SA-VASQIP had 18.8% missing data and A-VASQIP had 8.4% missing data. For the 3507 cases with values for both methods, 95.1% of the values were recorded on the same day, and A-VASQIP was more proximal for 88.4% (153 of 173) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.99).

For Total Cholesterol, SA-VASQIP had 22.5% missing data and A-VASQIP had 2.3% missing data. For the 3493 cases with values for both methods, 94.8% of the values were recorded on the same day, and A-VASQIP was more proximal for 86.7% (157 of 181) of values recorded on different days. Congruence between SA-VASQIP and A-VASQIP was high (ICC = 0.98).

Discussion

The VA CDW is a well-established repository of enterprise-wide electronic health record (EHR) data. Information derived directly from the EHR (including laboratory data) are automatically updated nightly within CDW. While CDW is routinely utilized for research and is the source of a variety of operational and quality data platforms for individual facilities and their individual services, VASQIP remains the current standard for the assessment of cardiac surgical quality across VA facilities. VASQIP data are manually abstracted and verified from the EHR by local SQNs at each VA facility where surgery is performed. Each VA has at least one SQN (although some have more than one) who collects VASQIP data and engages in other local QI activities. In this study, we developed methods to automate extraction from the VA CDW the ten preoperative laboratory values and dates included in cardiac VASQIP. When automated lab results were compared to results produced by the semi-automated process currently used by VASQIP, we found a high intraclass correlations (range from 0.90 to 0.99). However, the methods had notable differences in terms of frequency of missing values, conformance to data definitions, and proximity to surgery.

For the 7 variables that allowed SQNs to enter “NS” or “No Study” (HDL, HbA1c, LDL, Bilirubin, Potassium, Triglyceride, Cholesterol), the automated method resulted in lower rates of missing data. For example, 22.5% of the HDL values were missing for the SQN-extracted vs 2.4% for the automated method. For variables restricted to labs in the 30-days before surgery that did not allow missing values (Hemoglobin, Albumin, Creatinine) the SQN-extracted approach contained 1–2.9% of values outside that defined time range as compared to 0% using the automated method. However, there were more missing data with automation because the method did not look for older values when none existed in the 30-day window. For example, 0% of the Hemoglobin values were missing for the SQN-extracted vs 1.3% were missing for the automated method, but 1.4% of the SQN values were >30 days before surgery and therefore should not have been entered. In terms of the temporal proximity of the lab value relative to the surgery, the automated and semi-automated values were identified on the same day the vast majority of the time (72% to 98%) and neither method appeared superior in terms of consistently finding values closer to the surgery date.

Given these are lab values, it is not surprising that we found very high concordance between the established semi-automated method and our fully automated approach. However, it was striking that we found notable discordance in regard to the degree to which VASQIP’s current approach to extraction is inconsistent with its own established data definitions (i.e., lab values found outside of the specified 30-day preoperative period) and that our automated methods actually captured lab values closer to the actual surgical date (as specified by the VASQIP data definition).

This work is important for three reasons. First, it suggests VASQIP’s current data collection processes, even for what some might consider a straightforward variable like a laboratory value, may need to be critically evaluated. Second, our work appears to support the concept that there are variables within cardiac VASQIP that could be fully automated without sacrificing reliability and could even improve data accuracy. Third, our work has clear implications for other national QI programs both within and outside VA (e.g., STS national databases, ACS-NSQIP, etc). The data collection workflow in these QI initiatives is similar to VASQIP in that it is based on local, manual abstraction and verification of data from the electronic health record. While data from clinical registries is believed to be the most robust source of information for driving QI activities, a unique aspect of our study is that it provides a critical evaluation of the extent to which abstracted data conform to the established data definitions within VASQIP. Our work suggests there may be value in performing similar evaluations for programs that utilize data based on local abstraction.

Stakeholders considering and comparing the accuracy of extracted variables need to interpret these data in the context of how these variables are used. Specifically, they are not used in the clinical care of patients. An incorrectly extracted laboratory value in the VASQIP dataset will not directly impact the treatment of a patient. Rather, these variables are used by the VA National Surgery Office as model covariates to produce risk adjusted estimates of 30-day complication and mortality rates for VA hospitals—information that is subsequently used for QI purposes. This is an important distinction, but not because it suggests that inaccurate data should be considered acceptable. Rather, it simply means that programmatic leaders and surgical QI stakeholders need to consider the cost-benefit of investing in resources for data extraction relative to assigning these personnel to other tasks related to more proactive performance improvement. This is particularly relevant considering our data suggest that VASQIP’s preoperative lab values can be automatically extracted with a high degree of fidelity. As such, any effort spent by SQNs to verify these data can likely be repurposed to more meaningful local QI activities.

The VASQIP data are also used by VA researchers. Any VA researcher with an IRB-approved project can freely access VASQIP data through VA’s Corporate Data Warehouse.⁶ Currently, the VASQIP files are posted annually, sometimes producing a significant time lag for researchers who need data on recent surgeries. Due to the fact that our extraction methods are now available on VA’s public facing Github repository, researchers can directly extract these variables for specific surgeries without waiting for the annual VASQIP data updates. This option will become more useful as the number of automated variables increases.

This study has several limitations. First, the work described in this study was done in the context of cardiac VASQIP data. The extent to which these methods and results will generalize to non-cardiac VASQIP or non-VA surgical quality programs is unknown. The reason for focusing only on cardiac VASQIP in this study is that it is a separate data base, with different variables, and unique data definitions as compared to non-cardiac VASQIP. However, within VA, we would anticipate our methods should generalize robustly to non-cardiac VASQIP since all lab values are ascertained from CDW and the accuracy of LOINC within VA has been supported.⁷ While we cannot speak to the external validity of our findings to non-VA programs, we believe our work does provide support to the concept that full automation of lab values is feasible. Second, we did not evaluate the accuracy of the methods in more recent years. Third, we did not manually review patient charts to determine whether the automated or semi-automated values were more accurate when a difference existed between those values. Fourth, while the SQNs may occasionally have information about lab values derived from non-VA sources, by and large both the existing semi-automated method and our automated methods do not capture non-VA laboratory values. Finally, in this study we did not evaluate our automated lab values as covariates in the VASQIP risk adjustment models (the purpose for which they are actually used). However, given the high concordance with current VASQIP methods, we would anticipate that if anything having more accurate lab data that is more consistent with established VASQIP data definitions should only enhance the reliability of the assessment hospital-level surgical quality and safety.

Conclusions

Although SQN-extracted data are widely considered the gold standard for VASQIP and other national surgical QI programs, there may be advantages to automating extraction of some of variables. For laboratory values, automated extraction had high congruence with semi-automated SQN-extracted/verified values and appeared to reduce the amount of missing and out-of-date-range data. Automation of surgical QI program data extraction could improve the accuracy, timeliness, and efficiency of this process and liberate local resources that are currently dedicated to manual extraction of data to pursue more proactive and continuous engagement in QI.