Evaluation of California Non-Comprehensive Death File Against National Death Index

Abstract

The National Death Index (NDI) by the Centers for Disease Control and Prevention and Death Master File (DMF) by Social Security Administration are the two most broadly utilized data files for mortality outcomes in clinical research. NDI's high costs and the elimination of protected death records from California in DMF calls for alternative death files. The recently emerged California Non-Comprehensive Death File (CNDF) serves as an alternative source for vital statistics. This study aims to evaluate the sensitivity and specificity of CNDF compared to NDI.

Of 40,724 consented subjects in the Cedars-Sinai Cardiac Imaging Research Registry, 25,836 eligible subjects were queried through the NDI and the CDNF. After exclusion of death records to establish the same temporal and geographic availability of data, NDI identified 5,707 exact matches, while CNDF identified 6,051 death records. CNDF had a sensitivity of 94.3% and specificity of 96.4% compared to NDI exact matches. NDI also produced 581 close matches: all were verified as deaths by CNDF through matching death date and patient identifiers. Combining all NDI death records, CNDF had a sensitivity of 94.8% and specificity of 99.5%. CNDF is a reliable source for obtaining mortality outcomes and providing additional mortality validation. The use of CNDF can aid and replace the use of NDI in the state of California.

1. INTRODUCTION

All-cause mortality is widely accepted as a definitive outcome measurement for risk assessment in various disease states[1,2]. However, the availability of up-to-date reports, matching accuracy to other methods, and costs for ascertainment constitute the main challenges for researchers in obtaining mortality data. Comprehensive national databases, such as the Social Security Administration’s Death Master File (DMF) and the National Death Index (NDI), have traditionally been utilized to determine vital status in large cohorts. The major limitation of DMF is the exclusion of death records since November 2011 from “protected states”, such as California. At the time of this change, 4.2 million of the 89 million records in the DMF were removed, and a projected 1 million records have been excluded each subsequent year[3]. As a result, while it may be costly, researchers in California rely heavily on the NDI, which is a centralized database of death records obtained from the vital statistics offices of all states, not subject to the “protected states” exclusion, making it the preferred source for mortality ascertainment in the United States [4].

California Non-comprehensive Death File (CNDF), published by the California Department of Public Health Vital Statistics, has recently emerged as a low-cost alternative source for vital status determination, encompassing all deaths which have occurred and been registered in California. Therefore, despite being a current or former California resident, if the death occurred outside of California, the death event would not be available through CNDF. The CNDF receives its files from the California Electronic Death Registration System and contains all death records from 2005 when it was established through the present. The CNDF is updated monthly, thereby facilitating up-to-date queries [5].

NDI remains the gold standard for death ascertainment; however, no previous publication has investigated the sensitivity and specificity of CNDF in death ascertainment. Thus, we conducted a study to compare death ascertainment using CNDF and the NDI in a cohort of 40,724 subjects who underwent cardiac stress testing and consented to the Cedars-Sinai Cardiac Imaging Research Registry.

2. METHODS

Study participants

The eligible study population consisted of 40,724 subjects undergoing noninvasive cardiac imaging studies at Cedars-Sinai Medical Center (CSMC) between January 1, 1991 and December 31, 2011 who consented to enroll in the Cedars-Sinai Cardiac Imaging Research Registry. The conduct of this registry was approved by the CSMC Institutional Review Board. Each subject completed the informed consent process during radionuclide stress testing with positron emission tomography or single-photon emission computed tomography.

Query preparation and process

The list of subjects was sent to Cedars-Sinai Research Informatics and Scientific Computing Core (RISCC) to request information required for death record queries. At the time of query in May 2018, NDI had completed death record data up to December 31, 2016 and early release file of death records in 2017. The authors elected to obtain NDI death records up to December 31, 2016. The initial step was to obtain the subject’s last physical visit to CSMC and the Social Security Numbers (SSNs) from RISCC.14,888 patients who had physician visits, imaging tests, or emergency room visits at CSMC from January 1, 2017, to December 31, 2018, were excluded from the NDI query since no death records would be available for these patients. The remaining 25,836 records were submitted to NDI via a secure file transfer protocol. These records included the subject's available first name, middle initial, last name, date of birth, SSN, sex, and medical record number. NDI provides free query service for duplicate records for people who may have abbreviations of the names. Therefore, to increase the probability of matching to the NDI, duplicates records were created by our research staff to generate different variations of the same name, maiden names for females and diminutive forms of the given names (e.g., patient may report their name as “Billy” at our facility but may have "William” for the legal name).

California Non-comprehensive Death File query

RISCC obtains CNDF data from the California Department of Public Health. This data file includes over 3 million death records from 2005 to current. Using a matching algorithm, RISCC linked the CSMC Electronic Health Record data with the CNDF data using patients’ first and last names, date of birth, and sex. Our research team then submitted a request for RISCC to match the subjects from this study to the CNDF. RISCC returned the subject list with deceased patients flagged via CSMC secure server.

National Death Index query

We reviewed NDI death records according to the NDI User's Manual, which explained exact matches and class codes. NDI Exact Matches are defined as all identifiers submitted (first name, middle initial, last name, state of birth, birth month, birth year, sex, all digits of SSN) exactly matching with NDI records. These NDI Exact Matches are accepted as definitive deaths.

NDI assigns class code to close matches according to the availability of patient identifiers and the degree of matching of these identifiers, as shown in Supplemental Table 1 (National Center for Health Statistics). Class code 1 is given to exact matches. Class code 2 through 4 represent cases with close matches with probabilistic scoring, which is the weighted sum of the probability of matching between NDI and the submission record. According to NDI recommendations, records can be accepted as a true match if the followings conditions are met: class code 2 with a probabilistic score greater than 44.5; Class code 3 cases with a probabilistic score greater than 37.5; Class code 4 cases with a probabilistic score greater than 32.5 [6]. Class code 5 cases are considered false matches. This study used NDI exact matches as the gold standard to evaluate CNDF first. Then, NDI close matches with assigned class codes were verified with CNDF using patient identifiers and date of death.

Exclusion and Inclusion Criteria

After the return of queries, matched death records from NDI and CNDF both underwent further exclusions to generate the same geographic and temporal data availability. NDI contains death records from all states, while CNDF contains death records from California. NDI death records from non-California states were excluded.

CNDF contains comprehensive death records only after 2005, whereas NDI contains death records prior to 2005 but has an upper time limit to December 31, 2016 at the time of the query. To align the time span of data availability, NDI death records from prior to the year 2005 were excluded, and CNDF death records in the years 2017 and 2018 were excluded.

2.1. Statistical analysis

Continuous variables were displayed as mean ± standard deviation, while categorical variables were displayed as frequencies and percentages. Sensitivity, specificity, positive predictive value, and negative predictive value were calculated for NDI and CNDF linkage results. All statistical analyses were performed using STATA version 13 (StataCorp LP, College Station, TX, USA).

3. RESULTS

The characteristics of the subjects are shown in Table 1. The patient population's mean age was 60.95 years, and 55.4% of the patients were male. Nearly three-quarters of the patients were Caucasian and 14.4% were African American. 85.25% of the population identified as non-Hispanic.

Table 1.

Subject Characteristics

	Number of Subjects	Percentage of Total Number of Subjects (%)
Age
Less than 40	2,292	5.63
40-50	6,419	15.77
50-60	10,701	26.28
60-70	10,564	25.94
70-80	7,500	18.42
Greater than 80	3,249	7.98
Sex
Male	22,545	55.36
Female	18,179	44.65
Race
Caucasian	30,204	74.17
African American	5,863	14.40
Asian	2,067	5.08
American Indian or Alaska Native	56	0.14
Native Hawaiian or Other Pacific Islanders	32	0.08
Other	884	2.17
Unknown Race	1,618	3.97
Ethnicity
Non-Hispanic	34,757	85.35
Hispanic	3,082	7.57
Unknown	2,885	7.08

The results of our linkage of the NDI and CNDF queries are shown in Figure 1. Out of 25,836 eligible subjects queried through NDI, NDI returned 6,726 exact matches, which were accepted as deaths without further verification, whereas CNDF query returned 6,536 matches. The dataset then underwent further exclusion to establish the same geographic and time span of death record availability. First, 627 non-California NDI exact matches were excluded because CNDF only contains comprehensive death information in California to establish the same geographic death record availability. Then, to establish the same time span of death record availability, 392 NDI exact matches prior to the year 2005 were excluded because CNDF does not contain comprehensive death information prior to 2005. In addition, 485 CNDF death records from the years 2017 and 2018 were excluded. Following these exclusions, 5,707 deaths were identified by the NDI exact matches, and 6,051 by the CNDF death records remained.

Figure 1.

Flow Chart of Death Record Inclusion and Exclusions and Results of National Death Index and California Non-Comprehensive Death File Linkage

Altogether, 6,378 death records were identified by either NDI or CNDF. The concordance of matches was as follows: there was agreement of matches by both NDI exact matches and CNDF in 5,380 (84.4%) of the total deaths identified; 327 (5.1%) of the total deaths were identified by NDI only; and 671 (10.5%) were identified by CNDF only. CNDF identified 94.2% of 5,707 NDI exact matches. Table 2 shows the sensitivity and specificity of CNDF compared to NDI. Out of the 25,836 eligible subjects queried in our study, CNDF has a sensitivity of 94.3 (95% CI: 93.6-94.9) and a specificity of 96.4 (95% CI: 96.1-96.7) when compared to NDI exact matches, the gold standard used in the study.

Table 2.

All Verified National Death Index Death Records Linkage with California Non-Comprehensive Death File

	Deceased		Alive		Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)
	NDI	Not in NDI	NDI	Not in NDI
CNDF compared to NDI exact matches	5,380	671	17,955	327	94.3 (93.6-94.9)	96.4 (96.1-96.7)	88.9 (88.1-89.7)
CNDF compared to all NDI matches	5,961	90	17,955	327	94.8 (94.2-95.3)	99.5 (99.4-99.6)	98.5 (98.2-98.8)
Gender
Female	2,409	376	7,756	147	94.2 (93.3-95.1)	95.4 (94.9-95.8)	86.5 (85.2-87.7)
Male	2,971	295	10,199	180	94.3 (93.4-95.1)	97.2 (96.9-97.5)	91.0 (89.9-91.9)
Race and Ethnicity
American Indian	5	2	31	1	83.3(35.9-99.6)	93.9 (79.8-99.3)	71.4 (29.0-96.3)
Asian	205	34	1055	13	94.0 (90.0-96.8)	96.9 (95.7-97.8)	85.8 (80.7-89.9)
Black/African American	774	109	2669	62	92.6 (90.6-94.3)	96.1 (95.3-96.8)	87.7 (85.3-89.8)
Native Hawaiian	4	1	17	0	100.0 (39.8-100.0)⁎⁎	94.4 (72.7-99.9)	80.0 (28.4-99.5)
White	4,230	501	12,437	234	94.8 (94.1-95.4)	96.1 (95.8-96.5)	89.4 (88.5-90.3)
Other/Unknown	162	24	1,746	17	90.5 (85.2-94.4)	98.6 (98.0-99.1)	87.1 (81.4-91.6)

^⁎⁎One-sided, 97.5% confidence interval

Comparison of NDI versus CNDF after assessment for close matches

Additional analysis was performed on the 671 subjects identified by CNDF but not NDI exact matches. Besides exact matches, NDI also produces close matches that either have nonmatching patient identifiers or nonmatching SSN digits (Supplementary Table 1). These close matches can be accepted when verified via matching death date and patient identifiers with another mortality source, such as death files or medical records. For each close match, NDI assigns a class code indicating the likelihood of a true match and we performed an analysis to investigate the linkage of these 671 CNDF with the NDI close matches.

Figure 2 shows the distribution of cases with class codes and their matching process. Out of the 671 subjects identified by CNDF but were not part of the NDI exact matches, 503 records were assigned class code 2 by NDI, indicating that these records were very close matches; no records assigned class code 3 due to missing patient demographics; 36 CNDF death records were assigned class code 4; and 43 CNDF cases were assigned class code 5. The remaining 89 CNDF cases were not assigned class codes because there were no corresponding NDI death records. In total, 581 (99.8%) of 582 NDI death records with class codes matched with CNDF on patient identifiers and death dates, verifying these death records without further reviewing medical records. When comparing CNDF to all NDI death records (including exact and close matches), CNDF has a sensitivity of 94.8 (95% CI: 94.2-95.3) and 99.5 (95% CI: 99.4-99.6).

Figure 2.

Verification of 671 California Non-comprehensive Death File Records with National Death Index Close Matches Using Class Code

Out-of-state death records

Of the 6,726 NDI exact matches, we also analyzed the percent of non-California death records based on years of follow-up. Total non-California NDI exact matches account for 9.6% of the total NDI exact matches. The percent of non-California deaths was 6.53% at five years of follow-up. As the years of follow-up increased, the percent of non-California death records increased as well as shown in Table 3. The percentage of non-California death records of total NDI exact matches was 15.60% at 21-25 years of follow-up.

Table 3.

Non-California National Death Index Exact Matches Over Years of Follow-up

Years of follow-up	Percent Non-California Death Records of Total NDI Exact Matches (%)
1-5	6.53
6-10	9.50
11-15	11.13
16-20	12.44
21-25	15.60

Cost comparisons of NDI versus CNDF

 Table 4 shows the comparison of variables available to query in NDI and CNDF as well as other comparisons such as data range and costs. NDI has more variables for matching, such as SSN, maiden name, and middle initial. When comparing the cost of NDI and CNDF queries, NDI query incurred a cost of $12,464.20 whereas CNDF incurred a cost of $0 for Cardiac Imaging Research Team (RISCC provided CNDF file) and $400 for RISCC [[6], [7]].

Table 4.

Comparisons of Variable Available for Query in National Death Index and California Non-comprehensive Death File

	National Death Index	California Non-comprehensive Death File
Data variable	●First name ●Last name ●Middle initial ●Maiden name ●Sex ●Date of birth ●Date of death ●Social Security number	●First name ●Last name ●Sex ●Date of birth ●Date of death
Data Range	●Death Occurring from 1979 through 1-2 years prior to the time of request ●Death occurring within the United States	●Death occurring from 2005 to present ●Death occurring in California
Pros	●Option for cause of death ●A national database containing death data from all states	●Current data ●Affordable
Cons	●Expensive ●Not updated to current month/year	●No death data from outside of California ●Does not contain records prior to year 2005 ●Does not contain SSN
Costs	$0.21 per subject per year of death searched $5.00 per decedent for the cause of death $350.00 Initial submission of user records $100.00 Each Subsequent submission (At the time of study) Example: first-time query of 10,000 all-cause mortality records in California will cost $350+0.21*10,000=2,450	$400 First year of usage $65/year Additional year Quarterly update 4*$65=$260 Monthly update 12*$65=$780 Weekly update 52*$65=$3,380 (At the time of study) Example: first-time query of 10,000 all-cause mortality records in California will cost $400

4. DISCUSSION

Within cardiac practice, the strength of the association between test parameters and mortality rates is used to establish the ability of test variables to predict the future risk of adverse clinical outcomes Accurate ascertainment of vital status is critical to the assessment of a test's true prognostic efficacy. Comprehensive national databases, such as the Social Security Administration's Death Master File and the NDI, have traditionally been utilized for determining vital status among large patient cohorts. The availability of up-to-date records and weighing accuracy versus utilization costs for data search constitute the main challenges that researchers face in obtaining mortality data. To our knowledge, our study is the first study to assess the sensitivity and specificity of the CNDF as a means for obtaining accurate all-cause mortality data for epidemiological research.

Out of the 25,836 eligible subjects queried in our study, CNDF has a high sensitivity and specificity of 94.3% and 96.4% respectively. Importantly, we assessed the use of class code analysis to increase the agreement between CNDF and NDI. Patients who have missing or incorrect digits of SSNs with partially matched identifiers are given class codes by NDI, indicating close match. These close matches can be accepted as definitive deaths with another death file or documentation of death through medical records review. CNDF validated 581 NDI close matches, demonstrating that CNDF can be extended to encompass both NDI exact matches and close matches. When comparing CNDF to all NDI death records (including exact and close matches), CNDF has an increased sensitivity of 94.8% and specificity of 99.5%.

The main advantages of CNDF include the low cost as well as its up-to-date data. CNDF saved $12,464.20 in addition to staff wages compared to NDI while CNDF showed an excellent sensitivity and specificity in identification of death records. CNDF is updated monthly compared to the delay of at least one year of NDI death records, thereby facilitating up-to-date queries [5].

However, there are also limitations to the CNDF as a death file. First, the CNDF lacks SSN, which could lessen the degree of certainty of exact matches as studies have reported that SSN is the most sensitive identifier[8,9]. The missing SSNs lessen the degree of matching in the NDI query, resulting in close matches that cannot be definitively accepted as death. But our study shows that even lacking SSN, CNDF can be used to achieve a high agreement rate with NDI. Second, we used NDI as the gold standard, but inevitably, the NDI itself will miss some deaths due to unregistered deaths, demographic factors, incorrect patient information, missing SSNs [9]. Overall sensitivity with the NDI reportedly ranges from 87% to 97% [[9], [10], [11], [12], [13], [14], [15]]. Third, CNDF does not include data prior to the year 2005 or death records from states other than California. To this end, we also analyzed NDI non-California death records to investigate the number of patients who have moved and deceased in another state. This analysis helped us examine the extent of impact of emigration on a state-based death file. Only 6.53% of NDI exact matches were non-California records in follow-up extending to 5 years of follow-up. This percentage increased to 15.60% for NDI exact matches in follow-up extending to 25 years, demonstrating that more people have moved out of California and deceased in another state over time. Therefore, this loss to follow-up with emigration may affect study result as 10-25% loss may impact cohort study results depending on the statistics need to be calculated [16]. Lastly, while NDI provides cause of death information, CNDF does not include cause specific information and as a result, for studies that require cause of death information, CNDF would not be utilized.

In addition to this study, many researchers have investigated other death files in comparison to NDI. Cowper et al noted from multiple studies that Beneficiary Identification and the Record Locator System (BIRLS) had sensitivities ranging from 87.0% to 97.9% for identifying true deaths, and the Social Security Administration file’s sensitivity was approximately 83.0% [9]. They also reported that the Department of Veteran Affairs database’s sensitivity for identifying true deaths was 33%, and NDI’s sensitivity ranged from 87% to 97% [[9], [10], [11], [12], [13], [14], [15]]. In California, the Automated Mortality Linkage System has also been used and found to have a sensitivity of 0.89 compared to 0.94 in NDI when evaluated against four death clearance procedures[17]. Our study is the first to compare CNDF to NDI and contributes to the existing literature by validating a state-based death file.

In conclusion, our study validates CNDF as a new acquisition method for obtaining accurate data regarding all-cause mortality. As such, CNDF can serve as an important tool for researchers seeking to calculate the death rates associated with acute and chronic diseases, assess survival after treatments, and assess the comparative ability of clinical variables for predicting mortality [10]. For instance, in cardiac imaging, the prediction of all-cause mortality is commonly used to assess the relative potency of clinical and imaging variables as tools for physician decision-making [18,19]. Having accurate and cost-effective death files, such as the CNDF, may help researchers to reduce the costs of epidemiological research as well as to improve data quality.

The following are the supplementary data related to this article.

Supplementary Table 1

Class Code Number System (Adapted from National Death Index User’s Guide)

Funding

This work was supported by the Dr. Miriam and Sheldon G. Adelson Medical Research Foundation and NIH National Center for Advancing Translational Science (NCATS) UCLA CTSI Grant Number UL1TR001881.

Conflict of Interest

The authors report no conflicts of interest.