Discharge status validation of the Chang Gung Research database in Taiwan

Abstract

Background

The Chang Gung Research Database (CGRD) is the largest multi-institutional electronic medical records database in Taiwan and has been widely used to establish evidence studies. However, the accuracy of CGRD has rarely been validated. This study aims to validate the discharge status, especially with a focus on mortality, of admission data under CGRD.

Methods

We constructed an observational study using CGRD linked with TDR to validate the discharge status. The CGRD and TDR data were obtained from the Chang Gung Memorial Hospital system and the Health and Welfare Data Science Center, respectively. The accuracy, positive predictive value (PPV), and underestimated mortality rate (UEM) were employed as indicators for validation. Year, sex, age, and the primary cause for admission (PCA) were analyzed.

Results

A total of 1,972,044 admission records under CGRD were analyzed. The overall accuracy for mortality coding on discharge status was higher than 97% within one week after discharge. The accuracy increased by year and was more than 98% after 2010. A similar result was observed in UEM; the UEM within one week was lower than 10% after 2010. These indicators varied by age group and PCA—elderly patients had relatively lower accuracy and higher UEM (approximately 11%). The presence of UEM within one week was better but varied by disease.

Conclusions

Considering the data accuracy and UEM discharge status, prioritizing the use of inpatient data after 2010 under CGRD for mortality outcome follow-up studies is recommended.

At a glance commentary

Scientific background on the subject

The CGRD is the largest multi-institutional EMR-based dataset in Taiwan with follow-up over 20 years, and it is useful for clinical studies. However, due to the nonrandom enrolment of patients and free access to medical facilities under the NHI system, selection bias and loss of follow-up were two major limitations of CGRD.

What this study adds to the field

This study used admission data of CGRD linked with Taiwan Death Registry data from HWDC to validate the discharge status and estimate the underestimated mortality. Our study results provide the reference numbers for researchers who use CGRD admission data as their research materials and improve one of the major limitations of CGRD.

The Chang Gung Research Database (CGRD) is a multi-institutional electronic medical record (EMR) database collected from the Chang Gung Memorial Hospital system (CGMH), the largest medical system in Taiwan. By 2020, the CGMH had nine branches (including three medical centers, four regional/district hospitals, and two municipal hospitals operating under entrustment), over 9000 beds, and approximately 31,500 patients per day. All EMR data in CGMH were included under CGRD except for two municipal hospitals. The CGRD includes approximately 6–20% outpatient, and 10–12% inpatient claims records compared with the National Health Insurance (NHI) research database in Taiwan [1,2].

Although the population size is smaller than that of the NHI database, the CGRD contains more clinical information (laboratory results, examination and pathological reports, medical and nursing records) than the NHI database. To date, CGRD has been widely used to establish real-world evidence studies in many fields [[3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17]]. However, previous studies pointed out that selection bias and loss of follow-up were two major limitations of CGRD [2,4,5,8,16]. First, patients in CGRD are older and more severely ill than the general population [2,18]. Therefore, the use of this data may cause selection bias. Second, because patients have free access to any medical facilities under contract with the NHI, they may be lost to follow-up if they choose to visit a facility outside of the CGMH system. In addition, there are data validation studies for the NHI database [[19], [20], [21], [22], [23]], but validation for CGRD has not yet been reported.

Discharge status, particularly mortality, is one of the major prognostic indicators for many outcome studies. However, according to traditional culture belief in “the fallen leaves can return to their roots”, some patients choose to be discharged before they die (as a call against advice discharge under critical condition [critical AAD] or discharged due to terminal [DDT]), especially terminal patients [[24], [25], [26]]. Because the patient did not die in the hospital, they are recorded as discharged instead. Many different discharge status records cover this situation. Therefore, this study aims to validate the discharge status of CGRD patients, especially those who died after discharge, by linking the database with Taiwan Death Registry (TDR) data from the Health and Welfare Data Science Center (HWDC), Ministry of Health and Welfare (MoHW). The TDR data come from an official registry-based dataset collected from death certificates. Any citizen who dies in Taiwan must by law be reported to the District Office within seven days. Therefore, we used the TDR record as the gold standard to evaluate the discharge status records of CGRD.

Methods

Design and data source

This retrospective observational study was conducted at the CGMH system under Chang Gung Medical Foundation (CGMF) in Taiwan. We validated the discharge status of admission data between 2001 and 2017. The major discharge status under CGRD included discharge, discharge with outpatient follow-up, transfer to another hospital/medical facility, discharge against medical advice (DAMA), critical AAD, and death in the hospital (in-hospital death). In general, researchers identified both in-hospital deaths and critical AAD as death outcomes and the others as live discharges.

Admission data under CGRD were selected from the “Claims for Inpatient Expenditures by Admissions”, and the TDR data were obtained from the HWDC. We proposed a special project applying HWDC to link CGRD and TDR data. Due to privacy concerns and the patient's protection, the original identification (ID) number file was sent to HWDC directly from CGMF by an information engineer and was encrypted by HWDC. All researchers could not access the original ID data, and all analyses were completed at the HWDC workstation. This study protocol and usage rights of data were approved by the Institute Review Board of the CGMH (IRB No: 201900431B0) and the HWDC (Project No: H108260) established by Taiwan's Ministry of Health and Welfare.

Setting

The purposes of this study were to validate the discharge status, with a particular focus on death outcomes; therefore, we set the death records obtained from TDR to be the gold standard. The TDR is considered complete and accurate because physicians and/or their agents must complete all death certificates by law [27]. We linked admission data under CGRD to TDR by encrypted ID number from HWDC. Considering that most admission data varied due to NHI claims requirements, we only included the last admission record per person-year for our analyses to avoid inflation of the accuracy rate.

The death certificate records confirmed the discharge status of CGRD admissions from TDR via the same encrypted personal ID number. If a patient died after discharge, we obtained the death record by linking to the TDR. We computed the days between the date of discharge and death as recorded on the death certificate. All of the death records were identified as deaths by year in this study. According to traditional cultural beliefs about death at home, death within one week after discharge was identified as true death after discharge.

After establishing and linking the study data, we computed the validation indicators by year, sex, age, and the primary cause for admission (PCA). Age was divided into six groups: under 1, 1–24, 25–44, 45–64, 65–84, and 85 and above. The PCA was divided by the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM), and the Ninth Revision (ICD-9-CM) codes into 21 categories (Appendix Table).

Data analyses and statistics

The accuracy rate, positive predictive value (PPV), and underestimated mortality rate (UEM) for tracking the prognosis of death or survival after discharge were used as indicators for validation measurement. We compared the consistency of the discharge status records under CGRD and linked them to the records in the TDR. Accuracy indicates the consistency between CGRD and the reference standard (TDR), and the PPV indicates the probability of death in CGRD as confirmed by the TDR. Moreover, we also computed the UEM to estimate how many potential underestimations were under CGRD. The UEM is the difference in discharge mortality computed via the TDR and CGRD records. When a patient had a death record from TDR within our setting time, but the discharge status code reported alive, this patient will be counted as a case of underestimated mortality.

Results

There were 1,972,044 admissions collected from CGRD and included in our analysis. A total of 110,451 deaths were confirmed via TDR, the gold standard. Table 1 summarizes the accuracy, PPV and UEM for death within one week and one month after discharge. Overall, the accuracy and PPV of discharge status within one week and/or within one month were higher than 90%. After 2008, accuracy was maintained at approximately 97–98%, and the PPV was 95%. However, the UEM varies from 78.14% to 7.84%. The UEM was >65% before 2008, plummeted to <10% for deaths within a week of discharge from 2008 to 2010 and remained stable after 2010.

Table 1.

Comparison of the discharge status of CGRD by year.

Year	alive	dead	within one week			within one month
			Accuracy (%)	PPV (%)	UEM (%)	Accuracy (%)	PPV (%)	UEM (%)
2001	116,366	2537	94.36	96.14	73.03	93.06	96.22	76.96
2002	111,895	2491	93.64	98.51	74.69	92.29	98.72	78.14
2003	96,146	2543	93.20	98.23	72.73	91.63	98.51	76.64
2004	101,720	2674	93.45	96.56	72.31	92.03	96.86	76.07
2005	98,404	2777	93.85	96.72	69.53	92.39	96.90	73.88
2006	93,719	3024	94.25	93.45	65.50	92.91	94.08	70.14
2007	91,091	7024	97.92	96.51	20.92	96.79	97.89	30.39
2008	91,214	7416	98.33	96.55	16.31	97.19	97.61	26.40
2009	91,113	7423	98.37	94.49	14.57	97.39	96.74	24.47
2010	94,152	8319	98.78	95.42	9.84	97.78	97.33	20.21
2011	98,946	8708	98.94	95.56	8.33	97.96	97.34	18.78
2012	101,807	8587	98.99	95.69	8.32	97.98	96.96	19.09
2013	104,508	8999	98.99	95.76	8.11	98.02	97.22	18.55
2014	114,280	9118	99.07	95.50	7.84	98.13	97.38	18.85
2015	127,257	9483	99.14	96.29	8.26	98.24	97.64	19.09
2016	144,970	9868	99.20	96.54	8.66	98.33	97.96	19.81
2017	181,729	9331	99.35	95.76	8.61	98.76	97.27	18.88
Total	1,861,593	110,451	97.24	95.90	32.01	96.16	97.28	40.36

Abbreviations: PPV: positive predictive value; UEM: underestimated mortality rate.

Next, we analyzed data between 2008 and 2017 since the accuracy, PPV and UEM remained relatively stable in this period. In this period, the accuracy of deaths within one month was less than 92% for patients over the age of 85, but it was more than 95% for the remaining patients. For deaths within one week, the accuracy was as high as 96%. The UEM was slightly higher for deaths within one week and increased with patient age. However, for deaths within a week, the UEM was less than 11%. These data are reported in Table 2. There was no significant difference in accuracy, PPV, or UEM between males and females. Accuracies were close to the same for deaths within one week or a month of discharge, but the UEM was lower for deaths within one week of discharge than those within one month; the values were 9.89% for males and 9.52% for females within one week and 20.70% for males and 19.68% for females within one month.

Table 2.

Sex and age comparison in discharge status of CGRD, 2008–2017.

Variable	alive	dead	within 1 week			within 1 month
			Accuracy (%)	PPV (%)	UEM (%)	Accuracy (%)	PPV (%)	UEM (%)
Sex
Male	553,220	52,879	98.77	96.44	9.89	97.60	98.04	20.70
Female	583,553	34,214	99.18	95.19	9.52	98.51	96.76	19.68
Age
under 1	36,148	418	99.51	61.00	6.25	99.64	77.51	10.25
1–24	224,651	1440	99.91	90.83	5.83	99.88	91.53	10.10
25–44	288,470	6578	99.61	89.22	7.21	99.40	90.26	16.03
45–64	329,647	26,841	99.04	96.46	8.74	98.17	97.84	18.42
65–84	237,744	39,922	97.86	96.57	10.59	95.85	98.27	21.62
85+	33,316	12,053	96.05	97.01	10.90	91.75	98.55	23.10

Abbreviations: PPV: positive predictive value; UEM: underestimated mortality rate.

There were only 19 categories fitted into the data in the PCA analyses. Table 3 shows the coding accuracy for discharge status within one week or one month was higher than 95% between 2008 and 2017. There were considerable differences in the UEM for deaths within one month of discharge. The UEM was 61% among patients who were admitted due to mental disorders. UEMs were lower among patients whose primary cause for admission was diseases of the eye and adnexa (48.33%); diseases of the skin and subcutaneous tissue (32.44%); diseases of the musculoskeletal system and connective tissue (30.20%); endocrine, nutritional and metabolic diseases (32.82%); and symptoms, signs and ill-defined conditions (34.32%). The UEM for deaths within one week of discharge was below 15% for almost all categories, but not for endocrine, nutritional and metabolic (15.28%); symptoms, signs and ill-defined conditions (18.24); external causes of morbidity and mortality (20.60%); or mental disorders (42.05%).

Table 3.

Validation of CGRD discharge status by primary cause of admission, 2008–2017.

Diseases	alive	dead	within 1 week			within 1 month
			Accuracy (%)	PPV (%)	UEM (%)	Accuracy (%)	PPV (%)	UEM (%)
Infectious and parasitic	57,431	10,555	98.50	96.74	6.23	97.57	98.38	12.50
Neoplasms	123,475	25,251	97.57	97.05	10.49	95.21	98.61	21.38
Endocrine, nutritional and metabolic	24,127	1077	99.04	94.71	15.28	97.83	96.38	32.82
Blood and blood-forming organs	4644	497	98.42	96.78	11.90	96.60	98.39	25.46
Mental disorders	9303	57	99.54	89.47	42.05	99.08	89.47	61.07
Nervous system	21,748	689	99.58	95.07	8.39	98.97	96.23	23.53
Eye and adnexa	33,809	32	99.98	93.75	14.29	99.91	96.88	48.33
Circulatory system	95,057	11,099	98.59	95.42	8.58	97.50	96.57	17.52
Respiratory system	135,092	17,360	98.37	96.05	9.74	97.16	98.21	19.10
Digestive system	136,625	8614	99.27	97.05	8.83	98.48	98.19	19.53
Skin and subcutaneous tissue	31,747	799	99.51	96.12	14.48	98.80	98.00	32.44
Musculoskeletal system and connective tissue	90,776	674	99.86	97.18	14.38	99.67	98.07	30.20
Genitourinary system	94,816	2441	99.55	96.07	12.70	98.92	97.42	29.37
Pregnancy, childbirth and the puerperium	62,203	40	99.99	92.50	5.13	99.99	95.00	7.32
Conditions originating in the perinatal period	14,294	340	97.80	6.18	12.50	98.13	21.18	7.69
Congenital abnormalities	20,252	212	99.60	64.62	4.20	99.60	70.75	11.76
Symptoms, signs and ill-defined conditions	30,870	1013	99.17	95.16	18.24	98.28	95.95	34.32
Injury and poisoning	116,152	5000	99.50	95.42	7.40	99.07	96.22	16.27
External causes of morbidity and mortality	47,555	1502	98.89	85.95	20.60	97.71	86.62	41.48

Abbreviations: PPV: positive predictive value; UEM: underestimated mortality rate.

Discussion

This study used admission data in the CGRD system linked with TDR from HWDC to validate the discharge status and estimate the UEM. These results will guide CGRD users concerning the loss to follow-up bias. The primary findings include 1. The discharge status presented a better and stable level after 2010, as the accuracy and PPV were greater than 98% and 95%, respectively, and the UEM was less than 10%. 2. There was no significant difference in accuracy or PPV for discharge status according to sex. However, accuracy and PPV were relatively lower for patients aged 65 years and older than for younger patients. The UEM was more than 20% for elderly patients who died within one month of discharge. The UEM was approximately 11% for deaths within one week of discharge. 3. The accuracy and UEM varied among the PCA groups and presented very low numbers of deaths within a week of discharge. Accuracy and UEM were good performers (accuracy >98% and UEM <15%) when measuring deaths within one week of discharge by PCA except for mental disorders (42.05%) and external causes (20.60%).

Discharge status has been recorded in CGRD since 1995. Since 2008, its accuracy has substantially increased, while the UEM has decreased. These changes may have resulted from the launch of inpatient diagnosis-related group (DRG) payments under the NHI, which began in 2009 with 54 DRGs and grew to as many as 164 DRGs in 2010 [28]. Before the DRG payment system, discharge status was independently reviewed by the NHI. Under the DRG-based payment system, discharge status has become one of the DRG indicators. Because discharge status is linked with payments, hospitals are incentivized to improve their accuracy and reduce their UEM [29].

In Taiwan, critical AAD (also called IDD) is a unique behavior among terminal patients [[24], [25], [26],30,31]. Some Taiwanese, especially elderly citizens, believe dying at home is thought to bring good fortune [26]. Therefore, it becomes problematic for hospitals and physicians to record the discharge status of these patients, and our results are consistent with this practice. For this reason, data on elderly patients had lower accuracy and relatively higher UEM. Fortunately, although the accuracy and UEM rates were relatively higher, the recorded deaths within one week of discharge were lower than 11%, which may be acceptable for many studies.

The accuracy and UEM present large differences among different PCA groups in CGRD. Due to mental disorders, unclear diagnoses, and external causes, PCA included patients with a lower PPV and higher UEM. This result was similar to Hsieh et al. [32], who indicated that coding accuracy was higher for severe life-threatening illnesses. Further research using CGRD admission data needs to consider our report when choosing appropriate study patients.

To date, there have been many validation studies on the NHI database [2]. Some studies have validated CGRD/NHI data in their original research topics [[33], [34], [35]]. Three studies have used the CGRD and NHI databases to compare and analyze differences in the data characteristics [1,2,18] to understand the representativeness of CGRD relative to national data. However, there has been little validation of CGRD. Further study may focus on validating CGRD data and/or using laboratory, examination and pathological data under CGRD to validate the NHI database.

Conclusions

In conclusion, we found that the discharge status of CGRD admission data was more than 98% accurate and had less than 10% UEM when using data after 2010. In tracking the prognosis of inpatient discharge using CGRD, we have several recommendations: 1. Data after 2010 will result in higher accuracy and lower UEM discharge status for the general user, especially within the one-week mortality discharge status after discharge. 2. To study discharge status in elderly patients, it is advisable to use data on deaths within one week of discharge. 3. The use of follow-up discharge status to study mental disorders, external causes, and unclear disease diagnoses with symptoms, signs, and ill-defined conditions should be avoided.

Ethical statement

This study and data usage were approved by the Institutional Review Board of the Chang Gung Medical Foundation (IRB No: 201900431B0) and the Health and Welfare Data Center (Project No: H108260) established by Taiwan's Ministry of Health and Welfare.

Funding

This study was supported by Chang Gung Memorial Hospital, Linkou (grant number CGRPG3J0011).

Role of the funder

The funder had no role in any part of this study.

Conflicts of interest

The authors have no financial or ethical conflicts of interest to report.

Appendix.

Appendix Table 1.

Study classification of the primary cause for admission.

ICD-10	ICD-9	Classification of Disease
A00-B99	001–009	Infectious and parasitic diseases
C00-D48	140–239	Neoplasms
D50-D89	280–289	Diseases of the blood and blood-forming organs
E00-E90	240–279	Endocrine, nutritional and metabolic diseases
F00–F99	290–319	Mental and behavioral disorders
G00-G99	320–359	Diseases of the nervous system
H00–H59	360–379	Diseases of the eye and adnexa
H60–H95	380–389	Diseases of the ear and mastoid process
I00–I99	390–459	Diseases of the circulatory system
J00-J99	460–519	Diseases of the respiratory system
K00–K93	520–579	Diseases of the digestive system
L00-L99	680–709	Diseases of the skin and subcutaneous tissue
M00-M99	710–739	Diseases of the musculoskeletal system and connective tissue
N00–N99	580–629	Diseases of the genitourinary system
O00–O99	630–676	Pregnancy, childbirth and the puerperium
P00–P96	760–779	Conditions originating in the perinatal period
Q00-Q99	740–759	Congenital abnormalities
R00-R99	780–799	Symptoms, signs and abnormal clinical and laboratory findings
S00-T98	800–999	Injury, poisoning and other consequences of external causes
V01–Y98	E800-E999	External causes of morbidity and mortality
Z00-Z99	V01–V82	Factors influencing health status and contact with health services