Evaluation of a hospital-based injury surveillance system for monitoring road traffic deaths in Phuket, Thailand

Abstract

Objectives:

The objective of this study was to evaluate and injury surveillance (IS) system’s ability to monitor road traffic deaths and the coverage of road traffic injury and death surveillance in Phuket, Thailand.

Methods:

U.S. Centers for Disease Control and Prevention guidelines on surveillance system evaluation were used to qualitatively and quantitatively evaluate IS. Interviews with key stakeholders focused on IS’s usefulness, simplicity, flexibility, acceptability, and stability. Active case finding of 2014 road traffic deaths in all paper and electronic hospital record systems was used to assess system sensitivity, positive predictive value, and data quality. Electronic data matching software was used to determine the implications of combining IS data with other provincial-level data sources (e.g., death certificates, electronic vehicle insurance claim system).

Results:

Evaluation results indicated that IS was useful, flexible, acceptable, and stable, with a high positive predictive value (99%). Simplicity was limited due to the burden of collecting data on all injuries and use of paper-based data collection forms. Sensitivity was low, with IS only identifying 55% of hospital road traffic death cases identified during active case finding; however, IS cases were representative of cases identified. Data accuracy and completeness varied across data fields. Combining IS with active case finding, death certificates, and the electronic vehicle insurance claim system more than doubled the number of road traffic death cases identified in Phuket.

Conclusion:

An efficient and comprehensive road traffic injury and death surveillance system is critical for monitoring Phuket’s road traffic burden. The hospital-based IS system is a useful system for monitoring road traffic deaths and assessing risk behaviors. However, the complexity of data collection and limited coverage hinders the ability of IS to fully represent road traffic deaths in Phuket Province. Combining data sources could improve coverage and should be considered.

Introduction

In 2015, Thailand ranked first and second highest for rates of road traffic deaths in Southeast Asia and the world, respectively, with an estimated rate of 36.2 road traffic deaths per 100,000 people (World Health Organization 2015). In response to the United Nation’s proclamation of the Decade of Action for Road Safety 2011–2020 (UN Road Safety Collaboration 2011), Thailand launched its own road safety policy in an effort to reduce the burden of road traffic deaths by improving road user behaviors (e.g., helmet use) and increasing access to emergency medical services, treatment, and rehabilitation. An efficient road traffic injury and death surveillance system that can effectively monitor the frequency of road traffic injuries, deaths, and crash characteristics plays a key role in measuring achievement toward this goal.

Road traffic injury surveillance is a challenge for provinces in Thailand, because multiple injury surveillance systems, from multiple organizations, capture road traffic injuries and deaths. These organizations include the Royal Thai Police (custodian of crash investigation data), Ministry of Transportation (custodian of urban and rural road traffic information systems), Road Accident Victims Protection Company Limited (custodian of Electronic Vehicle Insurance Claim system, e-claims), National Institute for Emergency Medicine (custodian of information technology for emergency medical system), Department of Provincial Administration (custodian of national death registration), and Ministry of Public Health Bureau of Epidemiology (custodian of injury surveillance, IS). Each of these organizations and systems has different objectives, populations of interest, variables, and case definitions for road traffic injury and death. Therefore, it is difficult for provinces to accurately report the overall magnitude of road traffic deaths.

IS—a sentinel hospital-based surveillance system implemented by the Thailand Ministry of Public Health—is a key surveillance system for road traffic and other injuries. Because IS is the only surveillance system that captures road traffic injury and death risk behaviors, we sought to formally evaluate the ability of IS to capture road traffic deaths at the provincial level in Phuket, Thailand. We focused on road traffic deaths because deaths are routinely used to represent burden of road traffic injury in Thailand and because this work was part of a larger effort by the Thailand Ministry of Public Health to understand and improve surveillance of road traffic deaths. We focused on Phuket because it consistently ranks among the top 10 provinces for road traffic injury and death burden in Thailand and because it has historically had challenges in merging road traffic surveillance data from multiple sources (Thai Road accidents Data Center for Road Safety Culture [Thai RSC]) A secondary purpose of this study was to evaluate whether the coverage of road traffic death surveillance in Phuket would be improved by merging IS data with other relevant surveillance systems, including provincial-level vehicle insurance data and provincial death certificates.

Methods

Surveillance evaluation design

During January 2016, the Field Epidemiology Training Program at the Ministry of Public Health in Thailand performed a formal surveillance evaluation of IS at the provincial hospital in Phuket. This hospital serves the whole province and is the sole provincial-level hospital. Study methods included both qualitative and quantitative evaluation of IS’s ability to capture road traffic deaths during 2014 using the Centers for Disease Control and Prevention’s Updated Guidelines for Evaluating Public Health Surveillance Systems (2001).

Surveillance system attributes such as simplicity, flexibility, acceptability, and stability were assessed by in-person interviews with key stakeholders at the provincial hospital via semistructured questionnaires. Interviewed stakeholders included the hospital director, an emergency room (ER) doctor and nurse, an inpatient ward nurse, epidemiologists, a medical statistician, a medical coder, and the information technology officer. Interviews focused on the system’s importance, objectives, usefulness, operation, and key attributes.

Simplicity was assessed by determining the ease of IS in meeting operational needs. We assessed flexibility by evaluating IS’s potential to respond to new informational demands with minimal resources. Acceptability was assessed by determining the willingness of organizations and persons to participate in IS. Lastly, stability was determined by understanding the reliability and availability of IS data. Data analysis for qualitative data focused on identifying the strengths and weaknesses of IS for each system attribute.

Quantitative approaches were used to assess the sensitivity, positive predictive value, data quality, and timeliness of IS. To obtain data for the quantitative evaluation, active case finding was performed on all possible hospital data sources, both paper-based and electronic. For electronic data sources, active case finding included a review of electronic IS records, the electronic hospital database (these records are housed outside of the IS database), e-claims, and the information technology for emergency medical system (Figure 1A). Potential cases in electronic data sources were reviewed if they included an injury-related diagnosis code (S00-T88) and external cause of injury code V00-V99 (representing all modes and types of transportation), based on the International Classification Diseases (https://www.icd10data.com/ICD10CM/Codes/S00-T88), 10th Revision. Cases with an injury diagnosis code of S00-T88 and unknown external cause code (X59) were not included. Active case finding for paper-based sources included a review of ER records and medical records of deaths at this hospital (Figure 1B). For inclusion in the final data set, records from active case finding had to meet the road traffic death surveillance case definition—which included any person (including pedestrians) who died within 30 days as a result of road traffic crash and obtained medical services at the provincial hospital between January 1, 2014, and December 31, 2014. Data were abstracted for cases that met the case definition. Estimation of the true total number of road traffic deaths at this particular hospital was made by merging cases from all electronic and paper-based sources. Data were cleaned and duplicates—identified by a patient’s hospital number—were removed.

Figure 1.

Active case finding of road traffic deaths* at the provincial hospital in Phuket, Thailand, 2014: (A) Active case finding using electronic data sources and (B) active case finding using paper-based data sources. *Records were excluded if they contained an incorrect hospital number, ill-defined cause of death, cause of death not injury related, or otherwise did not meet the road traffic death surveillance case definition. Cases from both active case finding of electronic and paper-based sources were merged for a total of 187 records that met the road traffic death surveillance case definition.

Sensitivity was calculated by examining the proportion of fatal road traffic death cases captured by IS versus cases identified by active case finding. Positive predictive value was calculated by examining the number of cases electronically recorded by IS that met our case definition (i.e., true positive) in the numerator divided by the total number of cases recorded by IS (i.e., true positive + false positive) in the denominator. Representativeness was assessed by comparing the age and sex distribution of road traffic death cases in IS with road traffic death cases identified in active case finding.

Data quality was assessed by examining both accuracy and completeness of IS data. Accuracy was calculated for selected variables by (1) comparing the IS electronic record to its matched IS paper-based form and (2) comparing the IS electronic record to its matched paper-based medical record. Variables assessed included name, surname, personal identification number, age, sex, hospital number, date of crash, date of death, death at scene (yes, no), date of hospitalization, road user type (driver, passenger, pedestrian, unknown), vehicle type, risk behavior, type of injury, vital signs (i.e., systolic blood pressure, respiratory rate, and pulse rate), Glasgow Coma Scale score, and Abbreviated Injury Scale (AIS) score. Variables were selected based on their relevance in case management, road traffic injury prevention, and ability to be used for electronic data matching with other data sources. Variables examined had varying levels of availability within each data source analyzed. Completeness (i.e., no missing or blank values) for selected variables recorded electronically in IS was calculated by dividing the number of relevant IS cases with complete variable entries divided by the total number of cases in the IS database.

To assess timeliness, we determined the proportion of IS cases reported within the required 90 days for national reporting or 30 days for local strategic planning. Timeliness was calculated as the number of relevant IS cases reported on time (depending on national or local reporting) divided by total number of IS cases. Quantitative evaluation of IS was performed using Epi info Ver. 3.5 and Excel software.

Assessment of road traffic death surveillance coverage

To better estimate the total magnitude of fatal road traffic death cases in Phuket, we combined IS with other relevant provincial-level injury surveillance systems. We merged IS electronic data with active case finding, provincial-level e- claims, and provincial death certificates, using electronic data matching software developed by the Thai Ministry of Public Health (http://k4ds.psu.ac.th/rsis/rsdb). Electronic matching required a matching date of crash and either matching name and surname or matching ID. After automated matching, we manually matched records that may have been missed during the automated process (e.g., missed due to slight differences in name). We calculated the coverage of each individual surveillance system compared to that of the merged data sources to determine the effect of combining multiple data sources on overall road traffic death estimates.

Results

IS data description and flow

IS is a sentinel hospital passive surveillance system. The overall objectives of IS are to (1) improve patient treatment and referral to other appropriate treatment facilities; (2) serve as the primary database for provincial-level injury prevention and control; (3) monitor and evaluate injury programs (e.g., road traffic injury and death); (4) support injury program planning, decision making, and legislative and policy development; and (5) support hypothesis generation for injury epidemiological research. For road traffic injuries and deaths, IS is used for strategic planning and analysis of crash hot spots.

IS data flow within the study setting is described in Figure 2. Following a crash, victims are sent to the ER or autopsy room of the provincial hospital. All injured persons are first sent directly to the ER and screened prior to admittance. However, if a victim dies at the scene, some victims are sent to the ER before going to the autopsy room and some are sent directly to autopsy room. Those who are sent directly to the autopsy room are not entered into IS. At the ER screening point, IS paper forms are attached to medical charts, with data collected by ER nurses and sent to a public health officer to complete and validate data. Lastly, the public health officer enters the IS paper form into the electronic system. Complete and validated IS data are reported to the provincial public health office and academic section, where a monthly report is produced.

Figure 2.

Flow of road traffic crash victims and IS system data collection at the provincial hospital in Phuket, Thailand, 2014.

Qualitative evaluation of IS for surveillance of road traffic deaths

Overall, key stakeholders indicated that IS is a useful system for provincial strategic planning, informing policy decisions, and improvement of hospital services. Provincial data management of IS is supported by a national governmental policy. Furthermore, Phuket Province plans to open a trauma center and IS data will be used to track burden and evaluate services at this new center. Table A1 (see online supplement) summarizes results for the qualitative evaluation of IS’s simplicity, flexibility, acceptability, and stability.

Simplicity:

IS is a comprehensive system that collects data on a wide range of injuries. IS contains 104 variables and is the only hospital data source capturing risk behaviors for road traffic injuries (e.g., driving after alcohol consumption and use of a seat belt, helmet, or mobile phone). IS is not linked to the hospital database, which creates a need for duplicate data collection for ER nurses. Paper-based IS forms are not always complete because the vast number of variables are not available during initial data collection. At the provincial hospital there is only one coder entering paper-based IS data into the electronic IS system, which creates a 1-month lag for data entry. IS has web-based data entry, which has simplified data entry and dissemination processes. However, IS data must be collected first on paper forms and then entered electronically, which decreases simplicity of data collection in IS.

Flexibility:

IS has potential to adapt to new informational needs by incorporating new variables into the surveillance system without additional monetary resources. However, key informants reported that adaptation of IS would affect other attributes of this system. For example, IS simplicity and acceptability would be altered due to an additional workload for ER nurses collecting this data. At the time of this evaluation, no updates to IS had been made or were planned.

Acceptability:

Most of the interviewed stakeholders highlighted the importance of the road traffic injury burden within the province and showed a willingness to participate in IS. However, the considerable number of variables collected in the system was a burden on the 20 ER nurses performing initial data collection and the single public health officer entering electronic IS data at this provincial hospital.

Stability:

IS is a stable system that is available 24 hours a day, 7 days a week. Data collection and entry are consistent; however, if an IS paper form is lost before it is electronically entered, this case is lost permanently because paper forms are not copied or backed up. Currently there is not a system that captures the actual number of lost cases. A national mandate requires IS data to be collected for all injury cases, which increases system stability.

Quantitative evaluation of IS for surveillance of road traffic deaths

The estimated sensitivity and positive predictive value for IS were 55% (102/187) and 99% (101/102), respectively. Age and sex distribution of road traffic deaths were similar between IS and active case finding, although active case finding identified a slightly larger proportion of cases aged ≥50 years (16.56 vs. 13.59%; Figure 3). When comparing the IS electronic database to IS paper records, the accuracy of IS variables ranged from 48% for AIS score to 100% for death at the scene (Table 1).

Figure 3.

Representativeness of age and sex in the IS system as compared to active case finding at the provincial hospital in Phuket, Thailand, 2014: (A) Age group distribution of road traffic deaths in IS vs. active case finding and (B) gender distribution of road traffic deaths in IS vs. active case finding. IS system: n = 103; active case finding at provincial hospital: n = 187.

Table 1.

Data qualitya of IS system data for monitoring road traffic deaths at the provincial hospital in Phuket, Thailand, 2014.

	IS electronic
	Accuracyb (percentage of IS paper forms/medical records)
Variable	IS electronic compared with IS paper form	IS electronic compared with medical record	Completenessc (percentage of IS electronic records)
Demographics
Name	84% of 67	80% of 66	97% of 103
Surname	84% of 67	86% of 66	97% of 103
Personal identification number	88% of 67	50% of 2	67% of 103
Sex	84% of 67	100% of 66	100% of 103
Age	84% of 67	76% of 66	100% of 103
Crash
Date of crash	84% of 67	94% of 66	100% of 103
Date of death	98% of 54	89% of 66	100% of 103
Death at scene	100% of 67	—	100% of 103
Date of hospitalization	84% of 67	—	100% of 103
Type of cases	84% of 67	—	100% of 103
Vehicle	84% of 67	—	96% of 103
Risk behaviorsd	90% of 67	—	79% of 103
Treatment
Type of injury	84% of 67	—	99% of 103
Vital sign	98% of 54	50% of 36	56% of 103
Glasgow Coma Scale score	98% of 54	39% of 36	55% of 103
AIS score	48% of 54	—	43% of 103

^aNumber of records and variables available for analysis varied between data sources examined. —Indicates data not available in medical records.

^bAccuracy is an aspect of data quality and refers to correct representation of data collected and is calculated as (1) the percentage of records that matched between electronic IS and paper form IS and (2) the percentage of records that matched between electronic IS and medical records. The denominator varies because it is based on number of paper forms or medical records with data for a specific variable.

^cCompleteness is an aspect of data quality and refers to the absence of missing or unknown values for a select field in the electronic IS database.

^dIncludes driving after alcohol consumption and use of a seat belt, helmet, or mobile phone prior to/during a motor vehicle crash.

When comparing the IS electronic database to the medical record, accuracy ranged from 39% for Glasglow Coma Scale score to 100% for sex. Physical exam variables (e.g., Glasglow Coma Scale score and vital signs) were more complicated to collect and data accuracy was affected by which personnel performed the data collection (ER doctor vs. ER nurse) and timing of physical exam as it related to data collection. Completeness ranged from 48% for AIS score to 100% for sex, age, date of crash, death at scene, date of hospitalization, and type of case. Timeliness of reporting at the provincial level was 57%. However, the timeliness of data reporting to the national level was 85% (median 23 days, range: 1–661 days).

Surveillance coverage for road traffic deaths

To determine the implications of combining data sources on road traffic death surveillance, we merged IS electronic data with active case finding, provincial-level e-claims (insurance), and provincial death certificates. IS alone covered 41% (n = 102) of road traffic death cases identified across systems. Individual coverage of provincial death certificates and provincial e-claims was 52% (n = 128) and 32% (n = 79), respectively (Table 2). Active case finding at the provincial hospital provided the most coverage of any individual source (76%; n = 187) of road traffic death cases. The combination of active case finding, provincial level e-claims data, and provincial death certificates resulted in 247 road traffic death cases detected—142% more than IS alone.

Table 2.

Comparision of data coverage for the IS system alone and merged with other provincial surveillance systems capturing road traffic deaths (n = 247 deaths), including active case finding, death certificates, and e-claims.

Data source	Number	Coveragea (%)
IS	102	41
Active case finding	187	76
E-claims	79	32
Death certificates	128	52
IS + e-claims	147	60
IS + e-claims + death certificates	194	79
Active case finding + e-claims + death certificates	247	100

^aNot all Phuket hospitals are included; therefore, cases presenting at other hospitals may be missed.

Discussion

This study provides a formal evaluation of IS’s ability to monitor road traffic deaths in Phuket. We found that IS is considered a useful system and is currently used for provincial strategic planning, informing policy decisions, and improvement of hospital services. Moreover, IS allows for in-depth epidemiological analysis (i.e., person, place, time) of road traffic deaths captured within the province.

For provincial road traffic deaths, IS accurately captures true cases; however, its ability to capture the magnitude of road traffic deaths is more limited. IS only identified about half of the road traffic death cases that were found in hospital-based data systems via active case finding, in part due to cases lost within this provincial hospital. Cases could be lost for multiple reasons stemming from the complex data and patient flow within the surveillance system. Use of paper-based data collection, requiring double data entry, likely contributed to the reduced sensitivity of the system. In addition to missing cases that should have been captured, IS is unable to capture road traffic deaths that are not in the sampling frame (e.g., deaths at the scene that go directly to the autopsy room). Notably, IS displayed a high (99%) positive predictive value for road traffic death surveillance.

Data completeness and accuracy varied by data field, highlighting the importance of reporting data quality by field. The optional nature of fields contributed to this variability, which was also a reflection of the source of data (i.e., whether an ER doctor or ER nurse performs physical examination of patients) and when data were collected. Inconsistencies in data quality and completeness make IS data less useful for evaluation or for improving case management.

Overall, IS data collection is quite complex, which can affect other attributes of the surveillance system. For example, paper-based data collection and duplicate data entry likely resulted in reduced coverage. Moreover, there is a large workload for key staff collecting and entering IS data, which can delay reporting. Delays in local reporting of IS data to relevant stakeholders can cause a further delay in data analysis for targeted prevention efforts and strategic planning.

Though there is limited published information on evaluation of injury surveillance systems in the Asia-Pacific Region, a World Health Organization report from 2012 identified known ongoing injury surveillance in 5 countries (Thailand, Australia, Cambodia, China, Republic of Korea) and pilot surveillance systems in 8 countries (Bangladesh, Bhutan, India, Maldives, Mongolia, Myanmar, Sri Lanka, and Vietnam). The Thailand surveillance system appears to be similar to a system in Vietnam in terms of collected variables and responsible personnel (health staff), though data collection processes may differ (Chinh 2007). The Road Crash and Victim Information System in Cambodia is different in that it combines both police and hospital data into a single surveillance system (Parker et al. 2014).

Combining multiple provincial road traffic injury and death data sources resulted in a substantially higher detection of road traffic injury and death than using a single data source. Thus, combining data sources may be critical for obtaining a complete picture of the road traffic burden in Phuket. This result is consistent with a study performed in China that found that no single source was sufficient for detection of road traffic injuries and fatalities. (Ma et al. 2012) However, as evidenced by our study, data quality can vary within an injury surveillance system. Therefore, before data integration is done, data quality must be assessed for each individual system. Ensuring data quality is particularly critical for variables used to identify duplicate records across systems, such as date of event, name, and/or unique ID. Previous studies have shown that combining data from multiple sources is effective in better estimating the magnitude of the road traffic injury and death and providing additional information on circumstances involved in the crash (Menon et al. 2010; Petridou et al. 2009; Rojas Medina et al. 2011). However, the sustainability of data linkage is dependent on consistent participation of multiple stakeholders invested in road traffic injury and death surveillance, which will remain a key challenge (Rojas Medina et al. 2011).

Limitations to this evaluation should be considered when interpreting our findings. First, our evaluation findings for the IS system cannot be generalized to the overall function of IS in Phuket or in Thailand because we evaluated a single type of injury (i.e., road traffic deaths) within the IS system of a single provincial hospital. Second, the qualitative findings presented in this study are subject to any misinformation provided by the limited number of key informants interviewed. Third, timeliness of data entry could not be assessed because the relevant variable for “key in date” does not exist. Fourth, direct comparison of data quality between IS and medical records was limited because not all variables were collected routinely by each of these 2 sources. Fifth, the study did not review records for patients with an unknown external cause of injury or victims who went directly to the autopsy room, so the potential contribution of these cases to the overall volume of road traffic deaths is unknown. Lastly, for electronic data matching, data were combined based on date of event and name and/or unique ID, so cases lacking this information may have been double counted.

Based on the results of this surveillance evaluation, we developed several specific considerations for improving IS capacity to monitor road traffic deaths in Phuket. To increase the feasibility of considerations made for IS improvement, we present considerations by target audience.

To ensure capacity building within the provincial hospital, the findings of this evaluation support the need for routine training on IS data collection, entry, and analysis for relevant hospital personnel and an increase in the personnel managing electronic IS data collection/entry. To increase timeliness, accuracy, and sensitivity of IS data, the findings of this evaluation support the consideration of a direct connection between IS and the hospital database. This would reduce the need for double data entry.

The findings of this study demonstrated a required commitment from multiple stakeholder groups to achieve high coverage of road traffic death detection in the IS system. Therefore, the evaluators suggest the creation of a working group with representatives from the ER (primary data collectors/coders), informative section (hospital registry), academic section (technical scientific administration), and strategic section (strategic planning for hospital) to review IS data quality to ensure accuracy of data analysis and reporting. Because the hospital inpatient department and autopsy rooms also capture road traffic deaths that are not currently included in IS, evaluators suggest the inclusion of these 2 stakeholders in the working group. Lastly, to increase acceptability of IS, evaluators suggest routine (e.g., monthly) reporting of data to relevant stakeholders participating in IS data collection, entry, and analysis.

To provide the most comprehensive detection of road traffic deaths in Phuket, the findings of this evaluation support the consideration of merging multiple provincial-level injury surveillance systems (e.g., IS, e-claims, death certificates). However, evaluators do note the limited feasibility of death certificate integration with other injury surveillance systems due to challenges in accessing these data at the provincial level. To ensure the feasibility of integration of provincial-level IS, e-claims, and death certificate data, evaluators suggest a memorandum of understanding be created at the national level among Ministry of Public Health (IS custodian), Road Accident Victims Protection Company Limited (electronic vehicle insurance claim custodian), Ministry of Interior (custodian for death certificates), and other relevant organizations collecting road traffic death data. Prior to combining data sources, evaluators suggest evaluation of data quality for each participating data source.