Assessing the accuracy of the International Classification of Diseases codes to identify abusive head trauma: a feasibility study

Abstract

Objective

To assess the accuracy of an International Classification of Diseases (ICD) code-based operational case definition for abusive head trauma (AHT).

Methods

Subjects were children <5 years of age evaluated for AHT by a hospital-based Child Protection Team (CPT) at a tertiary care paediatric hospital with a completely electronic medical record (EMR) system. Subjects were designated as non-AHT traumatic brain injury (TBI) or AHT based on whether the CPT determined that the injuries were due to AHT. The sensitivity and specificity of the ICD-based definition were calculated.

Results

There were 223 children evaluated for AHT: 117 AHT and 106 non-AHT TBI. The sensitivity and specificity of the ICD-based operational case definition were 92% (95% CI 85.8 to 96.2) and 96% (95% CI 92.3 to 99.7), respectively. All errors in sensitivity and three of the four specificity errors were due to coder error; one specificity error was a physician error.

Conclusions

In a paediatric tertiary care hospital with an EMR system, the accuracy of an ICD-based case definition for AHT was high. Additional studies are needed to assess the accuracy of this definition in all types of hospitals in which children with AHT are cared for.

INTRODUCTION

Approximately 3.4 million reports of child maltreatment were made to child welfare agencies in 2011 in the USA.¹ There were 676 569 substantiated victims and the victimisation rate for the federal fiscal year 2011 (October 2010–September 2011) was 9.1/1000 children.¹ Nationally, an estimated 1570 children died from abuse and neglect during calendar year 2011.¹

Abusive head trauma (AHT) is the leading cause of death from child abuse as well as a leading cause of morbidity from traumatic brain injury (TBI) in young children. As a result, primary prevention of AHT has been a major focus of the Centers for Disease Control and Prevention (CDC) as well as other public health organisations. In order to plan, implement and assess the success or failure of prevention programmes, it is critical that an accurate measurement of the incidence of AHT can be done before, during and after an intervention.

Numerous studies over the past 15 years have used a wide variety of data sources to calculate the incidence of AHT. These data sources can be divided into two main categories—direct case ascertainment^{2– 6} and large-scale data sets, most of which use International Classification of Diseases (ICD) codes.^7–12 Direct case ascertainment is the gold standard for case identification,^13,14 but is time intensive and unlikely to be feasible or cost-effective on a large scale. In contrast, ICD codes are collected as part of standard medical care and are available without charge as part of several large-scale data sets, such as the Kids’ Inpatient Database (KID).¹⁵ The KID data set samples hospital discharges and not individual patients and therefore could have a duplicate admission(s) for the same subject. The ability to capture all cases of non-fatal AHT using ICD-9-CM (clinical modification) codes is theoretically possible since virtually all children diagnosed with AHT who do not die prior to hospital admission are admitted for further evaluation and treatment. With the exception of a single small study by Hooft and colleagues,¹⁶ however, no study has directly compared direct case ascertainment with ICD codes. The use of ICD codes to capture data on fatal cases of AHT, particularly those cases which never reach a hospital, is outside the scope of the current study, but has been the focus of other studies.^17,18

Determining which ICD-9-CM codes are likely to be the most sensitive and specific for the diagnosis of AHT is critical. In March 2008, the CDC convened an expert panel of paediatricians, child abuse paediatricians, ICD coders and state health department personnel to develop ICD code-based case definitions for AHT. The panel developed both broad and narrow operational case definitions, with the former emphasising greater sensitivity of case ascertainment and recommended for general population-based surveillance and the latter emphasising specificity and recommended for more focused assessments (eg, individual-level case study). The case definitions define AHT either by a single ‘shaken baby syndrome’ code—995.55—or by one of a list of clinical diagnosis codes (an ICD code) combined with one of a list of external cause of injury codes (an E-code). The CDC subsequently applied the panel’s broad operational case definition to the Nationwide Inpatient Sample (NIS) Database from the Healthcare Cost and Utilization Project (HCUP) to measure the incidence of non-fatal AHT in children less than 5 years old.¹⁹ The results demonstrated an incidence similar to other published studies suggesting that the definition is valid. However, the sensitivity and specificity of this definition when compared with case surveillance have not yet been determined.

In addition to the need to evaluate the accuracy of the CDC definition of AHT at the time of initial diagnosis, it is important to ensure that any patient with AHT not be inappropriately coded as AHT during any subsequent readmissions, thereby overestimating the incidence of AHT, an issue raised by Parks and colleagues in their publication.¹⁹ Because the NIS Database represents hospital discharges rather than individual patients, this could potentially be a limitation of applying an ICD code-based definition to such a database.

The objectives of this study were therefore to (1) assess the sensitivity and specificity of the CDC-derived ICD-9-based definition of AHT using a cohort of children evaluated for AHT by a Child Protection Team (CPT) at a tertiary care children’s hospital with a completely electronic medical record (EMR); (2) identify the source of miscoding (eg, coder vs physician) in subjects who were miscoded and (3) determine whether readmissions for children with a previous diagnosis of AHT would be improperly recoded as AHT.

MATERIALS AND METHODS

Children were eligible for inclusion if they were evaluated for possible AHT by the CPT at Children’s Hospital of Pittsburgh of UPMC during their hospitalisation. Consultation by the CPT is similar to other physician consultations and includes a history, physical examination and overall assessment, including determining the likelihood that abuse occurred. Documentation of this consultation is part of the medical record in the same way that any other subspecialist consultation is part of the medical record. This consultation was the data source for the designation of subjects as AHT versus non-AHT TBI. Children given a diagnosis of ‘probable’ or ‘definite’ AHT by the CPT were considered to have AHT; children who were either not diagnosed with abuse or diagnosed with ‘possible’ abuse by the CPT were considered non-AHT TBI. The diagnosis of the CPT physician who evaluated each subject during his/her hospitalisation was, therefore, considered the gold standard. Using the assessment of a CPT as a gold standard is one which is frequently used in clinical research related to child physical abuse and one which we have used previously.^3,20,21

For each subject, up to five ICD diagnosis codes and up to two E-codes, coded by hospital coders, were collected. Each subject was then classified as a AHT or non-AHT TBI using the CDC operational definition in table 1. The CPT assessment was then compared with the ICD code classification. Importantly, for purposes of coding, diagnoses which are ‘definite’ or ‘probable’, but not ‘possible’, receive diagnosis codes. All subjects for whom the CPT assessment did not match the ICD code classification were considered to have been miscoded. For each of these cases, the CPT consultation and hospital discharge summary were reviewed to determine whether the miscoding was the result of ‘coder error’ or ‘physician error.’ A coder error was defined as miscoding despite a clear statement of abuse or lack of abuse by either the CPT physician or within the discharge summary. A physician error was defined as a lack of clear statement of abuse or an incorrect statement by a physician other than the CPT physician. For all coder errors, the identification number associated with the coder was documented in order to determine whether a small number of coders were contributing to a large proportion of the coding errors. For all physician errors, the name of attending physician was collected.

Table 1.

ICD-9-CM and external cause of injury codes for defining non-fatal AHT in children under the age of 5 years

	ICD-9-CM injury code	ICD-9-CM external cause of injury or abuse code
Definite or probable AHT	781.0–781.4, 781.8, 800, 801, 803, 804.1–804.4, 804.6–804.9, 850, 851, 852.0–852.5, 853.0, 853.1, 854.0, 854.1, 925.1, 950.0–950.3, 959.01, 995.55*	E960.0, E967, E968.1, E968.2, E968.8, E968.9, E987, E988.8, E988.9 995.50, 995.54, 995.59

^*Does not require a cause (E) code.

AHT, abusive head trauma; ICD, International Classification of Diseases.

In addition to the ICD diagnosis codes and E-codes, the following data were collected for each subject: group (AHT vs non-AHT TBI), year of admission, race (white, non-white), gender, age at injury, insurance (private, public/uninsured) and whether he/she died during the hospitalisation, For cases, the number of readmissions within 6 months of the initial diagnosis was documented and for each readmission, the ICD diagnosis and E-codes were compared with the CDC definition and each admission was classified as meeting or not meeting the CDC definition of AHT. The CDC definition defines AHT either by the single 995.55 code which refers specifically to ‘shaken baby syndrome’ or by a combination of one of a group of injury codes (eg, subdural haemorrhage code) plus an E-code which designates the injury as non-accidental (table 1 and see online supplementary appendix A).

The study was approved by the institutional review board of the University of Pittsburgh. The decision was made a priori to enrol up to 120 subjects with AHT and 120 with non-AHT TBI. All subjects were part of database of children who were evaluated for AHT by the Children’s Hospital of Pittsburgh of UPMC (CHP) CPT beginning in November 2006 and ending at the time that data collection for the current study began in July 2012. The subjects who were most recently evaluated were enrolled first (eg, subject 1 was the subject who was most recently evaluated). There were 120 consecutive AHT cases which occurred from March 2008 until July 2012; there were 107 non-AHT TBI subjects in the entire database.

Statistical analysis

Descriptive statistics were used to describe the characteristics of the subjects. Sensitivity and specificity of the ICD-9-CM clinical diagnosis/E-codes for classifying the subjects as AHT and non-AHT TBI were also calculated. 95% CIs were calculated for all measures of sensitivity and specificity. All calculations were performed using SPSS 19.0 (SPSS Inc., Chicago, Illinois, USA).

RESULTS

Demographics

A total of 227 subjects were enrolled: 120 had AHT and 107 had non-AHT TBI (table 2). Of these, ICD-9-CM codes were available for 98% (117/120) of AHT subjects and 99% (106/107) of non-AHT TBI subjects. The 117 AHT subjects and 106 non-AHT TBI subjects make up the study subjects. Overall, 64% of all subjects were men, 81% had public insurance or were uninsured and 74% were white. The median (range) age was 3.9 (0.2–39.7) months. There was no difference in the proportion of men or the racial distribution between AHT and non-AHT TBI subjects. AHT subjects were significantly older than non-AHT TBI subjects (mean (SD): 10.1 (10.6) vs 4.3 (4.0) months, p<0.000) and were significantly more likely to have public insurance (94% vs 62%, p<0.000). Non-AHT TBI subjects were significantly less likely to die (0% vs 10.2%, p<0.000).

Table 2.

Demographic characteristics of AHT and non-AHT TBI subjects

	AHT (n=117)	Non-AHT TBI (n=106)	p Value
Gender (% men)	66	61	NS
Mean (SD) age (in months)	10.1 (10.6)	4.3 (4.0)	<0.000
Race (% white)	74	74	NS
Insurance (% public insurance)	94	62	<0.000

AHT, abusive head trauma; TBI, traumatic brain injury.

Coding

Of the 117 AHT subjects, 91.5% (107/117) were properly coded as AHT; 21.5% (23/107) of the properly coded AHT subjects were coded with a 995.55 code and the remaining 78.5% (84/107) were coded with a combination of ICD-9-CM clinical diagnosis and E-codes. Ten AHT subjects were miscoded; nine of the 10 had clinical diagnosis codes which were part of the CDC operational definition, but had either no E-code (n=3) or E-codes which were accident E-codes rather than abuse E-codes (n=6). One AHT subject had two ICD-9-CM clinical diagnosis codes which were not part of the CDC operational definition (432.1 (subdural haemorrhage, non-traumatic) and 781.99 (other symptoms involving nervous and musculoskeletal systems)).

Of the 106 non-AHT TBI subjects, 96% (102/106) were properly coded. Of the four non-AHT TBI subjects who were miscoded, all four were incorrectly coded with an ICD-9 code in addition to one of the three physical abuse codes (995.50, 995.54 and 995.59). The sensitivity and specificity of the CDC operational definition were therefore 91.5% (95% CI 85.8 to 96.2) and 96.2% (95% CI 92.3 to 99.7), respectively (table 3).

Table 3.

Classification of AHT subjects and non-AHT TBI subjects by CPT and ICD-9 codes

		CPT diagnosis
		AHT subjects (n=117)	Non-AHT TBI subjects (n=106)
ICD-9-CM code	AHT subjects	107	4	111	Positive predictive value 96.4% (107/107+4)
	Non-AHT TBI subjects	10	102	112	Negative predictive value 91.1% (102/102+10)
		117	106	223
		Sensitivity 91.5% (107/107+10)	Specificity 96.2% (102/102+4)

AHT, abusive head trauma; CPT, Child Protection Team; ICD, International Classification of Diseases; TBI, traumatic brain injury.

Sources of coding error

Of the 10 AHT subjects who were miscoded, all 10 were coder errors. Of the four non-AHT TBI subjects who were miscoded, three were coder errors and one was a physician error. The coder errors were distributed throughout the coding period with one error in 2006, two errors in 2007, four errors in 2008, two errors in 2009, one error in 2010, zero errors in 2011 and four errors in 2012. The coder errors were not, however, distributed among all coders. Of the 13 coder errors, 62% (8/13) of them were made by only two coders each of whom had four errors. All three of the coder errors for non-AHT TBI subjects (eg, coding 995.5 in a patient without AHT) were made by the same coder. The single physician error was the result of the attending physician who dictated the discharge summary and listed AHT as a discharge diagnosis. The CPT physician assessed that the infant had ‘possible AHT’.

Readmissions

Of the 117 AHT subjects, 105 survived to discharge. Of the 105 who survived, 26% (27/105) were readmitted at least once during the 6 months after the initial admission. There were 54 admissions among these 27 children with a range of 1–4. One of the readmissions was for a second episode of abuse. The other 53 were not due to AHT, but due to complications of the AHT (eg, seizures and placement of a ventriculoperitoneal shunt). The single readmission for a repeated AHT was properly coded as AHT, and the 53 other readmissions were properly coded not to meet the CDC’s operational definition of AHT.

DISCUSSION

This is the first study to compare the accuracy of the CDC’s ICD-9-based definition of AHT with clinical assessment by a hospital-based CPT, a commonly used gold standard for the diagnosis of child abuse. The high sensitivity and specificity demonstrated in our study suggest that the ICD-9-based definition is a valid method of case ascertainment and may be able to be used for research purposes in place of data collected at the practitioner level. It is important to recognise that our data were collected at a single level I paediatric trauma centre with a completely EMR and a hospital-based CPT. This set of circumstances was felt to represent the ‘best case scenario’ for the use of the ICD-9-CM-based definition for several reasons. First, because all records are electronic, coders do not need to interpret the handwriting of physicians and other medical professionals. Misreading of physician handwriting can be an important reason for coding error. In many cases, a clear list of diagnoses was included within the CPT consultation and/or discharge summary, thereby maximising the possibility of proper coding. Second, because Children’s Hospital of Pittsburgh of UPMC is a level I trauma centre with a large catchment area, the coders code hundreds of cases of trauma and approximately 30 cases of AHT annually. Although this is a small number of AHT cases, it represents a much larger sample size than would be found at smaller or non-trauma hospitals. Finally, because the hospital has a CPT, the coders know that the consultation written by the CPT physician is the place in the medical record where they are most likely to be able to find clear information about the aetiology of a child’s injury and therefore insure that the correct ICD-9-CM and E-codes are used. The CPT physician is more likely than a non-CPT physician to clearly document in his/her consultation the likelihood of abuse; this makes proper coding by the coder more likely.

Our sensitivity of 91.5% is higher than the sensitivity of 77% described in the smaller study by Hooft and colleagues. The difference is not statistically significant due to the small sample size and wide CIs.¹⁶ Interestingly, there was a difference in the source of coding errors between the two studies—all of miscoded AHT subjects in this study were the result of coder errors, while in the study by Hooft and colleagues, the errors were equally divided into coder errors and physician errors. If one were to eliminate all the physician errors in the Hooft study, the sensitivity of the ICD-9-CM codes would have been 88% (95% CI 78% to 98%), much closer to our sensitivity. Importantly, however, the Hooft study included all types of physical abuse; children with AHT made up only a small proportion of all the subjects.

The fact that the coding errors were distributed over the coding period, but were made by only a few coders suggests that feedback to individual coders rather than a change in the way in which coders are trained could result in significant improvements in the sensitivity of the codes.

Just over 20% of the AHT subjects in the current study were coded with a 995.55 code rather than with an injury code and accompanying E-code. In contrast, in the study by Parks and colleagues, 37.6% of the AHT cases were coded with the 995.55 code.¹⁹ This suggests that there may be significant variability among hospitals in the way in which AHT cases are coded. This variability, however, is likely to decrease with the introduction of ICD-10 codes in October 2014. Since there is no equivalent of a 995.55 code, all AHT cases will be coded with a combination of a diagnosis/injury code and a certainty of abuse code. A recent study by Fujiwara and colleagues examining the incidence of AHT using ICD-10 codes in Canada demonstrated a very similar incidence to studies which have used the ICD-9-CM codes.⁹ Therefore, while the transfer to ICD-10 will change the actual codes in the CDC-based definition and the AHT coding scheme will need to be carefully translated to ICD-10 equivalency, the data from Fujiwara and colleagues suggest that the change to the new coding scheme should not change whether the approach itself is accurate.

In addition to assessing the accuracy of the CDC’s ICD-9-CM definition at the time of AHT diagnosis, we were able to demonstrate that AHT codes were not used when children with AHT were readmitted with non-AHT diagnoses in the 6 months after their initial diagnosis. While almost all of these readmissions were related to sequelae of AHT (eg, placement of a ventriculoperitoneal shunt, replacement of a feeding tube and seizures), none of them was miscoded as AHT, an issue which would overestimate the incidence of AHT by potentially including the same subject multiple times.

The finding that AHT subjects were significantly older than non-AHT TBI subjects might initially seem contrary to what one would expect. This finding, however, is consistent with our hospital’s protocol that all children less than 1 year of age with an intracranial injury not due to a motor vehicle crash get evaluated by the CPT physician. As a result, there are many infants being evaluated for abuse. The majority of these children are assessed as having injuries which are the result of non-AHT TBI (eg, infant falling out of a car seat and sustaining a skull fracture) and/or poor parenting (eg, parent falls sleep and drops child on the floor and child sustains a skull fracture with a subdural haemorrhage). In contrast, children over the age of 1 are only assessed by the CPT if the attending physician has a specific concern for AHT. As a result, there are fewer children greater than 1 year of age who are evaluated by the CPT and assessed as having non-AHT TBI. In fact, if one removes all the AHT subjects greater than 1 year of age from the data set, then the mean (SD) age in months of the AHT subjects and non-AHT TBI subjects is the same (4.3(2.7) vs 4.0 (3.0) months, p=0.6), which supports the hypothesis that the age difference is result of the hospital protocol.

Limitations

There are two important limitations which need to be recognised. First, the current study assessed the accuracy of the CDC’s ICD-9-CM-based definition in a specific setting—a single level I paediatric trauma centre with an EMR and a CPT. As a result, our conclusions cannot be extended beyond this scenario. Even among level I paediatric trauma centres there may be differences in coding practices. Further assessments of accuracy will need to be done to evaluate the utility of the CDC’s definition for evaluating incidence in primary or secondary prevention studies. Data from the HCUP Database demonstrate that only 20.5% of children with moderate/severe AHT are treated in a level 1 or 2 paediatric hospital, 42.3% are treated in an adult level 1 or 2 hospital, 19.8% are treated in a hospital which is both a paediatric and an adult level 1 and 2 hospital and 17.4% are cared for in a non-trauma hospital (Monica Vavilava, University of Washington, Seattle, personal communication). The ICD-9-CM-based definition will need to be evaluated in each of these settings.

Another second potential limitation is the use of a CPT as a gold standard for identification of AHT subjects. Use of a hospital-based CPT assessment can underestimate the number of AHT cases since additional information is sometimes obtained after discharge by Child Protective Services or police. While the CPT physicians would likely document in the medical record the additional data and how these data would change the likelihood of abuse, this documentation would be placed in the medical record after discharge and would, therefore, not be used by the coder to code the diagnosis during the hospital admission. The possibility of additional information becoming available after discharge is less likely in cases of AHT compared with other forms of physical abuse since hospital admissions often last many days or even weeks; this allows for some investigation by Child Protective Services and police to be done during the hospitalisation. There is also potential for variability among individual experts in their assessment of whether a child has been abused.^22,23 The use of a CPT assessment rather than the assessment of a single expert opinion, however, is more likely to result in an accurate diagnosis. It is important to recognise that neither active case surveillance nor ICD-9-CM coding will be an accurate representation of the true incidence of AHT if cases of AHT are not brought for medical attention or are brought to medical attention, but are not recognised as abuse. While the proportion of children who are not brought to medical attention is unknown, one large study demonstrated that close to one-third of children with AHT were initially misdiagnosed.²⁴

In conclusion, our data suggest that the sensitivity and specificity of the CDC operational case definition for AHT are high in the setting in which we evaluated it: a paediatric tertiary care centre with a CPT and an EMR. Additional studies will be needed to assess the accuracy of the definition in all types of hospitals in which children with AHT may be cared for, including other tertiary care children’s hospitals, adult trauma centres and non-trauma centres.

What is already known on the subject

• ►Abusive head trauma (AHT) is the leading cause of death from child abuse.
• ►Accurate measurement of the incidence of AHT is critical in order to assess when prevention programmes are effective.
• ►The Centers for Disease Control and Prevention developed an International Classification of Diseases-9 code-based definition of AHT.

What this study adds

• ►The Centers for Disease Control and Prevention (CDC)’s International Classification of Diseases (ICD)-9 code-based definition of abusive head trauma (AHT) is both sensitive and specific when compared with direct case ascertainment, the gold standard for AHT diagnosis.
• ►The CDC ICD-based definition will need to be evaluated in other types of hospitals where children with AHT are cared for before it can be used as part of primary prevention programmes.