Retrospective Validation of a Surveillance System for Unexplained Illness and Death: New Haven County, Connecticut

Abstract

Objectives. This study investigated retrospective validation of a prospective surveillance system for unexplained illness and death due to possibly infectious causes.

Methods. A computerized search of hospital discharge data identified patients with potential unexplained illness and death due to possibly infectious causes. Medical records for such patients were reviewed for satisfaction of study criteria. Cases identified retrospectively were combined with prospectively identified cases to form a reference population against which sensitivity could be measured.

Results. Retrospective validation was 41% sensitive, whereas prospective surveillance was 73% sensitive. The annual incidence of unexplained illness and death due to possibly infectious causes during 1995 and 1996 in the study county was conservatively estimated to range from 2.7 to 6.2 per 100 000 residents aged 1 to 49 years.

Conclusions. Active prospective surveillance for unexplained illness and death due to possibly infectious causes is more sensitive than retrospective surveillance conducted through a published list of indicator codes. However, retrospective surveillance can be a feasible and much less labor-intensive alternative to active prospective surveillance when the latter is not possible or desired.

Globalization of food supply, intracontinental and intercontinental travel, climactic changes, and overcrowding, among other factors, have increased the mobility of microbial agents and thereby the risk of infectious diseases posed to humans. Outbreaks of Ebola hemorrhagic fever in Zaire and bubonic plague in India during the early 1990s are reminders that emerging and reemerging infectious diseases remain a threat to the health and well-being of the global community.1^–3 As demonstrated by the emergence of West Nile viral encephalitis, hantavirus pulmonary syndrome, and AIDS, the United States is not impervious to emerging epidemics.4^,5 The agents that cause AIDS, Lyme disease, Legionnaires' disease, toxic shock syndrome, and hepatitis C were identified only after the occurrence of significant morbidity or mortality.6

Reliance on traditional responsive methods to identify infectious agents may delay prevention and control efforts. While advancements in biomedical technology have allowed for more rapid identification of microbial agents, population-based surveillance networks capable of identifying trends in infectious disease symptomatology have deteriorated.7 Systematic prospective study of the epidemiology of infectious disease syndromes is needed for earlier detection of and response to emerging infections.6^,8

EMERGING INFECTIONS PROGRAMS

In 1994, the Centers for Disease Control and Prevention began providing funds for emerging infections programs in Connecticut, California, Minnesota, and Oregon to conduct population-based epidemiologic studies of infectious disease in these regions. Surveillance for unexplained life-threatening illness and death due to possibly infectious causes is being conducted at these 4 sites to prospectively identify trends and putative causative agents.

A preliminary study estimated that the annual incidence of unexplained death due to possibly infectious causes among previously healthy New Haven County, Connecticut, residents aged 1 to 49 years was 14.2 per 100 000.6 This figure was based on a retrospective review of multiple cause-of-death data included in the 1992 National Center for Health Statistics death record that selected for 77 International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes believed by the study authors to be indicative of unexplained death due to possibly infectious causes.9 Persons not previously healthy, as indicated by another series of ICD-9-CM codes, were excluded.

Beginning in August 1995, the Connecticut Emerging Infections Program conducted surveillance for unexplained illness and death due to possibly infectious causes (hereafter “unexplained illness and death”) in the 7 acute care hospitals of New Haven County. Between August 1, 1995, and December 31, 1996, 16 cases of unexplained illness and death were prospectively identified in New Haven County, yielding an annualized incidence of 1.9 episodes per 100 000 residents aged 1 to 49 years (based on an estimated surveillance population of 584 507).10 This annualized incidence rate was well below the 14.2 deaths per 100 000 population estimated for 1992. This disparity was unexpected, because the prospective surveillance aimed to identify both critical illnesses and deaths, whereas the preliminary 1992 retrospective study examined only deaths.6

Given the personnel, time, and financial resources required for prospective surveillance; the significance of the study's objective to the public's health; and the discrepancy between preliminary estimates, it was crucial to evaluate the efficacy of this system. We report on a retrospective validation study performed to assess the sensitivity of prospective surveillance for unexplained illness and death at the 7 participating acute care facilities for the period August 1, 1995, through December 31, 1996.

METHODS

Prospective Surveillance in New Haven County

Unexplained illness and death due to possibly infectious causes was defined as a case involving a previously healthy individual aged 1 to 49 years who was hospitalized in an intensive care unit or died from a critical illness of potentially infectious etiology for which no etiologic agent was identified on initial laboratory testing. Previously healthy individuals were defined as those without preexisting chronic medical conditions, including malignancy; HIV infection; chronic cardiac, pulmonary, renal, hepatic, or rheumatologic disease; or other known chronic illness (e.g., diabetes mellitus). Also, these individuals did not have compromised immune systems, and their hospitalization was not due to trauma, nosocomial infections, or toxic ingestion or exposure.

Persons not falling in the 1- to 49-year age range were excluded because of increased susceptibility to infection and increased occurrence of underlying morbidity. Intensive care units were selected as the point of access for the prospective surveillance on the basis of the assumption that individuals with life-threatening illnesses would probably be admitted to an intensive care unit during their course of treatment. The clinical definition of unexplained illness and death was based on the methods and recommendations of Perkins et al.6 with 1 major modification: Perkins et al. included only unexplained death in their investigation, whereas we included both unexplained illness and unexplained death.

Both active and passive surveillance techniques were used to identify cases. Active surveillance refers to that in which surveillance staff make regular contact with physicians or other qualified individuals or use electronic medical record systems to elicit reports of disease occurrence. In contrast, passive surveillance refers to that in which surveillance staff receive disease reports from physicians, other qualified individuals, or electronic medical record systems.11

Active prospective surveillance was conducted at hospital A, the largest hospital in the county. This surveillance consisted of routine contact with physicians, nurses, and infection control personnel to identify incidents of unexplained illness and death. In addition, an Emerging Infections Program epidemiologist (Constance J. Heye) reviewed weekday computerized intensive care unit census information to identify potential cases based on preliminary diagnoses.

Passive prospective surveillance was conducted at the 6 additional hospitals (hospitals B–G). Physicians, nurses, and infection control personnel at these hospitals were encouraged to report potential cases of unexplained illness and death to our study staff. In the case of passive surveillance, study staff did not review computerized intensive care unit census information and did not work as closely with hospital personnel to identify cases of unexplained illness and death. In both active and passive surveillance, suspected cases were referred to the study physician (Andre N. Sofair), who made the final determination as to whether a patient satisfied the case definition.

Retrospective Validation

The unexplained illness and death case definition for the retrospective validation was identical to the definition used for prospective surveillance. To conform with the inclusion criteria of the prospective surveillance, we requested data from each of the 7 participating hospitals on all patients aged 1 to 49 years admitted to an intensive care unit during the period August 1, 1995, through December 31, 1996. The following information was requested for each patient: medical record number or name (or both), date of birth, admission and discharge dates, residence zip code, and all discharge ICD-9-CM codes.

In assessing cases of unexplained illness and death, data were sorted via an Epi Info program to identify specific ICD-9-CM codes.12 A patient's chart was abstracted if his or her computerized discharge diagnoses contained at least 1 of the 77 inclusion ICD-9-CM codes previously reported by Perkins et al.6 or 7 inclusion ICD-9-CM codes added by the Emerging Infections Program (Table 1 ▶). These 7 codes were determined to be possible indicators of unexplained illness and death during a limited pilot study of the retrospective validation carried out in 1 New Haven County hospital. The conditions indicated by the 7 codes would have triggered further investigation in the prospective surveillance; thus, addition of these codes did not bias the retrospective validation.

TABLE 1—

Study Inclusion and Exclusion Criteria

Inclusion Criteria	Exclusion Criteria
Aged 1–49 years	At least 1 of the following ICD-9-CM codes delineated by Perkins et al.6: 042–044.9 and 795.8 (HIV disease), 140–239.9 (neoplasms), 250.0–250.9 (diabetes mellitus), 279.0–279.9 (disorders of the immune mechanism), 295.5 (other diseases of the spleen), 800–999 (injuries and poisonings)
Admitted to an intensive care unit, August 1996–December 1996
New Haven County resident
Exhibited infectious symptomatology or signs, including fever, rash, sepsis, shock, blood in sputum, chills, cough, diarrhea, ear pain or discharge, excessive sweating, headache, sinus congestion, sore throat	Underlying conditions, including chronic alcohol disease, chronic cardiac disease, chronic lung disease, immunosuppression, kidney disease, liver disease, rheumatologic disease
At least 1 of 84 ICD-9-CM codes
77 codes delineated by Perkins et al.6
7 codes determined to be possible indicators of unexplained illness and death by the authors: 345.3 (seizure, grand mal status), 426.0 (atrioventricular block, complete), 427.4 (ventricular tachycardia), 427.5 (cardiac arrest), 429.0 (myocarditis, no organism specified), 429.9 (heart disease, unspecified), 436.0 (acute, ill-defined cerebrovascular disease)

Note. ICD-9-CM = International Classification of Diseases, Ninth Revision, Clinical Modification

In addition, as a means of conforming with the exclusion criteria of the prospective surveillance, medical records were reviewed only if the patient's discharge record did not include any ICD-9-CM codes indicative of underlying morbidity (Table 1 ▶). Medical records identified by computerized search were reviewed by trained abstractors (Michael D. Kluger and Rajesh K. Sodhi) unaware of the case findings of the prospective surveillance. Records were carefully reviewed to determine whether patients fit all inclusion criteria and did not exhibit any of the exclusionary conditions listed in Table 1 ▶ (i.e., whether they should have been identified by the prospective surveillance). Abstracted data were recorded on a standardized screening form. Those medical records in which the patient appeared to satisfy inclusion criteria were referred to the study physician for final classification. Incomplete, inconsistent, or questionable medical records were also referred for final classification.

Cases of unexplained illness and death identified through the prospective surveillance (n = 16) were combined with cases identified through the retrospective chart review to form a reference population against which the sensitivity of the 2 surveillance techniques could be measured. Because prospective surveillance involved both active (hospital A) and passive (hospitals B–G) surveillance techniques, sensitivity was assessed separately for each technique.

We calculated rates of unexplained illness and death by using the reference population in the numerator and the estimated total surveillance population of 584 507 in the denominator.10 In addition, we calculated incidence rates based on a capture–mark–recapture estimate of the total number of cases of unexplained illness and death in the numerator (n = 48).13 This allowed for a conservative estimate of the incidence of unexplained illness and death in the study county.

RESULTS

The aggregate data set obtained from the 7 participating New Haven County acute care hospitals consisted of 3267 hospital discharge records. Filtering of these data on the basis of inclusion and exclusion ICD-9-CM codes revealed 319 admissions indicative of unexplained illness and death. Five of these patients were admitted to intensive care units twice during the surveillance period; only their first admission was reviewed. Six medical records could not be located.

Upon chart review, 78 patients whose computerized discharge data satisfied all inclusion criteria were found to violate demographic and study-period inclusion criteria. Of these 78 patients, 34 had not been admitted to intensive care units during the surveillance period, 26 did not meet the age criteria, and 18 resided outside of New Haven County. These patients were excluded from the analysis. Another 186 patients who satisfied all computerized inclusion criteria were excluded on the basis of information abstracted from their medical records. Of these 186 patients, 93 had underlying conditions, 69 had illnesses with noninfectious causes, and 24 had illnesses in which a likely infectious agent was identified.

The 44 remaining patients were referred to the study physician for final classification. Of these patients, 9 were classified by the study physician as representing “definite” cases of unexplained illness and death, and the remaining 35 were excluded because of underlying conditions (n = 19), identification of probable infectious agents (n = 11), or noninfectious etiologies (n = 5). Overall, 97% (310 of 319) of the subjects who met the inclusion criteria based solely on computerized administrative data were excluded when their medical records were reviewed. Figure 1 ▶ illustrates the flow of case identification.

FIGURE 1—

Retrospective identification of unexplained illness and death: New Haven County, Connecticut, August 1995–December 1996.

Of the 9 cases identified through retrospective validation, only 3 had been previously identified by the prospective surveillance. Conversely, only 3 of the 16 cases identified through the prospective surveillance were identified by the retrospective validation. Thus, 81% (13 of 16) of prospectively identified cases were missed by the retrospective validation. Of these 13 cases, 8 (61%) were absent from the original discharge data files received from the 7 hospitals. The remaining 5 (39%) were present in the original discharge data files. In 4 of these 5 cases, however, none of the ICD-9-CM codes in the patients' discharge data corresponded with ICD-9-CM inclusion codes used in the Epi Info program. The final case was missed as a result of human error (i.e., the trained chart reviewers incorrectly categorized the case).

Sensitivities of the retrospective validation and prospective surveillance were measured against a reference population composed of the total cases identified through either method (n = 22). Overall, the retrospective validation was 41% (9/22) sensitive, whereas the prospective surveillance was 73% (16/22) sensitive (Table 2 ▶). The retrospective validation conducted at hospital A, a major tertiary care institution, was only 21% (3/14) sensitive. The active prospective surveillance was 86% (12/14) sensitive at this hospital. The retrospective validation performed at hospitals B through G was 75% (6/8) sensitive. The passive prospective surveillance at these hospitals was 50% (4/8) sensitive.

TABLE 2—

Sensitivity of Case Identification, by Methodology: New Haven County, Connecticut, 1995–1996

Hospital	Cases Identified Retrospectively	Cases Identified Prospectively	Total Cases Identified	Retrospective Sensitivity, %	Prospective Sensitivity, %
Aa	3	12	14	21.43	85.71
Bb	2	2	2	75.00c	50.00c
Cb	1	0	1
Db	0	1	1
Eb	0	0	0
Fb	3	0	3
Gb	0	1	1
Total	9	16	22	40.90	72.72

^aActive and passive surveillance.

^bPassive surveillance only.

^cCombined value for hospitals B–G.

The annualized rate of unexplained illness and death due to possibly infectious causes among New Haven County residents aged 1 to 49 years ranged from 2.7 per 100 000 (reference population as numerator) to 6.2 per 100 000 (capture–mark–recapture estimate as numerator) during the study period.

DISCUSSION

Systematic prospective study of unexplained illness and death due to possibly infectious causes may allow for earlier detection of emerging and reemerging infections. Ideally, infectious disease surveillance should draw on and integrate multiple sources of information to produce a complete and accurate description of the epidemiology of an infectious disease.14 Such an approach would provide clues to assist in isolating an unknown agent, thereby allowing public health professionals to mobilize prevention efforts. Given the resource burden involved with extensive surveillance networks, ideal systems are not always practical; a balance must be attained on the basis of monetary considerations, desired sensitivity, and the objectives of the surveillance. Given these considerations, the Connecticut Emerging Infections Program set out to retrospectively validate its prospective unexplained illness and death surveillance system.

In the current investigation, cases of unexplained illness and death identified by prospective surveillance were combined with retrospectively identified cases to assess the sensitivity of the 2 surveillance techniques. Our findings demonstrate that, overall, prospective surveillance was more sensitive than retrospective surveillance (73% vs 41%). In other words, the majority of cases identified by the prospective surveillance would not have been identified had only retrospective surveillance relying on ICD-9-CM codes been used. The limitations of retrospective surveillance, despite the benefits, help to explain this difference in sensitivity.

Since its inception, the ICD-9-CM nosologic coding system has played a central role in clinical research and disease surveillance throughout the world. Assigned by hospitals to designate symptoms, diagnoses, and procedures and entered into administrative databases, ICD-9 coding has a number of advantages for retrospective surveillance. Most important, because administrative databases include virtually the entire patient universe, they potentially offer the best estimates of rare events.15^,16 This is critical in investigations, similar to the present study, in which incidence is expected to be extremely low.6 However, administrative data sets are not fundamentally designed for research use; therefore, their sensitivity, specificity, and timeliness in terms of any given use may not be optimal. Review of patients' medical records is necessary if greater accuracy is desired.15^,17

In a study of ischemic stroke, it was concluded that a retrospective review involving ICD-9-CM codes could be accomplished without examination of discharge summaries only if an error rate of 15% to 20% was deemed acceptable.17 In the current study, which included intricate inclusion and exclusion criteria for the retrospective validation, a much larger error rate was observed. This was largely a consequence of ICD-9-CM codes not being specifically designed to identify newly emerging infectious diseases, thereby leading to inclusion of inappropriate cases.6 Imprecise and poorly defined codes, multiple codes describing similar pathologic processes, and misleading conventions compound this problem.18 Even when ICD-9-CM codes are well defined, they may not be applied correctly by nosologists unfamiliar with the cases they are coding.6^,15 These are probably the reasons that 61% of cases identified through the prospective surveillance were absent from the administrative databases obtained from the 7 participating hospitals for the retrospective validation.

The difference in sensitivity between the prospective surveillance and retrospective validation was also due in part to the greater sensitivity of the active vs passive prospective surveillance techniques: prospective surveillance was 86% sensitive when conducted actively at hospital A and only 50% sensitive when conducted passively at hospitals B through G. In addition, the retrospective validation was found to be 25% more sensitive than passive prospective surveillance at the 6 hospitals where only passive surveillance was used. In light of the many limitations of retrospective surveillance previously discussed, these findings further illustrate the shortcomings of passive prospective surveillance.

Despite the benefits of passive surveillance—chiefly, integration of the medical community in the recognition of unusual and potentially new infections, and the smaller resource requirements to operate the system—it is understandable that this technique was not as sensitive as the active prospective surveillance and retrospective validation. Unlike passive surveillance, in which reporting relies on individuals not closely involved with or dedicated to the surveillance project, active surveillance hinges on the efforts of individuals fully committed to identifying possible cases. Underreporting by infectious disease practitioners, nurses, and physicians may be a consequence of inconvenience or a result of these individuals' simply forgetting to report a rare event given the multiple responsibilities of their daily work.

For these reasons, among others, it is well recognized that even common communicable diseases that require mandatory reporting are underreported in passive surveillance systems, thereby affecting sensitivity.19 It can be expected that diseases not requiring mandatory reporting, as in the current investigation, will be reported even less often.

Active surveillance may have limitations as well, to the extent that case identification is based on information systems designed for clinical care rather than case detection. At hospital A, where active surveillance was conducted, cases were identified by reviewing weekday intensive care unit patient census reports that included patients' preliminary diagnoses. Opportunities for missed cases included preliminary or working diagnoses that did not fit the patient profile we were seeking and instances of very short or weekend intensive care unit stays that may have resulted in the exclusion of a case patient from the real-time census report.

In our study, the annualized incidence of unexplained illness and death in New Haven County ranged from 2.7 to 6.2 per 100 000 residents aged 1 to 49 years during the study period. This conservative range is well below the annual rate of 14.2 per 100 000 found by Perkins et al. in their review of 1992 National Center for Health Statistics multiple-cause-of-death data.6 The discrepancy can be largely explained by the fact that the Perkins et al. study was limited to a computerized search of death certificates based on ICD-9-CM inclusion and exclusion codes and did not include review of medical records of each potential case meeting the inclusion criteria. In our experience, more than 97% of cases from hospital discharge databases identified by ICD-9-CM code data are excluded upon abstraction and review of medical charts.

Active prospective surveillance proved to be the most sensitive technique used, followed by retrospective surveillance and passive prospective surveillance. The ICD-9-CM coding system and administrative databases contain many inherent problems that compromise the efficacy of surveillance systems based on their use, including inaccurate and incomplete coding and recording. Therefore, suspect medical charts must be abstracted and reviewed if retrospective surveillance is to be an accurate technique. Furthermore, retrospective surveillance is limited by timeliness, which may be critical to an investigation; in the current prospective unexplained illness and death surveillance, collection of clinical specimens and exposure information was integral to the project. Despite the fiscal costs of active prospective surveillance, this system most effectively meets the objective of identifying unexplained illness and death due to possibly infectious causes.