Clinical and Epidemiologic Profiles for Identifying Norovirus in Acute Gastroenteritis Outbreak Investigations

Abstract

Background

Noroviruses are the leading cause of acute gastroenteritis (AGE) outbreaks in the United States. However, outbreaks attributed to norovirus often lack confirmation by diagnostic testing. Clinical and epidemiologic profiles, such as the Kaplan criteria (vomiting in >50% cases, mean incubation period of 24–48 hours, mean duration of illness for 12–60 hours, and negative bacterial stool culture), have been used to distinguish norovirus outbreaks from those caused by bacteria.

Methods

Kaplan criteria were evaluated among 10 023 outbreaks reported to the National Outbreak Reporting System (NORS) during 2009–2012. An alternate profile for distinguishing norovirus outbreaks from outbreaks caused by nonviral etiologies was identified using classification and regression tree (CART) modeling. Performance of the Kaplan criteria and alternate profile were compared among laboratory-confirmed outbreaks.

Results

The Kaplan criteria were 63.9% sensitive and 100% specific in discriminating norovirus from nonviral outbreaks, but only 3.3% of norovirus and 1.2% of nonviral outbreaks reported all criteria. Clinical and epidemiologic characteristics identified with CART modeling (ratio of proportion of cases with fever to the proportion of cases with vomiting <1, proportion of cases with bloody stool <0.1, proportion of cases with vomiting ≥0.26) were 85.7% sensitive and 92.4% specific for distinguishing norovirus from nonviral outbreaks and were applicable to more than 8 times as many outbreaks compared with the Kaplan criteria.

Conclusions

Compared with the Kaplan criteria, the CART-derived profile had higher sensitivity and broader application in reported AGE outbreaks. Thus, this alternate profile may provide a more useful tool for identifying norovirus during outbreak investigations.

An estimated 179 million cases of acute gastroenteritis (AGE), defined as diarrhea or vomiting, occur annually in the United States, of which 142 million (79%) are of unknown etiology [1, 2]. Even in outbreaks, which are more intensely investigated for specific cause than sporadic cases, a confirmed etiology is identified in less than half of reported outbreaks, due largely to lack of specimen collection or testing [3, 4]. In 2009, the Centers for Disease Control and Prevention (CDC) launched a new national surveillance system, the National Outbreak Reporting System (NORS), that improved and expanded upon 2 existent food and waterborne disease surveillance systems [3, 5]. The NORS allows for local, state, and territorial health departments to report all outbreaks of foodborne, waterborne, or other enteric disease regardless of etiology and confirmation status (confirmed, suspected, or unknown). Moreover, the NORS provides a national surveillance system for all pathways of AGE outbreaks in the United States, including those that are spread through direct person-to-person contact, animal contact, contaminated environments, and other or unknown transmission routes. Detailed information on temporal trends, specific pathogens, and exposure pathways provides a greater understanding of AGE epidemics and can help guide appropriate interventions for future outbreaks [6].

Noroviruses are the leading cause of both sporadic and outbreak-associated cases of AGE in the United States [3, 7]. Laboratory confirmation of norovirus relies primarily on molecular methods (ie, real-time polymerase chain reaction [RT-PCR]) as rapid clinical tests such as enzyme immunoassays have demonstrated inadequate sensitivity for routine use [5, 8]. In 1982, prior to development of RT-PCR assays for norovirus in the 1990s, Kaplan et al. proposed a set of criteria to distinguish outbreaks caused by norovirus from outbreaks caused by bacterial pathogens [9]. These criteria include vomiting in ≥50% cases in an outbreak, an average incubation period of 24–48 hours, an average duration of illness of 12–60 hours, and lack of identification of a bacterial etiology from stool culture. These criteria were subsequently shown to be highly specific (99%) and moderately sensitive (68%) when evaluated among foodborne disease outbreaks reported to the CDC in 1998–2000 [10]. Other clinical and epidemiologic criteria that describe the relative proportion of specific symptoms among outbreak-associated cases, such as ratios of fever-to-vomiting and diarrhea-to-vomiting, have also been proposed for identifying likely outbreak etiologies in the absence of laboratory confirmation [11, 12].

The objectives of this study were 2-fold: (1) to reevaluate the Kaplan criteria and other potential clinical and epidemiologic characteristics using recent outbreak surveillance data reported through the NORS and (2) to identify an alternate, data-driven clinical and epidemiologic profile to better distinguish norovirus from other outbreak etiologies. Such data can help guide public health practitioners during outbreak investigations to rapidly identify a likely etiology in the absence of confirmatory laboratory testing and thereby direct public health action to reduce illness.

METHODS

Data Source and Definitions

Data for all AGE outbreaks that occurred during 2009–2012 were extracted from the NORS. Outbreak data were collected from all 50 US states and the District of Columbia. Finalized reports of outbreaks with date of first illness onset of January 1, 2009, through December 31, 2012, and any of the following primary transmission modes were included: foodborne, person-to-person, environmental, animal contact, and indeterminate/unknown transmission.

The CDC provides guidance to designate confirmed outbreak etiology through appropriate laboratory testing; reported outbreak etiologies that did not meet these criteria are considered suspect. Four categories of single-etiology outbreaks were defined and used in this analysis. Confirmed norovirus outbreaks were defined as single-etiology outbreaks with laboratory confirmation of norovirus in at least 2 cases. Suspected norovirus outbreaks were defined as single-etiology outbreaks that reported norovirus but lacked laboratory confirmation. Confirmed nonviral etiology outbreaks were defined as single-etiology outbreaks with laboratory confirmation that excluded any viral etiologies such as astrovirus, hepatitis A virus, rotavirus, or sapovirus. Unknown etiology outbreaks were defined as outbreaks that reported no etiology. Outbreaks that did not meet the criteria for any of these 4 categories were excluded from analysis.

Clinical and Epidemiologic Characteristics of Outbreaks

Eight clinical and epidemiologic characteristics extracted from NORS data, including those in the Kaplan criteria, were examined among the 4 etiology groups. Characteristics included proportion of cases with bloody stools, proportion of cases with diarrhea, proportion of cases with fever, proportion of cases with vomiting, proportion of cases with fever divided by the proportion of cases with vomiting (fever-to-vomiting ratio), proportion of cases with diarrhea divided by the proportion of cases with vomiting (diarrhea-to-vomiting ratio), median incubation period, and median duration of illness. As these characteristics are not required fields in the NORS, not all outbreak reports included such information. The proportion of cases for each symptom (eg, diarrhea, fever, vomiting) was calculated by dividing the number of cases with symptoms by the total number of cases for whom symptom information was available. Because of non-normal distributions, medians and interquartile ranges (IQRs) were calculated, and nonparametric Kruskal-Wallis tests followed by post hoc Steel-Dwass all-pairs comparison tests were used to assess differences in characteristics compared with confirmed norovirus outbreaks [13]. Analysis was performed with SAS, version 9.4 (SAS Institute Inc., Cary, NC).

For confirmed norovirus and nonviral outbreaks, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were evaluated for diagnostic performance of the Kaplan criteria and their individual characteristics. Evaluation of Kaplan et al.’s fourth criterion of a negative bacterial culture was excluded from this study to focus solely on clinical and epidemiologic criteria instead of laboratory criteria. In addition to the Kaplan criteria, the fever-to-vomiting ratio ≤1 and diarrhea-to-vomiting ratio <2.5 proposed by Hedberg et al. and Dalton et al., respectively, were also evaluated [11, 12]. OpenEpi, version 3.03 (Dean AG, Sullivan KM, Soe MM), was used to calculate the sensitivity, specificity, PPV, and NPV with 95% confidence intervals. Likelihood ratios were also calculated in OpenEpi to assess the diagnostic value of each characteristic, where values close to 1 are considered less useful [14].

Classification and Regression Tree Model Development

An alternative profile was identified through classification and regression tree model development (CART) modeling with 2939 confirmed norovirus and 1573 nonviral outbreaks using the rpart package in R, version 3.1.1 (R Foundation for Statistical Computing, Vienna, Austria). Clinical and epidemiologic characteristics assessed by CART modeling included the proportion of cases with bloody stools, proportion of cases with diarrhea, proportion of cases with fever, proportion of cases with vomiting, the fever-to-vomit ratio, and the diarrhea-to-vomit ratio. Model selection was performed by reviewing a complexity parameter with the lowest cross-validation error within 1 standard error of the best tree.

Evaluation of CART-Derived Characteristics

Characteristics derived from CART modeling were assessed to determine predictive accuracy among confirmed norovirus and nonviral outbreaks. Characteristics best able to distinguish norovirus from nonviral outbreaks and with >50% of norovirus outbreaks at the terminal node were evaluated. Characteristics with different cutoff values as splitting variables in the model were also evaluated for predictive accuracy, where norovirus was the predominant terminal node.

The diagnostic accuracy of CART-derived characteristics among confirmed norovirus and nonviral outbreaks was evaluated with Cohen’s kappa statistic for agreement, sensitivity, specificity, PPV, NPV, likelihood ratios, and number of outbreaks, with all characteristics reported and compared with the Kaplan criteria. Suspect norovirus outbreaks and unknown etiology outbreaks were evaluated with CART-derived characteristics to assess the proportion of outbreaks that were likely attributable to norovirus.

RESULTS

A total of 9527 AGE outbreaks reported through the NORS during 2009–2012 were included in the analysis. Of these, 2939 were confirmed norovirus outbreaks, 1321 were suspected norovirus outbreaks, 1573 were confirmed nonviral outbreaks, and 3694 were unknown etiology outbreaks (see Supplementary Tables 1 and 2 for additional details).

Clinical and Epidemiologic Characteristics

Distributions and frequency of reporting for 8 clinical and epidemiological characteristics among the 4 outbreak etiology groups are shown in Table 1. Overall, the proportion of cases with diarrhea was the most frequently reported characteristic in all 4 groups, reported in 67.8%–74.9% of outbreaks. Conversely, the median incubation period was the least frequently reported characteristic in all 4 groups, reported in 2.1%–5.4% of outbreaks. There were no significant differences between suspected norovirus and confirmed norovirus outbreaks among median values of any of the characteristics assessed, except for fever-to-vomiting ratio (P = .002). In contrast, distributions of confirmed nonviral characteristics compared with confirmed norovirus characteristics were all significantly different (all P < .0001). Among unknown etiology outbreaks, the distributions of median incubation period, median duration of illness, proportion of cases with diarrhea, and proportion of cases with fever were significantly different compared with those among confirmed norovirus outbreaks (all P < .01).

Table 1.

Distribution of Acute Gastroenteritis Clinical and Epidemiologic Characteristics Among Confirmed Norovirus, Suspected Norovirus, Confirmed Nonviral, and Unknown Etiology Outbreaks^ain the NORS 2009–2012

Characteristic	No. (%) With Characteristic	Median	IQR (Q1, Q3)b	P c
Median incubation period, h
Confirmed norovirus	156 (5.3)	30.0	(24.0, 37.0)	Ref
Suspected norovirus	43 (3.3)	30.0	(24.0–34.0)	.92
Confirmed nonviral	33 (2.1)	60.0	(48.0, 120.0)	<.0001
Unknown	200 (5.4)	24.0	(11.3, 34.0)	<.0001
Median duration of illness, h
Confirmed norovirus	1192 (40.6)	48.0	(24.0, 48.0)	Ref
Suspected norovirus	378 (28.6)	42.5	(24.0, 48.0)	.63
Confirmed nonviral	177 (11.5)	144.0	(96.0, 204.0)	<.0001
Unknown	1184 (32.1)	36.0	(24.0, 48.0)	<.0001
Proportion of cases with bloody stools
Confirmed norovirus	759 (25.8)	0.0	(0.0, 0.0)	Ref
Suspected norovirus	355 (26.9)	0.0	(0.0, 0.0)	.56
Confirmed nonviral	827 (53.6)	0.3	(0.04, 0.50)	<.0001
Unknown	911 (24.7)	0.0	(0.0, 0.0)	.29
Proportion of cases with diarrhea
Confirmed norovirus	2200 (74.9)	0.86	(0.75, 0.98)	Ref
Suspected norovirus	944 (71.5)	0.88	(0.74, 1.00)	.30
Confirmed nonviral	1261 (81.7)	1.00	(0.99, 1.00)	<.0001
Unknown	2506 (67.8)	0.94	(0.75, 1.00)	<.0001
Proportion cases with fever
Confirmed norovirus	1542 (52.5)	0.22	(0.08, 0.40)	Ref
Suspected norovirus	686 (51.9)	0.26	(0.09, 0.45)	.05
Confirmed nonviral	1012 (65.5)	0.58	(0.33, 0.83)	<.0001
Unknown	1473 (39.9)	0.18	(0.01, 0.43)	.006
Proportion of cases with vomiting
Confirmed norovirus	2164 (73.6)	0.72	(0.58, 0.87)	Ref
Suspected norovirus	919 (69.6)	0.71	(0.56, 0.89)	.69
Confirmed nonviral	1031 (66.8)	0.39	(0.22, 0.60)	<.0001
Unknown	2369 (64.1)	0.75	(0.49, 1.00)	.8
Fever-to-vomiting ratio
Confirmed norovirus	1506 (51.2)	0.31	(0.12, 0.56)	Ref
Suspected norovirus	665 (50.3)	0.39	(0.17, 0.64)	.002
Confirmed nonviral	795 (51.5)	1.33	(1.00, 2.00)	<.0001
Unknown	1329 (36.0)	0.33	(0.05, 0.71)	.84
Diarrhea-to-vomiting ratio
Confirmed norovirus	2139 (72.8)	1.33	(1.00, 1.44)	Ref
Suspected norovirus	908 (68.7)	1.12	(1.00, 1.50)	.8
Confirmed nonviral	899 (58.2)	2.00	(1.44, 3.40)	<.0001
Unknown	2214 (59.9)	1.00	(1.00, 1.67)	.88

Abbreviations: IQR, interquartile range; NORS, National Outbreak Reporting System.

^aSingle-etiology outbreaks with norovirus reported as laboratory confirmed (laboratory confirmation of 2 or more cases, n = 2939), norovirus reported as suspected (laboratory confirmation in fewer than 2 cases, n = 1321), or a confirmed nonviral etiology (n = 1573) (Supplementary Table 2); unknown etiology outbreaks reported without any suspected or confirmed etiology (n = 3694).

^bIQR (Q1, Q3) is the interquartile range, where Q1 is the 25th percentile and Q2 is the 75th percentile.

^c P values were obtained by Kruskal-Wallis tests with post hoc Steel, Dwass comparisons with laboratory-confirmed norovirus outbreaks.

Clinical and Epidemiologic Characteristics for Norovirus

The diagnostic performance of the individual criteria proposed by Kaplan et al., Hedberg et al., and Dalton et al. in discriminating between confirmed norovirus and nonviral outbreaks is summarized (Table 2). Among individual components of the Kaplan criteria, the median duration of illness performed well, with a high likelihood ratio of 12.9, 79.6% sensitivity (95% confidence interval [CI], 77.2%–81.8%), and 93.8% specificity (95% CI, 89.3%–96.5%). Additionally, 1192 (40.6%) confirmed norovirus outbreaks reported information for duration of illness, but only 178 (11.3%) nonviral outbreaks reported this information. The fever-to-vomiting ratio had high sensitivity (97.8%; 95% CI, 96.9%–98.4%), a likelihood ratio of 2.3, and was reported in approximately 50% of both norovirus and nonviral outbreaks.

Table 2.

Clinical and Epidemiologic Characteristics Used to Discriminate Between Norovirus or Nonviral Etiologies With NORS-Reported Outbreaks From 2009–2012

Clinical and Epidemiologic Characteristics	Confirmed Norovirus (%)a	Confirmed Nonviral (%)b	Likelihood Ratio	Sensitivity (95% CI), %	Specificity (95% CI), %	Positive Predictive Value (95% CI), %	Negative Predictive Value (95% CI), %
Median duration of illness, hc
12–60	949 (79.6)	11 (6.2)	12.9	79.6 (77.2–81.8)	93.8 (89.3–96.5)	98.9 (98.0–99.4)	40.7 (36.1–45.6)
Not 12–60	243 (20.4)	167 (93.8)
Total No. of outbreaks with all criteria	1192 (40.6)	178 (11.3)
Proportion with vomitingc
≥50%	1857 (85.8)	438 (41.7)	2.1	85.8 (84.3–87.2)	58.5 (55.5–61.4)	80.9 (79.3–82.5)	66.8 (63.7–69.7)
<50%	307 (14.2)	612 (58.3)
Total No. of outbreaks with all criteria	2164 (73.6)	1050 (66.8)
Median incubation period, hc
24–48	117 (75.0)	14 (41.2)	1.8	75.0 (67.7–81.1)	58.8 (42.2–73.6)	89.3 (82.9–93.5)	33.9 (23.1–46.6)
Not 24–48	39 (25.0)	20 (58.8)
Total No. of outbreaks with all criteria	156 (5.3)	34 (2.2)
Fever-to-vomiting ratio
≤1	1451 (97.8)	343 (42.8)	2.3	97.8 (96.9–98.4)	57.2 (53.7–60.6)	80.9 (79.0–82.6)	93.3 (90.7–95.2)
>1	33 (2.2)	458 (57.2)
Total No. of outbreaks with all criteria	1484 (50.5)	801 (50.9)
Diarrhea-to-vomiting ratio
<2.5	2050 (95.8)	558 (61.3)	1.6	95.8 (94.9–96.6)	36.7 (35.6–41.9)	78.6 (77.0–80.1)	79.8 (75.8–83.3)
≥2.5	89 (4.2)	352 (38.7)
Total No. of outbreaks with all criteria	2139 (72.8)	910 (57.9)

Abbreviations: CI, confidence interval; NORS, National Outbreak Reporting System.

^aSingle-etiology outbreaks with norovirus reported as confirmed (laboratory confirmation of 2 or more cases, n = 2939).

^bSingle-etiology outbreaks with a confirmed nonviral etiology (n = 1573) (Supplementary Table 2).

^cKaplan criteria include vomiting ≥0.5 affected persons, median incubation period of 24–48 hours, and median duration of illness of 12–60 hours.

Performance of Kaplan Criteria and CART-Derived Characteristics

A CART model within 1 standard error of the best tree was developed with a complexity parameter of 0.002. Final clinical and epidemiologic characteristics selected from CART modeling and evaluated for other alternative splitting rules based on distinguishing confirmed norovirus from nonviral outbreaks included fever-to-vomit ratio <1, proportion of cases with bloody stools <0.1, and proportion of cases with vomiting ≥0.26 (Figure 1). Alternative cutoff values for the proportion with bloody stools and the proportion with vomiting as splitting variables in other branches of the model were assessed (Supplementary Table 3). Diagnostic performance of CART-derived characteristics was assessed among confirmed norovirus and nonviral outbreaks and compared with application of the Kaplan criteria (Table 3). The Kaplan criteria were 63.9% sensitive (95% CI, 54.5%–72.3%) and 100% specific (95% CI, 83.2%–100%); the likelihood ratio was undefined due to 100% specificity. However, only 108 (3.7%) confirmed norovirus outbreaks and 19 (1.2%) nonviral outbreaks had complete information for the Kaplan criteria. The CART-derived characteristics were 85.7% sensitive (95% CI, 82.9%–88.1%), 92.4% specific (95% CI, 90.0%–94.3%), and had a likelihood ratio of 11.3. Moreover, 706 (24.9%) confirmed norovirus outbreaks and 605 (20.6%) confirmed nonviral outbreaks had complete data for these criteria, far exceeding the proportion of outbreaks with information on Kaplan criteria. The CART-derived characteristics had a high Cohen’s kappa statistic (0.78; 95% CI, 0.72–0.83), demonstrating substantial agreement of confirmed norovirus and nonviral outbreaks that fit the criteria [15]. In comparison, the Kaplan criteria had only fair agreement based on Cohen’s kappa statistic (0.34; 95% CI, 0.22–0.48). Additionally, confirmed norovirus outbreaks were less frequently misclassified by CART-derived characteristics (13.3%) compared with the Kaplan criteria (36.1%).

Figure 1.

Classification of outbreaks reported to the National Outbreak Reporting System (NORS) based on classification and regression tree (CART) model-derived characteristics. Characteristics derived from CART modeling distinguish confirmed norovirus outbreaks from confirmed nonviral outbreaks with NORS 2009–2012 data. Starting with 2939 confirmed norovirus and 1573 confirmed nonviral outbreaks, clinical and epidemiologic predictors were assessed. Of the outbreaks with all reported characteristics, 706 confirmed norovirus and 607 confirmed nonviral outbreaks were categorized. Each rectangular partition represents a node, with the number of outbreaks and the outbreak percentage by category (total norovirus and nonviral outbreaks) displayed. Text in boldface represents the category with the most common class of outbreaks in each node. The “fever-to-vomiting ratio” represents the proportion of cases with fever divided by the proportion of cases with vomiting. The “proportion with bloody stool” represents the proportion of cases with bloody stool. The “proportion with diarrhea” represents the proportion of cases with diarrhea. The “proportion with vomiting” represents the proportion of cases with vomiting.

Table 3.

Performance of Kaplan Criteria and CART-Derived Characteristics Among Outbreaks Reported Through the NORS, 2009–2012

Clinical and Epidemiologic Profiles	Confirmed Norovirus (%)a	Suspected Norovirus (%)b	Confirmed Nonviral (%)c	Unknown (%)d	Cohen’s Kappa (95% CI)g	Likelihood Ratiog	Sensitivity (95% CI), %g	Specificity (95% CI), %g	Positive Predictive Value (95% CI), %g	Negative Predictive Value (95% CI), %g
Kaplan et al.e
Outbreaks with all criteria	108 (3.7)	29 (2.2)	19 (1.2)	121 (3.3)
Outbreaks that fit the criteria	69 (63.9)	12 (41.4)	0 (0.0)	35 (28.9)	0.34 (0.22–0.48)	Undefined	63.9 (54.5–72.3)	100 (83.2–100)	100 (94.7–100)	32.8 (22.1–45.6)
Outbreaks that did not fit the criteria	39 (36.1)	17 (58.6)	19 (100.0)	86 (71.1)
CART-derived characteristicsf
Outbreaks with all criteria	706 (24.0)	324 (24.5)	607 (38.6)	762 (19.2)
Outbreaks that fit the criteria	605 (85.7)	261 (80.6)	46 (7.6)	536 (70.3)	0.78 (0.72–0.83)	11.3	85.7 (82.9–88.1)	92.4 (90.0–94.3)	92.9 (90.7–94.7)	84.7 (81.8–87.3)
Outbreaks that did not fit the criteria	101 (14.3)	63 (19.4)	561 (92.4)	226 (29.7)

Abbreviations: CART, classification and regression tree; CI, confidence interval; NORS, National Outbreak Reporting System.

^aSingle-etiology outbreaks with norovirus reported as confirmed (laboratory confirmation of 2 or more cases, n = 2939).

^bSingle-etiology outbreaks with norovirus reported as suspected (laboratory confirmation in fewer than 2 cases, n = 1321).

^cSingle-etiology outbreaks with a confirmed nonviral etiology (n = 1573) (Supplementary Table 2).

^dOutbreaks reported without any suspected or confirmed etiology (n = 3694).

^eKaplan criteria includes vomiting ≥0.5 affected persons, median incubation period of 24–48 hours, and median duration of illness of 12–60 hours.

^fCART-characteristics include fever-to-vomiting ratio <1, proportion with bloody stools <0.1, and proportion with vomiting ≥0.26.

^gValues determined by comparing confirmed norovirus with confirmed nonviral outbreaks.

Lastly, the Kaplan criteria and CART-derived characteristics were applied to suspected norovirus and unknown etiology outbreaks to determine the proportion of outbreaks that each would attribute to norovirus (Table 3). Among suspected norovirus outbreaks, 324 (24.5%) had complete data for the CART-derived characteristics whereas only 2.2% had complete information for the Kaplan criteria. Among the suspected norovirus outbreaks with complete information provided, 261 (80.6%) met the CART-derived characteristics, similar to the proportion among confirmed norovirus outbreaks (85.7%); the proportion of suspected norovirus outbreaks with complete information that met the Kaplan criteria was substantially lower (41.4%). Among outbreaks of unknown etiology, 762 (19.2%) had complete data for the CART-derived characteristics while only 121 (3.3%) had complete data for the Kaplan criteria. Among these unknown etiology outbreaks with complete information provided, 536 (70.3%) fit the CART-derived characteristics, but only 35 (28.9%) fit the Kaplan criteria.

DISCUSSION

Analysis of AGE outbreaks reported through the NORS reveal distinct clinical and epidemiologic characteristics that can help readily identify norovirus in the absence of laboratory testing. The utility of such characteristics in public health practice depends on both diagnostic performance and the frequency with which data are available during outbreak investigations. We found that the Kaplan criteria were moderately sensitive and highly specific in distinguishing confirmed norovirus from confirmed nonviral outbreaks, but could not be applied to a majority of outbreaks due to lack of reported data. In comparison, CART-derived characteristics were far more sensitive and only slightly less specific compared with the Kaplan criteria in distinguishing confirmed norovirus outbreaks from confirmed nonviral outbreaks. Furthermore, CART-derived characteristics were reported in a greater number of outbreaks than the Kaplan criteria. Application of CART-derived characteristics to unknown etiology outbreaks reported in the NORS suggests that the majority of unknown etiology outbreaks may in fact be attributable to norovirus. These findings suggest that CART-derived characteristics may be a useful alternative to the Kaplan criteria for distinguishing norovirus from other nonviral etiologies during outbreak investigations.

Among outbreaks reported in the NORS, the Kaplan criteria were moderately sensitive (63.9%) and highly specific (100%), but were rarely reported (2.9%). These findings were consistent with those from a similar evaluation by Turcios et al., in which they found that the Kaplan criteria were highly specific (98.6%) and moderately sensitive (68.2%) [10]. For public health practice, these criteria may be unnecessarily too specific, at the expense of being insensitive, as they are often unknown or unavailable during outbreak investigations. Accurate exposure information can be difficult to determine for AGE outbreaks, and incubation periods are often short, making it difficult to distinguish between primary and secondary cases, particularly in nonpoint source outbreaks [16].

CART-derived characteristics identified in this study, including the fever-to-vomit ratio <1, proportion of cases with bloody stool <0.1, and proportion of cases with vomiting ≥0.26, performed statistically better than the Kaplan criteria in distinguishing confirmed norovirus outbreaks from confirmed nonviral outbreaks. The CART-derived characteristics had high sensitivity (85.7%) while still maintaining high specificity (92.4%). Improved sensitivity could be attributed to the lower cutoff value of 0.26 for the proportion of cases with vomiting in an outbreak. This cutoff, which is lower than Kaplan’s proposed ≥50% of cases with vomiting [17], may be the result of the criterion being used in conjunction with fever-to-vomiting ratio <1. Additionally, compared with the Kaplan criteria, CART-derived characteristics were reported in more than 8 times as many outbreaks. Higher reporting rates of CART-derived characteristics were at least partly due to training the CART model with predictors more frequently reported in clinical and epidemiologic characteristics. Previous studies have illustrated bias in variable importance measures where potential predictors differ [18, 19]. Overall, the CART-derived characteristics were effective in distinguishing more confirmed norovirus outbreaks from confirmed nonviral outbreaks within the NORS compared with the Kaplan criteria.

Applying clinical and epidemiologic characteristics identified by CART modeling to outbreaks reported in the NORS, 80.6% of suspected norovirus outbreaks and 70.3% of unknown etiology outbreaks with all reported characteristics would be attributed to norovirus. For suspected norovirus outbreaks and unknown etiology outbreaks, it is possible that misclassification by CART-derived characteristics could occur for a variety of reasons, including a small proportion of reported outbreaks that could have other viral etiologies or toxin-mediated events that exhibit similar clinical and epidemiologic characteristics as norovirus. Viral pathogens are the most common cause of gastroenteritis in industrialized countries [20–22], and other viral etiologies including sapovirus, rotavirus, astrovirus, and enteric adenovirus can exhibit similar clinical and epidemiologic characteristics as norovirus [16, 23–26]. Additionally, the CART-derived characteristics have the potential for some false positives (92.4% specificity) and some false negatives (85.7% sensitivity) among norovirus outbreaks with outlying characteristics. This misclassification was observed with 14.3% of confirmed norovirus outbreaks that do not fit the CART-derived characteristics. Incomplete reporting of clinical and epidemiologic characteristics could potentially bias the performance of these predictors if the reported proportion of cases with symptoms does not fully represent the total number of cases in an outbreak. Moreover, these characteristics do not account for mode of transmission, which investigators would consider to rule out toxin-mediated events when identifying an outbreak etiology. These potential biases and misclassifications notwithstanding, clinical and epidemiologic characteristics identified through CART modeling can still serve as a useful public health and analytic tool in identifying and better estimating the frequency of norovirus etiology among AGE outbreaks.

This study was subject to some limitations; perhaps most notable was incomplete reporting of clinical and epidemiologic characteristics of interest in the NORS. This limitation represents the real-world challenges of outbreak investigation and reporting in the context of limited resources at state and local health departments [27]. However, this limitation provided an opportunity to assess the reporting rate of various clinical and epidemiologic criteria as an indication of utility. Characteristics with good diagnostic performance but not often reported are generally of limited public health utility. Although incubation period was not part of the profile due to infrequency of reporting, when available, this criterion should still be considered for distinguishing norovirus from toxin-mediated events. We were also unable to directly compare the Kaplan criteria and CART-derived characteristics by likelihood ratios due to the 100% specificity of the Kaplan criteria. Lastly, due to the small number of other viral outbreaks and incomplete reporting of clinical and epidemiologic characteristics, we were unable to differentiate norovirus from other viral etiology outbreaks and had to exclude them from analysis.

In conclusion, clinical and epidemiologic characteristics identified through CART modeling were the most effective clinical and epidemiologic profile to differentiate the largest proportion of confirmed norovirus from confirmed nonviral outbreaks reported in the NORS from 2009 to 2012. Although the Kaplan criteria remain highly specific for identifying norovirus among NORS-reported outbreaks, a majority of outbreaks lacked complete information to make a diagnosis based on those criteria alone. While availability of clinical diagnostic tests for norovirus is increasing, clinical and epidemiologic criteria will undoubtedly have continued utility for public health practitioners to readily use during outbreak investigations. With ongoing improvements in surveillance and increased reporting of outbreaks, such as the Norovirus Sentinel Testing and Tracking (NoroSTAT) network [28], clinical and epidemiologic characteristics identified by CART modeling herein can aid in the rapid diagnosis of norovirus in outbreak investigations and lead to more targeted implementation of control measures.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.