Diagnostic accuracy between readers for identifying electrographic seizures in critically ill adults

Summary

Objective

Electrographic seizures in critically ill patients are often equivocal. In this study, we sought to determine the diagnostic accuracy of electrographic seizure annotation in adult intensive care units (ICUs) and to identify affecting factors.

Methods

To investigate diagnostic accuracy, interreader agreement (IRA) measures were derived from 5,769 unequivocal and 6,263 equivocal seizure annotations by five experienced electroencephalogram (EEG) readers after reviewing 74 days of EEGs from 50 adult ICU patients. Factors including seizure equivocality (unequivocal vs. equivocal) and laterality (generalized, partial, or bilaterally independent), cyclicity (cyclic vs. noncyclic), persistency (occurrence of status epilepticus), and patient consciousness level (coma vs. noncoma) were further investigated for their influence on IRA measures.

Results

On average, 70% of seizures marked by a reference reader overlapped, at least in part, with those marked by a test reader (any‐overlap sensitivity, AO‐Sn). Agreed seizure duration between reader pairs (overlap‐integral sensitivity, OI‐Sn) was 62%, while agreed nonseizure duration (overlap‐integral specificity, OI‐Sp) was 99%. A test reader would annotate one additional seizure not overlapping with a reference reader's annotation in every 11.7 h of EEG, that is, the false‐positive rate (FPR) was 0.0854/h. Classifying seizure patterns into unequivocal and equivocal improved specificity and FPR (unequivocal patterns) but compromised sensitivity only for equivocal patterns. Sensitivity of all and unequivocal annotations was higher for patients with status epilepticus. Specificity was higher for partial than for bilaterally independent unequivocal seizure patterns, and lower for cyclic all seizure patterns.

Significance

Diagnosing electrographic seizures in critically ill adults is highly specific and moderately sensitive. Improved criteria for diagnosing electrographic seizures in the ICU are needed.

Key Points.

• Seizure identification in ICU EEGs by human experts was highly specific but moderately sensitive

• High incidence of equivocal seizure patterns was a major factor compromising seizure diagnostic sensitivity

• Seizure location and cyclicity affected specificity without compromising sensitivity

• Sensitivity for status epilepticus was higher than for other seizures

• Improved criteria for electrographic seizures, especially equivocal seizure patterns, may improve diagnostic accuracy of ICU seizures

Immediate identification of electrographic seizures is important for early diagnosis, appropriate management, and accurate prognosis estimation in patients with all types of brain injuries,1, 2 especially in critically ill patients under intensive care.3, 4, 5, 6 However, interpretation of electrographic seizures can be difficult for even the most experienced experts7, 8, 9 and can be even more challenging during continuous electroencephalogram (EEG) monitoring in intensive care units (ICUs) because of added complexity and uncertainty.10, 11, 12 So far, the extent to which difficult‐to‐interpret seizure‐like patterns, often known as equivocal seizure patterns13, 14 or interictal‐ictal continuum,15 can compromise diagnostic accuracy and interreader agreement (IRA) in seizure detection in ICUs is still unknown.

Furthermore, it is not fully understood why seizures in ICUs are more difficult to interpret. Based on our experience, we postulate that some common features of ICU seizures, including higher incidence of equivocal seizure patterns,10, 11, 13, 16 cyclic seizures,17 status epilepticus10, 18, 19 and coma,4, 20, 21 may affect diagnostic accuracy of electrographic seizures in ICUs.

Here we investigated IRA between experts in annotating electrographic seizures in adult ICUs and its influencing factors to determine, and potentially improve, diagnostic accuracy in detecting electrographic seizures in ICUs.

Materials and Methods

Patients and EEGs

After approval from Columbia University's institutional review board, EEG reports dated between September 2009 and May 2012 at Columbia University Comprehensive Epilepsy Center were screened for patients under critical care with electrographic seizures, including equivocal seizures patterns, which refer to those described as “probable seizures,” “possible seizures,” “cannot exclude seizures,” or “interictal‐ictal continuum.” Available EEGs corresponding to the found reports were retrieved from an EEG database. Only EEGs longer than 1.5 h were included. For patients with more than 48 h of EEG, only the earliest 48 h of recording containing seizure markings were included. The screening found 50 patients, including 21 patients with equivocal seizures in EEG reports. The duration of EEGs studied for each patient ranged from 1.5 to 53.0 h (median 40.5, interquartile range 25.3–48.0). EEGs for all patients totaled 74 days (Table 1).

Table 1.

Summary of seizure annotations by the readers

Reader	N		Median (IQR) duration		Total duration
	Uneq.	Eq.	Uneq.	Eq.	Uneq.	Eq.
Reader 1	820	2,009	01:16 (00:41…02:06)	00:22 (00:15…01:08)	40:50:41	73:28:05
Reader 2	1,604	508	00:48 (00:20…01:41)	00:30 (00:16…01:35)	60:10:33	14:44:05
Reader 3	1,093	1,053	01:27 (00:41…02:19)	00:44 (00:21…01:27)	118:36:55	57:31:32
Reader 4	767	2,079	01:36 (00:40…04:05)	01:29 (00:38…03:43)	378:13:33	465:31:56
Reader 5	1,479	614	00:58 (00:23…02:00)	00:39 (00:14…01:54)	47:51:26	22:35:24

Eq., annotations marked as equivocal seizures; IQR, interquartile range; N, number of annotations; Uneq., annotations marked as unequivocal seizures.

Based on the EEG reports, patients were categorized according to seizure laterality (generalized, partial, bilaterally independent), cyclicity (cyclic vs. noncyclic), and occurrence of status epilepticus. Clinical records were reviewed to obtain Glasgow Coma Scores (GCS) rated by nurses or doctors at the time of the first seizure in EEG reports. Patients with GCS ≤ 8 were categorized as comatose.

EEG acquisition was similar to previously described.22 Ag/AgCl electrodes were placed on scalp based on international 10–20 system with collodion USP (Mavidon Medical Products, Riviera Beach, FL, U.S.A.), further secured by wrapping the entire head with gauze when possible. The ground electrode was placed between the left or right central and parietal electrodes. Impedances were kept below 5 kΩ. EEGs were recorded at a sampling rate of 256/s using Xltek hardware (Model EMU40 amplifiers) and reviewed with Neuroworks software (version 6.1.0 build 892; Natus Medical Inc., San Carlos, CA, U.S.A.).

Seizure annotation

EEG files were deidentified: existing annotations and videos were removed before the EEGs were read by five electrophysiologists. Three of the readers had fellowship training at Columbia University and 2–6 years of experience in EEG reviewing at the time of this study. The other two readers were trained elsewhere, with 4 and over 30 years of EEG reading experience, respectively. Each seizure was annotated with an onset and an offset and was classified as either unequivocal or equivocal on the basis of criteria derived from the 2009 and 2012 American Clinical Neurophysiology Society (ACNS) standardized critical care EEG terminology.23, 24 Unequivocal seizure patterns were defined as generalized spike‐wave discharges occurring at ≥3/s (with or without evolution) or evolving discharges of any type reaching a frequency >4/s, whether focal or generalized. Equivocal seizure patterns were suspicious for seizures on the basis of the EEG reader's experience but did not meet the above criteria. The readers were also advised not to mark any seizure shorter than 10 s to comply with published criteria.13, 15, 25, 26 Examples of an unequivocal and an equivocal seizure are shown in Figs. S1 and S2.

Diagnostic accuracy calculation

Diagnostic accuracy was based on IRA measures, which were calculated using annotations from each of the readers as a gold standard (i.e., reference). Annotations from each of the readers (test reader) were compared against those of the reference reader. Thus, 5 × 4 reader pairs were established, which unbiasedly gave each reader an equal chance of being the gold standard. For each reader pair, diagnostic accuracy was calculated separately for each of the 50 patients, generating 5 × 4 × 50 matrices for each IRA measure, as illustrated in Fig. 1. Each IRA measure was derived separately for all annotations (union of unequivocal and equivocal annotations), for unequivocal annotations, and for equivocal annotations.

Figure 1.

OI‐Sn versus OI‐Sp scattergrams between each reader pair. Data shown are 5 × 4 × 50 matrices used in the statistical analysis for this study. Each data point represents OI‐Sn (y coordinate) and OI‐Sp (x coordinate) of each patient (n = 50). Each row represents OI‐Sn and OI‐Sp value pairs of a test reader (R1–R5 in row labels) against each reference reader (R1–R5 in column labels).

Four IRA measures, that is, any‐overlap sensitivity (AO‐Sn), overlap‐integral sensitivity (OI‐Sn), overlap‐integral specificity (OI‐Sp), and false‐positive rate (FPR), were calculated as previously described27 and explained below:

1. AO‐Sn: Sensitivity = [True positives]/([True positives] + [False negatives]) ×100%. True positives: number of annotations that overlap in time for both readers. False negatives: number of annotations by a reference reader that do not overlap with any test reader annotation. The sum of true positives and false negatives is also the total number of seizures annotated by the reference reader.

2. FPR: False‐positive rate = [Number of false positives in a record]/[Length in hours]. Number of false positives in a record: number of test reader annotations that do not overlap in time with any reference reader annotation.

3. Overlap integral sensitivity (OI‐Sn) and specificity (OI‐Sp) were calculated by comparing binary (seizure or nonseizure) second‐by‐second time series between readers. In contrast to AO‐Sn and FPR, overlap‐integral measures take into account possible time lags between onset and offset marked by two readers, and true and false positives or negatives are the number of seconds instead of the number of seizures.

4. Specificity = [True negatives]/([True negatives] + [False positives]) ×100%. True negatives: the number of seconds both readers agree upon with no seizures. False positives: the number of seconds within the test reader's annotation, but outside the reference reader's annotation. The sum of true negatives and false positives is the total number of seconds the reference reader considered as no seizures.

For example, in an 8‐h record, the reference reader marked a 2‐h‐long (120‐min‐long) seizure, while the test reader marked only a 2‐min‐long seizure with total overlap with the reference reader seizure. In this example, the AO‐Sn is 100%, FPR will be 0/h, OI‐Sn is 0.02%, and OI‐Sp is 100%. When switching between reference reader and test reader, AO‐Sn and FPR remain the same, while OI‐Sn becomes 100% and OI‐Sp becomes 75%.

Briefly, AO‐Sn is the percentage of seizures annotated by a reference reader that overlaps, at least in part, with seizures annotated by a test reader; OI‐Sn is the percentage of seizure duration annotated by a reference reader that was agreed upon by a test reader; OI‐Sp is the percentage of overlapped nonseizure EEG duration in total nonseizure EEG duration of a reference reader. FPR is the number of seizures per hour annotated by a test reader that do not overlap with any annotation by a reference reader.

Statistical analysis

First, repeated measures analysis of variance (RANOVA) was conducted separately for each IRA measure to study variation over different annotation types using reader‐to‐reader pair as within‐model parameter. Pearson correlation coefficient analysis was used to exclude possible correlations between reader pair IRA measures. To satisfy normality criterion, normal probability plots of ANOVA residuals were monitored. Heuristic approach was used to add 0.01 to FPR value before a log‐transformation. For OI‐Sp a nonlinear transformation 100/(101 − [OI‐Sp]) was applied. Because of unequal variances, Greenhouse‐Geisser correction28 was used to derive RANOVA results. Bonferroni method was used for post hoc comparison.

Second, multiple factor RANOVA analysis with Greenhouse‐Geisser correction was conducted using only data within each annotation type (all, unequivocal, and equivocal). Seizure lateralization, seizure cyclicity, occurrence of status epilepticus, and coma were used as variables and reader‐to‐reader pairs as a within‐model parameter. Bonferroni method was used for post hoc comparison.

All statistical analyses were done using Matlab and Matlab Statistics Toolbox (ver. 2014b). All differences mentioned in this paper were statistically significant (p < 0.05).

Results

The studied patients consisted of 20 men and 30 women, aged 22.4–97.7 years old (60.9 ± 19.6, representing mean ± SD), admitted to neurological (n = 37), medical (n = 5), surgical (n = 3), cardiothoracic (n = 3), or cardiac (n = 2) ICUs following suspected seizures or status epilepticus (n = 27), subdural hematoma (n = 10), cardiac arrest (n = 6), subarachnoid hemorrhage (n = 5), intracerebral hemorrhage (n = 5), stroke (n = 3), or respiratory (n = 3), hepatic (n = 1), or renal (n = 1) failures. EEGs were ordered for suspected epileptic etiology related to altered mental status (n = 12), seizure‐like movements (n = 10), or both (n = 28). Altered mental status included persistent (n = 36) or episodic (n = 4) unresponsiveness, confusion, and/or aphasia. Seizure‐like movements included eye deviation or gaze (n = 9), staring (n = 1), eye twitching (n = 2), lip twitching (n = 3), face twitching (n = 7), limb (n = 11) or body (n = 5) twitching, shaking or other repetitive movements, and generalized tonic‐clonic movements (n = 9).

All seizures

In this paper, all seizures refer to union of unequivocal and equivocal seizure patterns. Each reader annotated 2,093–2,852 seizures (Table 1). The number of patients annotated with one or more seizures by all (5/5) and majority (≥3/5) of the readers was 44 (88%) and 49 (98%), respectively (Table 2). Figure 1 illustrates variation of OI‐Sp and OI‐Sn for all patients and reader pairs. The mean AO‐Sn for all seizures was 70.21% (95% confidence interval [CI]: 65.05–75.37%); OI‐Sp, 99.21% (99.01–99.38%); OI‐Sn, 62.26% (56.94–67.58%); and FPR, 0.0854/h (0.0597–0.1205/h) (Fig. 2).

Table 2.

Patient‐wise agreement among readers

Annotation scenario		Number of agreed readers (%)					Total number of cases
		Majority (≥3/5)			Minority (≤2/5)
		5	4	3	2	1
All seizures		44 (88)	4 (8)	1 (2)	0 (0)	1 (2)	50
Equivocality	Uneq.	19 (39)	9 (18)	7 (14)	10 (20)	4 (8)	49
	Eq.	14 (30)	18 (38)	9 (19)	4 (9)	2 (4)	47
	Both	6 (14)	8 (19)	6 (14)	13 (31)	9 (21)	42
Laterality	Gen	12 (86)	2 (14)	0 (0)	0 (0)	0 (0)	14
	P	28 (88)	2 (6)	1 (3)	0 (0)	1 (3)	32
	BI	4 (100)	0 (0)	0 (0)	0 (0)	0 (0)	4
Cyclic?	Y	10 (91)	1 (9)	0 (0)	0 (0)	0 (0)	11
	N	34 (87)	3 (8)	1 (3)	0 (0)	1 (3)	39
SE?	Y	18 (90)	2 (10)	0 (0)	0 (0)	0 (0)	20
	N	26 (87)	2 (7)	1 (3)	0 (0)	1 (3)	30
Coma?	Y	27 (87)	3 (10)	1 (3)	0 (0)	0 (0)	31
	N	17 (89)	1 (5)	0 (0)	0 (0)	1 (5)	19

For all patients, data represent numbers (percentages) of cases annotated by specified number of readers with either unequivocal or equivocal pattern (All seizures), with only unequivocal patterns (Uneq.), with only equivocal patterns (Eq.), and with both unequivocal and equivocal pattern (Both). For seizure laterality, including generalized (Gen), partial (P), and bilaterally independent (BI) seizures, data represent numbers (percentages) of cases annotated with either unequivocal or equivocal seizure patterns. Same for whether cyclic pattern, whether status epilepticus (SE), or whether patients in coma (Y for yes, N for no).

Figure 2.

Diagnostic accuracy of ICU seizure detection was highly specific (OI‐Sn of 99% and FPR of 0.085/h) but moderately sensitive (AO‐Sn of 70%, OI‐Sn of 62%), with higher specificity for both unequivocal and equivocal patterns and lower sensitivity for equivocal patterns. The sensitivity‐FPR plot for any‐overlap measures (A) and sensitivity‐specificity plot for overlap‐integral measures (B) demonstrate that, compared to annotation of all seizures (All, black data points), classification of seizure patterns as unequivocal (Uneq, red data points) or equivocal ones (Eq, pink data points) based on ACNS guidelines improved specificity and FPR for unequivocal patterns and lowered sensitivity (both AO‐Sn and OI‐Sn), but improved specificity, for equivocal patterns. (Values are means, with error bars representing 95% confidence intervals.)

Unequivocal and equivocal seizure patterns

The readers unanimously annotated unequivocal, equivocal, or both types of seizure patterns in 19 (39%), 14 (30%), and 6 (14%) patients, respectively (Table 2).

For unequivocal seizure patterns, the mean (95% CI) AO‐Sn was 75.50% (67.84–83.17%); FPR 0.0345/h (0.0225–0.0509/h); OI‐Sn 67.62% (59.71–75.53%); and OI‐Sp 99.55% (99.42–99.66%). For equivocal seizure patterns, the mean AO‐Sn was 34.58% (25.65–43.51%); FPR 0.0682/h (0.0472–0.0970/h); OI‐Sn 29.11% (19.90–38.33%); and OI‐Sp 99.54% (99.40–99.65%). This classification of seizure patterns as unequivocal and equivocal ones led to higher OI‐Sp for both patterns and lower FPR for unequivocal patterns at the cost of lower AO‐Sn and OI‐Sn for equivocal patterns (Fig. 2). Compared to unequivocal patterns, equivocal patterns were nearly equal in OI‐Sp but were lower in AO‐Sn and OI‐Sn and higher in FPR.

The EEG reports described clinical correlation of seizures in 14 patients. In one of the patients, six equivocal patterns were annotated by only one reader, and the EEG was characterized by ambiguously evolving patterns obscured by prominent artifacts (Fig. 3). Interestingly, the EEG report was written by another reader who did not annotate any seizure pattern in this patient.

Figure 3.

EEG of an exemplary equivocal seizure disagreed upon by four of the five readers. Only the left parasagittal chain in bipolar longitudinal montage was shown because other regions did not show suspicious patterns. The onset and offset (red vertical lines) were marked by one reader and stated in the EEG report. The evolving pattern in the left parasagittal region (F3 C3 P3 O1 electrode) was composed of a blunt, <3‐Hz, minimally evolving pattern, with high‐frequency artifacts at C3, where a burr hole for evacuation of a subdural hematoma was located. This pattern gradually became shorter and disappeared within 3 h after intravenous administration of 500 mg phenytoin.

Generalized, partial, or bilaterally independent seizures

In patients with EEG reports describing generalized, partial, or bilaterally independent seizures, the readers unanimously annotated seizure patterns in 86% (12/14), 88% (28/32), and 100% (4/4) of patients, respectively (Table 2). We found higher OI‐Sp for unequivocal seizure patterns in partial seizures compared to bilaterally independent seizures, but not compared to generalized seizure patterns (Fig. 4A).

Figure 4.

Diagnostic accuracy was affected by seizure laterality, status epilepticus, and seizure cyclicity. (A) For laterality of unequivocal seizure patterns (Uneq), annotation of bilaterally independent seizure patterns (BI) were lower in OI‐Sp than those of partial seizure patterns (P), but not for generalized seizure patterns (Gen). (B) For patients with status epilepticus (SE, green horizontal hyphens), seizure annotations had higher AO‐Sn and OI‐Sn for all seizure patterns (All) and unequivocal patterns (Uneq). (C) For cyclicity, annotation of EEGs containing cyclic seizures (cyclic, green horizontal hyphens) was higher in FPR for all and unequivocal patterns than for those without cyclic seizures (noncyclic, brown horizontal hyphens) for all annotation types. (D) Cyclicity also lowered OI‐Sp for all seizure patterns. (Data are displayed as mean [horizontal hyphens], with error bars representing 95% confidence intervals; *p < 0.05, Bonferroni correction.)

Status epilepticus

Readers unanimously annotated seizure patterns in 90% (18/20) of patients with status epilepticus described in EEG reports and in 87% (26/30) of those without (Table 2). Annotations of seizures in EEGs with status epilepticus were more sensitive (AO‐Sn and OI‐Sn) for all and unequivocal patterns, but not for equivocal patterns (Fig. 4B).

Cyclic seizures

In patients with cyclic seizures (n = 11), readers marked more seizures (51, 32–66; representing median, interquartile range) at shorter intervals (7.3, 6.0–11.5 min) than in patients without cyclic seizures (10, 8–16 seizures per patient at intervals of 63, 13.7–69.6 min). Readers unanimously annotated seizures in 91% of patients with cyclic seizures, and in 87% of patients without (Table 2). Annotations in patients with cyclic seizures had higher FPR for all and equivocal annotations (Fig. 4C) and lower OI‐Sp only for all seizure annotations (Fig. 4D).

Coma

Readers unanimously annotated seizure patterns in 87% (27/31) of comatose patients and in 89% (17/19) of noncomatose patients (Table 2).

Discussion and Conclusion

We demonstrated that detection of ICU seizures by human experts was highly specific but moderately sensitive. Classifying seizure patterns as unequivocal or equivocal based on ACNS guidelines improved specificity and lowered FPR only for unequivocal patterns. But as a trade‐off, classification compromised sensitivity for equivocal patterns. We also found that occurrence of status epilepticus increased sensitivity of all and unequivocal annotations, and that seizure laterality and cyclicity only affected certain aspects of specificity.

Equivocal seizure patterns compromised sensitivity of seizure annotation

Among the factors we studied, only equivocality of seizures and occurrence of status epilepticus affected sensitivity, with the former a compromising factor (Fig. 2) and the latter an improving factor (Fig. 4B). Equivocal seizure patterns were common in this study, annotated in 28% of patients unanimously and in 82% patients by majority readers (Table 2), similar to previous reports.13, 14 Because high sensitivity helps to rule out (or exclude) a disease,29 the effectiveness of EEG monitoring to exclude seizures may be lowered as a result of occurrence of equivocal patterns. Equivocal patterns are associated with poor outcome in ICUs;30 they are important and should be taken into consideration when determining diagnostic accuracy of seizures in ICUs. A previous study reported a higher sensitivity but comparable specificity in patients known to have epilepsy,27 likely due to less frequent equivocal seizure patterns in these patients.

Importance of annotating onsets and offsets of seizures in ICU

Because continuous EEG monitoring in ICU is a diagnostic tool for monitoring time, frequency, and trends of seizure occurrence,3 its diagnostic accuracy should include how often seizures are falsely reported (FPR) and how accurately seizure onsets and offsets are annotated (overlap‐integral measures, i.e., OI‐Sn and OI‐Sp). We found that OI‐Sn was nearly 10 percentage units lower than AO‐Sn (Fig. 2), indicating that differences existed in annotation of onsets and offsets when readers’ annotations overlapped. The above information cannot be obtained through kappa statistics, which was often used in many prior studies to demonstrate IRA on preselected short (often < 1 h) EEG segments containing mainly confirmed seizures without accounting for IRA on nonseizure periods.7, 8, 21 We focused on longer (>1.5 h) original EEGs, containing mostly (median of 94%, calculated from Table 1) nonseizure duration. Because the median seizure length in this study was approximately 1 min (20 s to 1.5 min, depending on reader; Table 2), EEGs longer than 1.5 h would mostly contain over 90% of nonseizure duration, giving readers enough nonseizure EEGs for evaluation of diagnostic accuracy. EEGs beyond 48 h were excluded to match the typical EEG monitoring duration suggested in ICU for seizure detecting.14 Because EEG reports in this study did not report new types of seizures in EEGs beyond 48 h, including longer EEGs with additional seizures similar to those annotated earlier would only have unnecessarily increased the workload of readers.

Expert reader's annotations were the best available gold standard

Gold standard of this study was the seizure annotation based on the electrographic features solely. To avoid bias, calculation of mean sensitivity, specificity, and FPR was based on all reader‐to‐reader pairs so that each reader was in turn regarded as the gold standard. Because the gold standard was another reader's annotation, the diagnostic accuracy also represented IRA levels. We did not study seizures without scalp EEG changes that could alternatively be determined clinically or via intracranial recordings;31 therefore, our results should apply only to electrographic seizures with scalp EEG correlates.

Ways to improve diagnostic accuracy of seizures in ICU

Better seizure criteria

Classification of seizure patterns into unequivocal and equivocal ones according to ACNS criteria significantly increased OI‐Sp and reduced FPR by more than half for unequivocal patterns (Fig. 2). However, as a trade‐off, sensitivity was compromised for equivocal patterns, indicating that the ACNS criteria are less than ideal. Slightly improved OI‐Sp probably relates to increased time reference reader has marked as a negative (the time period of no‐seizure is always longer or equal to the period of no‐unequivocal pattern or no‐equivocal pattern), thus reducing the effect of false‐positive classification on specificity value. The disagreement in equivocality criteria may be related to the equivocal nature of some EEG patterns, such as brief, blunt, or poorly formed discharges, of which frequencies are hard to evaluate (Fig. 3)15, 26, 32 or posthypoxic high‐frequency generalized discharges without clear evolution.33 Practice in merging multiple seizure patterns into one single pattern also varied among readers, as evidenced by lower seizure number and longer total seizure durations annotated by reader 4 (Table 1). Our readers identified equivocal seizure patterns on the basis of their experience, which was subjective and might contribute to lower sensitivity for equivocal annotations. Because of the highly variable nature and lack of clear features for some seizure patterns, especially equivocal patterns, a publicly available seizure database or atlas, with further characterization and classification of seizure morphology and clinical significance, should optimize existing seizure criteria and improve diagnostic accuracy of electrographic seizures.

Standardize reader's EEG training

The lower specificity for one of the readers (reader 4 in Fig. 1) may be related to different EEG training of this reader, which is supported by prior reports where different readers’ training affected IRA.7, 34 Further research is needed to evaluate the benefit of available standardized reader's training in improving diagnostic accuracy of seizures in ICU.

Compensating limitations of ICU EEG reading

Inspecting clinical correlate

Although annotating ICU electrographic seizures was only moderately sensitive (Fig. 2), the accuracy of seizure diagnosis may be improved by inspecting videos for clinical correlate to clarify equivocal patterns.13, 14, 19, 35 This is consistent with our observation that clinical correlate influenced the seizure identification in a patient with equivocal seizure patterns (Fig. 3).

Obtaining the majority opinion

Because seizure occurrence may change patient management, the number (percentage) of patients with seizures agreed upon unanimously or by the majority of readers reflects a clinically meaningful estimate for the level of agreement. Using seizure occurrence in EEG reports as a reference, the majority of readers agreed on 98% of patients with any type of seizure, on all patients with cyclic seizures and status epilepticus, and on all comatose patients with seizures (Table 2). Our observation indicated that the decision of the majority of readers on whether a patient has seizures, based on EEG only, was close to clinical EEG report.

Intracranial EEG and quantitative EEG

For equivocal seizure patterns, seizure diagnosis may be easier if intracranial EEG (iEEG) is available for clarification.31 Quantitative EEG (QEEG) was also used in ICU seizure detection with excellent IRA when using annotations from human experts as standards.36, 37 Further research is needed to determine whether QEEG can help to clarify equivocal seizures.

In summary, annotating electrographic seizures in ICU patients was highly specific but moderately sensitive, with sensitivity compromised by the presence of equivocal seizure patterns. Improving seizure criteria and standardizing reader training may improve diagnostic accuracy of electrographic seizure detection in ICUs.

Disclosure of Conflict of Interest

Dr. Särkelä is a current employee of GE Healthcare. Ms. Eerikäinen is a former employee of GE Healthcare. Drs. Tu, Young, and Mayer have served as paid consultants of GE Healthcare. The remaining authors have no conflicts of interest. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Supporting information

Figure S1. An exemplary unequivocal seizure.

Figure S2. An exemplary equivocal seizure.

Biography

Bin Tu is Director of CEEG Operations in Columbia University Comprehensive Epilepsy Center.