Optical flow estimation improves automated seizure detection in neonatal EEG

Abstract

Introduction

Existing automated seizure detection algorithms report sensitivities between 43–77% and specificities between 56–90%. The algorithms suffer from false alarms when applied to neonatal EEG, due to the high degree of nurse handling and rhythmic patting employed to soothe neonates. Computer vision technology that quantifies movement in real time could distinguish artifactual motion and improve automated neonatal seizure detection algorithms.

Methods

We utilized video EEG recordings from 43 neonates undergoing monitoring for seizures as part of the NEOLEV2 clinical trial. The Persyst neonatal automated seizure detection algorithm ran in real-time during study EEG acquisitions. We applied computer vision algorithms to extract detailed accounts of artifactual movement of the neonate or people near the neonate though dense optical flow estimation.

Results

Using the methods mentioned above, we identified and quantified 197 periods of patting activity, of which 45 generated false positive automated seizure detection events. A binary patting detection algorithm was trained with a subset of 470 event videos. This supervised detection algorithm was applied to a testing subset of 187 event videos with eight false positive events, which resulted in a 24% reduction in false-positives automated seizure detections and a 50% reduction in false-positive events caused by neonatal care patting, while maintaining 11 of 12 true-positive seizure-detection events.

Conclusions

This work presents a novel approach to improving automated seizure detection algorithms used during neonatal video EEG monitoring. Our artifact detection mechanism can improve the ability of a seizure detector algorithm to distinguish between artifact and true seizure activity.

Introduction

Automated software detection of seizures is a budding technology. Despite recent advances in software algorithms (1–25), artifact and false positive events often limit their overall utility in a clinical setting (26, 27). One artifact-generating event in neonates is gentle rhythmic patting done by nursing during routine care, such as diaper changing and burping. Neonatal EEGs are very susceptible to this type of artifact due to their frequent need for nursing care. Due to the rhythmicity of the patting, these events can often mimic seizure on EEG and cause false alarms by seizure detection software. One potential solution is to employ computer vision to detect rhythmic patting using the recorded video during continuous EEG monitoring. This paper presents a novel method of patting detection with optical flow imaging using a large dataset from the NEOLEV2 trial (NCT01720667). Optical flow provides the directionality and velocity of apparent movement of patterns between sequential images.

Materials and Methods

2.1. Video acquisition

We utilized video EEG recordings from neonates undergoing monitoring for seizures as part of the NEOLEV2 clinical trial. IRB approval was granted by the primary site (UC San Diego, San Diego, CA) and all other study sites. The Persyst Version 12 neonatal automated seizure detector ran in real time during study EEG acquisitions. Inclusion criteria for the NEOLEV2 study included: (1) newborns with or at risk for seizure, (2) term infants with gestational age 36–44 weeks and less than 2 weeks of age, and (3) greater than 2200 grams. Parental consent was obtained for all patients. Subjects were excluded if they had received any prior anticonvulsants, if serum creatinine was greater than 1.6 mg/dL, or if seizures were due to correctable metabolic abnormalities. Subjects in whom death was thought imminent were also excluded from the study. Subjects in whom video EEG monitoring could not be commenced prior to the need to treat definite clinical seizure activity were not recruited. NEOLEV2 inclusion criteria included recruitment of hypoxic ischemic encephalopathy (HIE) neonates. However, the studies utilized in this analysis did not include the HIE neonates. Standard neonatal anterior-posterior bipolar montages were utilized for first screening of the EEG. Because these EEGs were captured digitally, if necessary, other referential montages were utilized by the neurophysiologist for confirmation of events.

The video and EEG data were acquired by Cadwell systems (http://www.cadwell.com/). The video sequences were compressed with an MPEG-4 compression before they were stored. The resolution of the used video data was 320 × 240 pixels sampled at a rate of 30 frames per second. The camera was placed so that the neonate’s crib was in the center of the video. The data were labeled with the starting and ending time points of the patting epochs, automated seizure detection event times, and neurophysiologist-verified seizure event times. Each dataset included 24 hours of video with a varying number of labeled events. The quantified data from these videos was analyzed in conjunction with synchronized EEG annotated information, including automated seizure detections. Each annotated event was labeled as either (1) actual seizure event, (2) automated seizure detection event (ASDE), and/or (3) patting event. For each event, twenty seconds of video centered on each were extracted, converted to grayscale, and then time-smoothed by averaging over three frames of consecutive video. Since the image was centered to the neonate, we used radial coordinates with the center of the video as the origin, and we limited our detection algorithm to the inner 25% portion of the video.

2.2. Optical flow algorithm

For each video sequence, we applied computer vision algorithms to extract detailed accounts of neonate and surrounding movement behavior through dense optical flow estimation. We used the Farneback optical flow method (28) with three pyramidal layers each reducing the image by half in each layer, three search iterations per pyramid, a neighborhood pixel size of five, and a Gaussian filter size of 15.

Then, using polar coordinates with the center of the image as the origin, we obtained the summation of the radial-projecting component (V_in) of each Cartesian optical flow velocity vector (V_x, V_y). Shown in Eq 1, a negative V_in was defined as outward projecting, and positive V_in was defined as inward projecting. Thus, an overall negative V_in could represent a person’s hand with a velocity projecting away from the center of the image (or neonate in this scenario), and a positive V_in could represent an inward-projecting hand. Height and width were pixel size 320 and 240, respectively.

$$V_{in}\left(x,y\right)=\sqrt{V_x^2+V_y^2}\cos \left({\tan }^{-1}\left(V_y,V_x\right)-{\tan }^{-1}\left(height/2-y,width/2-x\right)\right)$$ Eq. 1

We next obtained the average spectral density of V_in surrounding each actual seizure detection event, automated seizure event, and patting event. A frequency-domain patting detection algorithm was then built using the spectral analysis of patting events. We used a grid search method to test video sample lengths from one to 20 seconds, fast Fourier transform (FFT) Welch filter lengths from one to 20 seconds, power spectral density (PSD) frequency target frequency ranges from zero to six hertz, and a threshold level from zero to 100 percent of the summated target PSD frequency range. Finally, using a testing set of videos, when the filtered optical flow was more than the PSD threshold level, the motion events were decided, and we showed that the optical flow visual learning algorithm could recognize patting events.

Results

3.1. Data Description

We analyzed 320×240 pixel 24 hour-length videos and annotated EEG files (n=43) in the neonatal intensive care unit at our hospitals. Using the methods mentioned above, we quantified and identified 197 periods of patting activity, 310 automated seizure detection events (ASDEs), and 99 actual seizure events across all 43 babies(Table 1). Of note, ASDEs were omitted if they were within 20 seconds of a prior ASDE. Concomitant seizure and ASDEs were identified if they occurred within 60 seconds of each other. Likewise, concomitant ASDE / patting events and seizure / patting events were identified if they occurred within 20 seconds of each other. With these parameters, we identified 45 concomitant patting and ASDEs, of which there were no associated actual seizures, and 152 individual patting events not associated with any ASDEs. Overall, the false detection rate of the automated seizure detector was 0.25 false positives per hour. Patients had an average and standard deviation of 4.6 ± 9.4 patting events, 7.3 ± 13.9 automated seizure detection events, and 2.3 ± 6.3 seizure events.

Table 1:

List of individual and concomitant seizure, seizure detection events, and patting events.

Event	Quantity
Seizure	99
Seizure Detection Event (ASDE)	310
Patting	197
Patting + Seizure	0
Patting + No Seizure	197
Patting + ASDE	45
Patting + ASDE + Seizure	0
Patting + ASDE + No Seizure	45
Patting + No ASDE	152
Patting + No ASDE + Seizure	0
Patting + No ASDE + No Seizure	152

Automated seizure detection event (ASDE)

3.2. Performance Metrics

A binary patting detection algorithm was trained with subset of random 470 event videos (140 patting events, 68 seizure events, 222 ASDEs, 40 random infant movement events). Optimal results were found with a sample size of four seconds, FFT Welch length of two seconds, and threshold level of the power spectral density (PSD) between one to six hertz (Fig 1). This supervised detection algorithm was applied to the remaining testing subset of 187 events, which resulted in 50% reduction in false positive automated seizure detection events caused by neonate care patting, while maintaining 11 of 12 true positive seizure events (Table 2, 3). These results were further confirmed with a 1000-fold validation run on randomized event subsets, which revealed a (mean ± standard deviation) 20.4% ± 4.0 decrease in false positive seizure detection events, 57.5% ± 12.9 decrease in false positive seizure detection events due to patting, 2.7% ± 4.1 decrease in true positive seizure detection events, 4.9% ± 3.9 decrease in sensitivity, and a 18.8 % ± 6.6 increase in positive predictive value with our algorithm.

Figure 1:

A) Optical flow is performed on the inner 25% of the video. After patting commences, a repetitive ~3Hz radial velocity V_in patting signal is observed. B) Power spectral density of patting behavior versus other types of behavior. Patting and nursing manipulation generates a large amount of low-frequency 0–6 Hz optical flow with a peak around 3–4Hz. Other activities including random behavior, automated seizure detection events (ASDEs), and seizures generate low-frequency spectral optical flow, but at a lower amplitude that patting events. C) Power spectral density of patting behavior versus other types of behavior (shaded region represents 95% confidence interval in B and C). D) Overall signal root mean square power of the V_in time-series of patting behavior versus other types of behavior. In general, patting events generate larger amplitude of optical flow as compared to other events.

Table 2:

Description of events used in training and testing sets.

	Training Set (n=470)	Testing Set (n=187)
Seizure	68	31
ASDE	222	88
Patting	140	57
Random Infant Movement	40	11

Automated seizure detection event (ASDE)

Table 3:

Automated seizure detection algorithm performance metrics of training and testing sets.

	Training Set (n=222 ASDEs)			Testing Set (n=88 ASDEs)
	Without Algorithm (n,%)	With Algorithm (n,%)	% Change	Without Algorithm (n,%)	With Algorithm (n,%)	% Change
False Positive	190 (85.6)	153 (68.9)	19%	76 (86.4)	58 (65.9)	24%
False Positive (due to patting)	33 (14.9)	13 (5.9)	61%	8 (9.1)	4 (4.5)	50%
True Positive	32 (14.4)	32 (14.4)	0%	12 (13.6)	11 (12.5)	8%
False Negative	34	37	9%	9	9	0%
Sensitivity	48	46	4%	57	55	4%
PPV	14	17	20%	14	16	17%

Automated seizure detection event (ASDE), Positive Predictive Value (PPV)

3.3. Patting Metrics

The mean interval between patting events was found to be 95.1 ± 158.0 minutes. However, most subsequent patting events happened within 10 minutes, with a fluctuating amount of patting events in the next 6 hours.

Discussion

Most neonatal seizures are subclinical, and so neonates at risk for seizures are monitored with continuous EEG (26). Early neonatal seizure detection and rapid treatment often leads to better seizure control (29) and there is increasing evidence that a higher seizure burden is associated with worse short and long-term neurological outcome (30–32). Neonatal EEGs are susceptible to high degrees of artifact from patient movement, medical procedures, and basic care needs and thus video EEG with human review is typically used to classify artifact visually. In this article, we present a computer vision approach to augment, or eventually replace, human review of the video with automated motion analysis.

Most current automated seizure detection algorithms utilize input from EEG signal alone while video files are excluded from detection processing. High false positive rates may also limit the effectiveness of automated seizure detection algorithms, which increasingly have the ability to alert caregivers to detected events by paging or text messaging. When false positive rates are high, neurologists tend to ignore or disable alerts, as demonstrated in a recent study in which only two out of nine study neurologists utilized instant messaging alerts for seizure detection. Others found the false-positive alerts too disruptive to work commitments and sleep (26), essentially rendering the automated seizure detection algorithm ineffective for use in real-time.

We have employed optical flow spectral analysis of the video files for artifactual detection. In video-EEG epilepsy research, optical flow has been used for seizure detection in both adults and neonates (7, 9, 33, 34) and quantification of patient motion (33, 35, 36). Our study is the first to look at patting. When added in parallel to the seizure detection software, our algorithm significantly reduced false alarms due to rhythmic patting. The gold standard of continuous video EEG interpretation consists of a neurophysiologist who reviews EEG and opens the video files to differentiate between rhythmic artifact and rhythmic seizure activity. However, this gold standard approach is not feasible for real-time continuous seizure detection in most NICU’s due to resource constraints. Even real-time review from a commercial EEG technician is still inadequate to consistently assist the neurophysiologist in real-time review and response (26). In NEOLEV2, human staffing solutions such as asking nurses to pause the alarms were intermittently effective despite attempt at training, and it was felt by all providers that a technological solution would be preferable to maximize efficiency and alleviate the burden on bedside staff. Such a technological solution would also be useful for offline and after-the-fact review. Our computer vision optical flow spectral analysis endeavors to perform the visual function of artifact differentiation in an automated fashion such that real time detection and response may be achieved.

In our testing cohort, our algorithm resulted in a 50% reduction of false positive seizure detection events due to patting, while retaining 11 of 12 true positive events. However, despite this significant finding, our study has limitations. Our study was retrospective in design and not done in real-time. Our study found that patting was largely localized to lower spectral frequencies, and was associated with a large amplitude of overall optical flow. Thus, it is unlikely that our algorithm can be applied to automated seizure detection algorithms of older patients who may exhibit tonic-clonic seizures, as spectral analysis of optical flow of tonic-clonic seizure activity would likely overlap with patting activity.

As automated seizure detection technology continues to improve, and the age of the patient and other medical conditions is factored into the seizure decision algorithm, patting or other nursing manipulation detection could be appropriately merged into the algorithm and result in a significant improvement in the reliability of the seizure detection algorithm. One of our future goals is to create a real-time software plug-in to enhance the overall accuracy of our neonatal seizure detection algorithm, by merging features extracted from optical flow computer vision, the patient’s medical history, and complementary techniques and procedures developed to analyze the neonatal EEG. While recognition of seizures will remain dependent on human observation, an appropriate combination of automated video and EEG analytical approaches would be expected to result in a significant improvement in the ability to promptly detect seizure in this population. A highly sensitive and specific automated neonatal seizure detection algorithm would radically improve the management of neonatal seizures for all at risk infants.