Resident training and interrater agreements using the ACNS critical care EEG terminology

Abstract

Purpose:

Electroencephalography (EEG) remains the gold standard for identifying rhythmic and periodic patterns in critically ill patients. Residents have frequent exposures to EEG and critically ill patients during their training. Our study aimed to assess resident competency in the use of the American Clinical Neurophysiology Society (ACNS) critical care EEG terminology.

Methods:

After self-guided reading and a 2-hour session reviewing the ACNS critical care EEG Terminology training slides, 16 adult neurology residents (PGY 2–4) completed the ACNS certification test. Performance scores were reported as average percent agreement (PA%) with a previously established 5-member expert panel. Interrater agreement was calculated to gauge consensus among peers within the resident cohort. Self-reported comfort levels using the terminology were also obtained.

Results:

The overall pass rate for our cohort was 50% and the median score was 74%. The terms with the highest PA% were: seizures (86.4%), main term 1 (78%), main term 2 (74%). Interrater agreement scores (kappa values) were almost perfect for seizure, and substantial for main terms 1 and 2.

Conclusions:

Our data suggests that with minimal investment, adult neurology residents at various stages of training can effectively learn the ACNS critical care EEG Terminology.

1. Introduction

Nonconvulsive status epilepticus (NCSE) is an important diagnosis to consider when patients present with altered level of consciousness. NCSE has been documented in approximately 8% of comatose patients admitted to ICU without clinical signs of seizure activity [1]. Delay in the diagnosis of NCSE > 24h results in increased mortality from 39% to 75% [2]. An EEG is essential for the diagnosis of NCSE.

The American Clinical Neurophysiology Society (ACNS) recommends using the ACNS Critical Care EEG terminology to describe EEG patterns seen with critically ill patients such as those in NCSE [3,4] (please refer to supplemental material for review of this nomenclature). The interrater agreement for most terms has been found to be high among neurophysiologists and fellows but this was not specifically assessed in adult neurology residents [5]. Residents are currently an untapped resource in the management of critically ill patients with non-convulsive epileptic seizures. Since they form the majority of the patient-facing workforce in Canada/USA, increasing their competency in EEG readings for critically ill patients could potentially improve their recognition of electrographic seizures. It has been shown that adult neurology residents who are asked to cover night call in resource poor settings can be trained to set up limited coverage EEGs for critically ill patients [6]. To perform EEGs in the critical care setting, residents would benefit from familiarity with the ACNS terminology which would in turn help them when interpreting EEG records, if they are asked to do so.

The goal of this study was to determine if adult neurology residents could be trained and certified in the use of the 2012 ACNS Critical Care EEG Terminology as well as identify whether this subset of trainees could achieve high interrater agreements for the concepts utilized in the terminology.

2. Methods

2.1. Ethics and participant consents

This study was approved by our institutional review board. All of the residents consented to participate in this study.

2.2. Canadian residency and EEG curriculum

Our academic centre is an acute tertiary care hospital located in Ottawa, Ontario, Canada. Like all Canadian neurology residencies, we are a five-year adult neurology program. In all Canadian residency programs EEG/Epilepsy is a core 2-month rotation and is a mandatory component of residency training. Our program currently numbers 20 residents ranging from postgraduate years (PGY) 1–5. In our study, we enrolled 16 residents from PGY 2–4. In our institution, the two-month EEG/Epilepsy rotation occurs in PGY3, hence only a subset of residents at the time of this study had previously completed the core epilepsy block of their residency training. It is important to note that some junior residents had not had prior EEG training before the launch of this study.

2.3. Participant selection

The study was open to all junior and senior residents on a voluntary basis. PGY 1 residents, whom had never been exposed to EEG or Epilepsy during their training, declined to participate. Similarly, PGY 5 residents did not participate given their proximity to board examinations. Participants included residents from PGY 2–4 levels of residency training. Of note, 1 paediatric neurology resident, from the Children’s Hospital of Eastern Ontario, who was completing a 6-month mandatory adult neurology rotation, also participated in this study.

2.4. EEG samples and ACNS cEEG terminology examination

ACNS Critical Care EEG training for this cohort of residents consisted of self-study of two articles [7,8] and standardized training slides retrieved from the ACNS website (https://www.acns.org/research/critical-care-eeg-monitoring-research-consortium-ccemrc/education). Then residents had the opportunity to review the training slides with an epilepsy specialist who was trained and certified in the ACNS terminology (TF) and have their questions answered during a two-hour didactic teaching period. Subsequently, 16 residents wrote the previously described ACNS Terminology Certification test. Details of this examination can be found elsewhere [5] but to briefly summarize: this online examination consists of 37 EEG samples, each comprising 11 standardized questions about the EEG samples, testing 15 individual concepts or terms from this nomenclature (e.g. seizure, main term 1, main term 2, plus modifier, sharpness, amplitude, frequency, phases, evolution, triphasic morphology, etc). The examination is open book and can be done in multiple sittings. An average percent agreement score of 70% or higher was considered a passing grade for this examination. Overall individual performance scores were calculated as average percentage of correct answers, averaged over all questions.

As part of the certification assessment, participants were also asked to rate their comfort levels with the terminology using the following qualifiers: Very Uncomfortable (VU), Uncomfortable (U), Neutral (N), Comfortable (C), Very Comfortable (VC).

2.5. Statistical analysis

The concepts such as seizure, main term 1, main term 2, plus modifier, +F, +R, +S, and triphasic morphology were considered as categorical concepts, whereas sharpness, absolute amplitude, relative amplitude, frequency, number of phases, and evolution were considered as ordinal concepts. Interrater agreements (IRA) was quantified using Gwet’s multirater agreement coefficients AC1 and AC2 for categorical data and ordinal data respectively [10]. Agreements on categorical assessments were considered to be all-or-none. Partial agreement for ordinal data was scored using a conventional quadratic penalty function, adjusted for the number of possible choices [9].

Possible effects of comfort with the ACNS EEG terminology were investigated by linear regression analysis applied to plots of the scores of each resident participant (in terms of percent agreement) versus degree of comfort. Microsoft Excel (2016) and SAS (Cary, NC, USA) were used for the data analysis.

3. Results

16 neurology residents ranging from the second to fourth year of training (PGY 2–4) were enrolled in the study. The distribution of residency level in our cohort is outlined in Table 1. Five of the residents were at a PGY 4 level, 5 were at a PGY 3 level and 6 were at a PGY 2 level. Residents were considered “seniors” if they were in their PGY 3–4 years while PGY 2 residents were considered to be “junior”. Fifty percent of our cohort had no prior EEG training other than the initial introductory lecture to the ACNS EEG terminology while the remaining half had already finished their 2-month mandatory EEG/Epilepsy rotation as part of their residency curriculum.

Table 1.

Participating Resident characteristics.

Resident training (N= 16)	N (%)
PGY level
PGY2	6 (38%)
PGY3	5 (31%)
PGY4	5 (31%)
EEG exposure
Previous EEG training	8 (50%)
No EEG training	8 (50%)

3.1. ACNS terminology testing results

The percent agreement and estimated interrater agreements using the Gwet kappa values are outlined in Table 2. The passing rate for the certification test was 50% in our cohort of residents. The median percent agreement in our cohort was 74% (IQR 63.6%−88.4%).

Table 2.

Percent Agreement and Kappa values for tested terminology items.

Terminology item	Percent Agreement	Gwet Kappa	Lower CI	Upper CI
Seizure	86.44%	82.31%	80.67%	83.95%
Main Term 1	78.04%	72.87%	70.95%	74.78%
Main Term 2	73.98%	63.40%	61.84%	64.95%
Plus modifier	41.68%	33.36%	32.01%	34.71%
Any “+”	57.40%	43.63%	42.74%	44.51%
+ Fast Activity (F)	63.55%	51.69%	50.00%	53.39%
+ Rhythmic Activity (R)	66.32%	51.93%	50.10%	53.75%
+ Spike or Sharply contoured (S)	73.19%	65.43%	63.33%	67.53%
Sharpness	90.35%	71.98%	70.58%	73.38%
Absolute amplitude	94.32%	88.49%	87.95%	89.03%
Relative amplitude	76.09%	36.20%	32.44%	39.96%
Frequency	91.47%	77.35%	75.97%	78.73%
Number of phases	81.86%	35.06%	32.54%	37.58%
Evolution	73.72%	38.56%	35.93%	41.19%
Triphasic Morphology	49.91%	28.88%	27.66%	30.11%

Agreement was almost perfect for “seizure” and “absolute amplitude”, while agreements were substantial for “main term 1”, “main term 2”, “+ S modifier”, “sharpness”, “frequency”. In particular, percent agreement scores for “seizure”, “main term 1” and “main term 2” were 86.4%, 78% and 74% respectively.

Agreement was noted to be fair for “evolution”, “triphasic morphology” or “plus modifier”. No terms were found to have poor agreement. Similar to a companion paper initially outlining the interrater agreements of ACNS cEEG terms, individual scores on each of 15 tested concepts were displayed as a heat map in (Fig. 1). The participants were rank ordered by their overall performance score.

Fig. 1.

Heatmap diagram of individual scores. Scores (calculated as percent agreement, are coded by color according to the color bar on the right, for all 16 individuals and for each of 15 different ACNS Critical Care EEG terminology concepts. Of note, participants are rank-ordered according to overall score.

Resident comfort levels with the tested concepts and ACNS terminology overall were obtained as part of a post examination survey and are included in Table 3. Interestingly 50% of participants felt uncomfortable or very uncomfortable using the revised terminology.

Table 3.

Resident self-reported comfort level with Terminology.

Participant comfort levels (N = 16)	N (%)
Very uncomfortable	4 (25%)
Uncomfortable	4 (25%)
Neutral	5 (33%)
Comfortable	2 (12.5%)
Very uncomfortable	1(6.25%)

For each of the 15 concepts, linear regression was fitted between comfort level and percent agreement score (Fig. 2); this analysis failed to reveal a significant association for all ACNS concepts except the “+ R” modifier where a negative association was observed.

Fig. 2.

Effects of comfort level with ACNS Standardized Critical Care EEG Terminology. Scores (percent agreement) were plotted for all resident participants within five groups of self-reported EEG comfort levels with the terminology: (UC, very uncomfortable; U, uncomfortable; N, neutral; C, comfortable; VC, very comfortable). Linear regression lines were fitted to the data, with x values reflecting comfort levels, as described. The slopes and associated p-values are shown on each graph.

3.2. Differences amongst resident groups

Our senior resident (n = 10) passing rate was 70%, while our junior resident (n = 6) passing rate was 17%. The ability of a resident to successfully score above 70% for each of ‘seizure’ recognition, ‘main term 1’ and ‘main term 2’ was reviewed and the passing rate for this combined competence was 90% for seniors while it was 50% for junior residents. The median passing score for seniors was 75%, compared to 70% for juniors.

4. Discussion

In our study, the goal was to determine whether adult neurology residents could be trained and certified in the 2012 ACNS Critical Care EEG Terminology. Our results suggest that this can be achieved with minimal investment on the preceptor’s part. It is worth nothing that while 50% of our study cohort achieved an overall score of 70% or greater and passed the certification examination, 12/16 (75%) of our cohort achieved a score of 70% or higher on the major concepts of “seizure”, “main term 1” and “main terms 2”, (data not shown). This was true of senior and junior residents alike (see above 3.2). This would suggest that for the critical concepts of this terminology, namely seizure recognition and identifying rhythmic or periodic patterns, residents can be taught and achieve a passing level of proficiency with these terms regardless of level of training or prior EEG/epilepsy exposure. Indeed, this is supported by the substantial or almost perfect IRAs in our cohort for the major terms, namely “seizure”, “main terms 1 and 2”, which is in line with prior studies where subspecialist IRAs were obtained for these terms [5]. There are practical implications to these findings. Residents are often the first line of neurological care in university hospitals; their ability to identify the more malignant patterns in the terminology, including identification of nonconvulsive seizures, would allow prompt recognition, treatment and management of this critically ill population. This has the potential to enhance overnight care in tertiary care institutions while refining the residents’ skillset.

In line with prior studies, even at our cohort’s level, the lowest IRAs were observed for the “plus modifier” concepts as well as the concepts of “evolution” or “triphasic morphology”. Even for seasoned electro-encephalographers (EEGers) these concepts can be difficult to integrate, making this an unsurprising finding in our cohort of neurology residents. This would also suggest that these concepts may be more difficult to define than previously anticipated, rather than being difficult to grasp or acquired by trainees or EEGers.

Interestingly our resident cohort may have had difficulty judging their comfort levels with the terminology. Though 50% were self-described as uncomfortable using this terminology, 75% showed proficiency in identifying and correctly using the main 3 terms (seizure, main terms 1 and 2) thereby suggesting that their comfort levels were gauged to be lower based on the remaining concept terms, perhaps supported by their lower IRAs for those terms. It is also possible that for some residents the discomfort stemmed from lack of formal exposure to EEG, prior to the commencement of the study. The negative association between comfort level and PA% scores for the “ + R” modifier would suggest a paradoxical overconfidence with some terms however this is hypothetical.

Though our study was not designed to address this question, some observations can be made about the use of resident level EEG data for the purposes of research studies. The ACNS critical care EEG terminology has allowed a common framework for approaching EEG studies across different institutions with electroencephalographers of varying expertise. As a result, large cross institutional collaborations are underway. Our results suggest that residents can be trained and certified in this terminology, which would help maintain institutional validity for the resident level EEG data being acquired. This would allow another source of critically ill EEG data capture to occur, provided it were validated by a certified epileptologist or neurophysiologist.

There are a number of limitations to our study. Many of the inherent limitations in using the ACNS critical care EEG terminology or the ACNS training slide sets and certification examination are outlined elsewhere [5] and will not be re-described here. The first limitation lies in the low number of residents recruited for the study. For logistical considerations, we employed a volunteering based approach to recruitment, as a result we were left excluding PGY 1 and PGY 5 level residents, the latter being our most seasoned trainees. Practically speaking however PGY 2–4 level residents are the cohorts most likely to use the acquired concepts and terminology in clinical practice given their more frequent exposure to critically ill neurology patients and EEGs.

Ours is a single centre study and this limits generalizability to an extent. Canadian residency programs however are generally uniform in their structures and all carry Royal college accreditation, as a result we feel that all adult neurology residents across different institutions can show similar levels of proficiency in the terminology if properly trained. The caveat may be our sister French speaking institutions which may require a translated training slide deck and certification examination, but this could be accomplished in the future. It would of course remain incumbent on staff neurologists to provide adequate supervision to trainees using the terminology in the clinical setting, which would not change current practice at this stage.

This study highlights interesting future directions given its results. We have currently trained our residents in the application of full-montage EEG caps to determine if they are able to identify malignant or non-malignant EEG patterns in real life conditions, using the concepts they have learned through the ACNS terminology. This study is ongoing at this time.

5. Conclusion

In this study we observed that 16 neurology residents can be trained in the ACNS critical care EEG terminology. Half of the residents became proficient enough to pass the certification examination offered by the ACNS, and senior residents had a superior pass rate compared to their junior counterparts. Importantly, there was very high percent agreement as well as substantial or near perfect IRA for the key terms in the terminology such as seizure, main term 1 and main term 2, and no terms showed poor or no agreement in our cohort. This study validates the use of the critical care EEG terminology by neurology residents.