A New Animated Training and Certification Application for the NIHSS

Abstract

Background:

The National Institutes of Health Stroke Scale is used globally to rate neurological deficits in acute stroke patients. Originally designed for research use, the scale uses idiosyncratic scoring rules and requires thorough training for proper use. Currently available training and certification systems are timeworn and limited in the range of demonstrated neurological findings. We aimed to develop a new training and certification method to address the limitations of the prior system.

Methods:

We describe a new animated system that uses state-of-the-art digital motion capture, avatar rendering, and digital animation that enables illustration of all possible scale item responses within a training module. For certification, we created 20 unique case vignettes, which can be viewed from many angles allowing better scoring scenario presentation. Training to use the scale via telemedicine was added. Alpha and beta testing was completed to enhance iterative development. From a penultimate version, we measured interrater agreement among 42 pilot users who each viewed 6 cases. After further improvement, we finalized the application and confirmed interrater agreement in a second sample of 365 consecutive, unselected users viewing 3 cases. Clinometric analysis methods followed our prior studies of the scale.

Results:

Users agreed the new, animated training module is more engaging and informative than legacy training videos. We compared the new system to the legacy live video certification system for clinometric reliability. In the confirmatory sample, the intraclass correlation coefficient was 0.979 [97.5% CI of 0.919 to 0.99]. Kappa scores ranged from 0.25 to 0.90 across all 15 scale items. These results are comparable to previous studies.

Conclusions:

A new, animated training and certification system showed equivalent clinometric properties to legacy systems, providing a reliable method for NIHSS training and certification. The new system allows modification and reconfiguration if needed to maintain user interest and case novelty.

Graphical Abstract

Cover. The NIHSS+ application provides a virtual reality experience for learning and certifying on the NIH Stroke Scale. Avatars are produced from live patients using motion capture technology, but animation software allows demonstration of all possible responses on each NIHSS item. Simulations are more engaging and interesting than legacy videos.

Introduction:

The National Institutes of Health Stroke Scale (NIHSS) is valid and reliable for measuring neurological deficits in acute stroke patients¹. The NIHSS was originally intended as an outcome of the National Institutes of Neurological Disorders and Stroke (NINDS) tissue plasminogen activator (tPA) for acute stroke trials; videotapes were made, clinometrically evaluated, and used to train and certify the investigators to use the NIHSS in that trial². Despite the fact that the NIHSS certification and testing program has been used globally for the past 20 years³, several issues have arisen that raise concerns about the efficacy of the ongoing training and certification system⁴.

A crucial limitation of the existing training system is the failure to demonstrate all possible findings across all scale items in the training module. In the original videotaping, patients presenting to a stroke center were enrolled as a convenience sample; because the variety of demonstrable deficits was limited by the obtainable convenience sample available not all possible response items on all stroke scale items could be documented during the videotaping.

Another limitation of the current method is that the cases used for certification are now widely recognized due to regulatory requirements for annual or biannual recertification. Unfortunately, an online search will reveal several purported ‘answer keys’ that claim to allow a user to pass the online certification examination without training to use or even understand the NIHSS (date of Google search 7/19/2024 using “NIHSS Answer Keys” as search terms).

Scoring of the certification cases online may be too lenient⁵. This lenience resulted from the limitations of the digital video recording process. Due to inadequate visualization of the patient’s response to the examiner’s command, despite professional recording in a modern television studio, for some items multiple answers are allowed—the original definition of a passing score was broadened. Over multiple certification studies, the items that are most difficult to present in digital video recordings are facial symmetry, eye movements, and ataxia^{3, 6}.

Another issue is that the NIHSS is now being used in a telemedicine environment⁵. Although remote use of the NIHSS is feasible, the current training system does not include guidance specific to telemedicine applications^7–11.

To address these and other problems, we designed and built a virtual reality-based, animated training and certification system. The training module included the verbatim text used in the original NIH investigators’ script, with additional text written for telemedicine use and supplementary ‘tips and tricks’ advice.

Here, we report on the preliminary reliability of the new system in comparison to the previous iterations of NIH stroke scale certification.

Methods

We followed Standards for Reporting Diagnostic Accuracy Studies¹². Collected data is available upon request from the corresponding author. The NIHSS consists of 11 ratings of neurological findings, some of which include sub-items for a total of 15 scorable items⁴. Each item has a set of possible responses that indicate the increasing severity of the observed neurological deficit. To assure reproducibility across users of variable training—neurologists, emergency physicians, research nurse coordinators, paramedics—there are scoring rules unique to each item that are idiosyncratic: sometimes a test is performed differently than a specialist in neurology would perform it. For example, motor drift is checked only with the arm fully pronated⁴. Therefore, thorough training and then certification is used to assure users fully understand the scoring rules. Over several demonstrations of the clinometric properties of the scale, we have shown repeatedly that training and certification is effective in achieving reproducibility across several different types of users^{1, 3, 13–18}.

To overcome the limitations of selecting cases only from available live patients, we created an animated application with patients presented as digitally rendered avatars (Figure). We used photogrammetry (Reality Capture, Epic Games Inc.) to scan human models and to create digital characters’ likenesses. (The model shown in the Figure provided consent for this picture to be published). To portray the reality of character motion, both the evaluator and patient performances were recorded with motion capture technology (Body: Rokoko Smart Suit Pro 2 with Smart Gloves, model # 50300001, Rokoko Electronics Inc., Facial: iPhone X a1901 TrueDepth Camera Sensor, Apple Inc.). This motion capture process uses body, finger, and face movement to drive the digital characters in scene. The motion capture solution is an internal markerless inertial base system, using data generated from an accelerometer, gyroscope, and magnetometer to deliver full spatial and rotational tracking. The motion capture suit has 19 Inertial Measurement Units that record 9 degrees of freedom. The sensors are connected to a hub, with 3 dimensional orientation accuracy: ± 1 degree; each sensor provides 200 Frames Per Second. These sensors are used to track human movement and feed data to be re-targeted as an animation to the digital character. For the facial performance capture, we used the TrueDepth Camera system, which projects and analyzes more than 30,000 invisible dots to create a precise depth map of the face. A demonstration video is available at https://nihss.plus/media/promo.

Figure. Build Overview and Animation workflow.

A) Each case vignette used for training or certification begins with a recording of a live subject. Using motion capture technology, the live action is captured digitally and then manipulated to create an animated character or avatar. Using gaming technology, each character is placed into an animated environment where additional props can be added and audio overlaid. B) Motion capture technology includes a body suit and facial recognition that are wirelessly acquired to allow actors freedom of unrestricted motion in the studio.

To recreate the training modules offered in prior NIHSS applications, we accessed the original script used by NIH investigators describing the proper scoring technique². The script was then re-recorded using high fidelity sound technology. To better illustrate the full range of possible stroke deficits, we recorded a trainer demonstrating all possible responses across all scale items. Additional enhancements were written and recorded to explain the proper performance of the NIHSS items in a telemedicine environment. Finally, tips and tricks sections were recorded for each item.

For the certification module, we devised 20 case vignettes. The certification test system was designed so that a user would see only 3 of the 20 patients every time they certified and would never see the same 3 patients in combination again. All 20 vignettes were carefully scripted to assure coverage of a balance of mild/severe and right/left sided strokes, and then actors were used in the motion capture environment to illustrate all the examination findings. Similarly, actors in motion capture equipment performed the stroke scale as examiners. One investigator served as the in-studio ‘director’ to assure faithful performance of each stroke scale item and continuity among all responses from all actors.

To document the new system’s clinometric performance, we used the same approach that we used to assess reliability in scoring the original 2 videotapes and 3 digital video disk (DVD) recordings^{1, 2}. First, a small group of expert users performed extensive alpha and beta testing by reviewing each training and certification case animation. Based on that feedback, the animations were enhanced or re-edited to ensure that all findings were clear and unambiguous. We then tested the first 6 animated case vignettes prospectively among 42 test users (all registered nurses new to the NIHSS and never previously trained) to determine ease of use with the user interface. This was Sample 1. Based on feedback from this small group, further enhancements were made to the user interface.

In a confirmatory sample, Sample 2, we prospectively studied 365 test users who had previously passed the legacy certification online and presented themselves for recertification at an online credentialing site—all users were included without restriction. These test users were assigned 3 animated certification cases for assessment and their scores were recorded whether or not they passed certification. To match our previous methodology, we used the scores from their first attempt, even if they failed. A ‘passing score’ was defined as it was in the original NINDS certification process: correctly scoring 93% of all test items across the 3 certification cases. That is, only 3 incorrect responses in total were allowed across the 45 tested items (3 cases × 15 items per case). In the event a user attempted the certification test more than once, only the first exam was used in this analysis whether it was a passing or a failing score. There were no missing data for the 45 responses per exam due to the nature of the software.

The intraclass correlation coefficient (ICC) was calculated to measure the strength of inter-rater agreement across the entire scale. The ICC captures the measurement variability for the same rater, compared to the variability measurement among all raters. An ICC value between 0.75 and 0.90 is considered good, and above 0.90 is excellent¹⁹. We calculated separately an ICC for the group that eventually passed and the group that never passed. Fleiss’s unweighted Kappa was used to estimate the interrater agreement for each scale item²⁰. The Fleiss Kappa is an extension of Cohen’s Kappa to multiple raters, which permits identifying how strongly agreement among raters may be attributed to chance. Perfect agreement would show Kappa=1, while poor agreement can produce values less than zero.

A mixed effects general linear model with a binomial link function was used to predict passing an exam, using the patients selected for assessment, question ID, and rater credentials (MD, RN) in the model. This model treated both subjects and raters as random effects sources. The Rater ID, as a random effect, adjusts for repeated measures and within-rater ability. This model permits the results to be generalized to new subjects similar to the raters used in this study. ICC was calculated using the random effects model and the dim_icc() routine in the agreement package within R (R 4.0.3 GUI 1.73 Catalina build (7892). Model type 1A was selected, with unit=single and type =agreement.

Fleiss Kappa was computed in R using the kappam.fleiss() routine in the IRR library within R. Fleiss Kappa was calculated within each of the 14 exam variations within each item and then averaged over all variations to obtain item-level Kappas.

Results

Sample 1 (42 raters, 6 patients each) consisted of nurses (all RN) who were not previously trained or certified and was collected from 8/8/2022 to 10/27/2022. Sample 2 (365 raters, 3 patients each) was collected from 11/22/2022 to 1/31/2023. In Sample 2, 233 users passed after one or more attempts, and 132 never passed. Of 233 users who passed on their first attempt (n=78), or eventually passed after one or more failed attempts (n=155) users self-identified as RN (192, 82%), MD (24, 10%), Nurse Practitioner (11, 5%), licensed practical nurses (3, 1%), or DO (3, 1%).. The group that never passed included 132 users who self-identified as RN (119, 90%), NP (4, 3%), LPN (1, 1%), MD (6, 4%), PA (1, 1%) and DO (1, 1%). (percents do not sum to 100 due to rounding). There was no significant difference across professions on GLM analysis.

The new animated application shows excellent reliability in both test samples with narrow confidence bounds. In Sample 2 the ICC for 233 users who passed was 0.979 [97.5% CI of 0.919 to 0.986]. For the 132 users who never passed, the ICC was 0.945 [0.813 to 0.963]. For the 78 users who passed on their first try, the ICC was 0.99 [0.934 to 0.995] and for the 155 who initially failed but then eventually passed, the ICC for their first attempt was 0.974 [0.911 to 0.982]. Because the sample sizes were modest and drawn from similar but not identical populations we deferred formal significance testing to compare the ICC from this study to prior work.

The item Kappa values are shown in the Table for Sample 1, and then separately for the Sample 2 users who eventually passed versus those who never passed. For comparison to the present sample, we chose the historical Kappa values estimated for the original DVD and web-based validation papers^{1, 31}. In Sample 1 some Kappa values were greater in the present, animated videos (Table). In particular, improved Kappas are seen for some of the notoriously difficult items, such as Limb Ataxia and Facial Palsy. In the larger Sample 2, however, the Kappa for Facial Palsy returned to that of prior studies. This greater level of agreement may be influenced, however, by the number of patients tested (n=3) as agreement is more straightforward when fewer items are assessed.

Table. Kappa scores for each NIHSS item across the current and prior validation studies.

The NIHSS includes 15 items with multiple choices. The Kappa statistic describes the agreement among users for each of the items; higher values connote better agreement. For comparison, Kappa values are shown for the original DVD validation study and a sample of web-based users as estimate and 95% confidence interval. Sample 1 included novice users. Sample 2 included certified users, of whom 233 passed and 132 failed to certify. Items that have traditionally shown poor agreement, such as Facial Palsy, Ataxia, and Dysarthria appear to have improved with the animated version.

	Original DVD	Web	Sample 1	Sample 2 Passed	Sample 2 Failed
Number of Users Tested	724	7419	42	233	132
1a. LOC	0.46 (0.39, 0.53)	0.43 (0.01, 0.51)	0.63 (0.61, 0.66)	0.56 (0.42,0.70)	0.44 (0.32,0.56)
1b. LOC Questions	0.77 (0.64, 0.90)	0.77 (0.66, 0.84)	1.00	0.76 (0.66,0.86)	0.72 (0.58,0.86)
1c. LOC Commands	0.92 (0.75, 1.0)	0.79 (0, 0.85)	1.00	0.90 (0.86,0.94)	0.81 (0.7,0.92)
2. Best Gaze	0.70 (0.39, 1.0)	0.44 (0.03, 0.63)	0.86 (0.84, 0.87)	0.71 (0.6,0.82)	0.59 (0.47,0.71)
3. Visual	0.72 (0.57, 0.87)	0.56 (0.20, 0.61)	0.92 (o.91, 0.93)	0.68 (0.58,0.78)	0.50 (0.43,0.57)
4. Facial Palsy	0.38 (0.27, 0.49)	0.25 (0.14, 0.32)	0.67 (0.64, 0.69	0.25 (0.13,0.37)	0.29 (0.19,0.39)
5a. Motor Arm (Left)	0.65 (0.51, 0.79)	0.51 (0.18, 0.61)	0.85 (0.83, 0.86)	0.71 (0.58,0.84)	0.66 (0.55,0.77)
5b. Motor Arm (Right)	0.72 (0.54, 0.90)	0.51 (0.26, 0.65)	0.82 (0.80, 0.83)	0.63 (0.58,0.68)	0.55 (0.47,0.63)
6a. Motor Leg (Left)	0.64 (0.51, 0.77)	0.66 (0.53, 0.74)	0.79 (0.77, 0.81)	0.67 (0.53,0.81)	0.63 (0.49,0.77)
6b. Motor Leg (Right)	0.64 (0.53, 0.75)	0.59 (0.49, 0.64)	0.77 (0.75, 0.79)	0.63 (0.54,0.72)	0.58 (0.51,0.65)
7. Limb Ataxia	0.21 (0.12,0.30)	0.15 (0.06, 0.22)	0.57 (0.54, 0.60)	0.48 (0.36,0.6)	0.40 (0.27,0.53)
8. Sensory	0.73 (0.53, 0.93)	0.53 (0.16, 0.69)	0.81 (0.79, 0.83)	0.52 (0.42,0.62)	0.33 (0.2,0.46)
9. Best Language	0.64 (0.53, 0.75)	0.59 (0.36, 0.72)	0.89 (0.88, 0.90)	0.63 (0.54,0.72)	0.54 (0.46,0.62)
10. Dysarthria	0.56 (0.39, 0.73)	0.46 (0.27, 0.59)	0.96 (0.95, 0.96)	0.80 (0.72,0.88)	0.77 (0.67,0.87)
11. Extinction and Inattention	0.57 (0.40, 0.74)	0.61 (0.49, 0.65)	0.84 (0.82. 0.85)	0.66 (0.53,0.79)	0.60 (0.48,0.72)

κ (95% CL)

Discussion

The data presented here establish that a new, animated NIHSS certification system shows clinometric properties that are nominally similar to previous systems that were based on live patient video recordings. In our prior studies of the NIHSS across multiple platforms, including DVD and online, the ICC values ranged from 0.83 to 0.93¹. Interrater reliability across all items (Table) and overall ICC values in this study resemble those previously documented by us for videotapes, DVDs and online training^1–3. The distribution of user occupations in Samples 1 and 2 resemble that of our prior clinometric studies of the NIHSS^1–3. This data confirms that the NIHSS performs as intended in an animated environment, although further confirmatory testing is needed with a larger sample size that would allow for hypothesis testing and calculation of statistical significance.

Although the NIHSS was designed as a research tool, it is now ubiquitous in clinical vascular neurology. Thus, many users are required to certify but may have little interest in learning the details of proper NIHSS administration⁴. Re-certification is required annually or bi-annually by regulatory authorities despite our previously published data showing such frequency may be unnecessary²¹.

As knowledge of proper certification and use of the scale has degraded and online ‘answer keys’ have become available, we see more users pass and become certified who do not fully understand the use of the scale. Confirming this observation, of 365 previously certified users in Sample 2, only 233 (64%) passed this new certification test and 132 (36%) never passed even after repeated attempts, suggesting a lack of thorough understanding. The new animated training and certification application seeks to address this issue. To pass, users are expected to understand the scoring rules sufficiently well to achieve a passing score, 93% correct ratings. This is the same requirement as was used in the original scale NINDS study of TPA². In that study users were allowed to miss the equivalent of one item per patient and were certified using either 5 or 6 patients, meaning they were allowed to miss either 5 or 6 items depending on which videotape they used. This method of scoring was extended into the DVD certification era during which users certified on 6 patients and so were allowed to miss up to 6 items to pass. In the present study users viewed 3 patients and therefore are allowed to miss 3 items. It is important to note, that during the videotape and DVD era, there was concern that the video did not clearly illustrate all exam findings. Therefore, we adopted a modal based system in which more than one answer were allowed to be correct if users frequently scored two answers equally often. The practice led to criticism that the scoring became overly lenient⁵. To address overly lenient pass rates, the new scoring algorithm allows only one correct answer for each item.

The new system addresses other problems with the current live-video based system. With a complete library of demonstrated patient responses, the animated training module assures that users can view and learn all possible responses to every NIHSS item. The large number of certification cases guarantees that repeat users will not see the same cases recurrently. Further, the animated case vignettes can be modified at any time so that the neurological findings can differ each time a user encounters a repeat case, even if the user recognizes the patient avatar. This should mitigate the publication of ‘answer keys’ online. Due to the greater flexibility in manipulating the case presentation—including zoom, pan and selecting different viewing angles—scoring can be rigidly limited to only correct responses without the need to allow incorrect responses as passing due to poor video quality, as we were forced to do previously. Updated pictures have been substituted for legacy pictures that included outmoded stereotypes and images that may not be identifiable in less developed nations around the globe^{22, 23}.

Live video failed to illustrate subtle neurological findings, such as facial asymmetry, in previous certification videos (Table)^{1–3, 24}. The motion capture software package analyzes more than 50 different muscle movements to mirror facial movement and expressions. The result is a highly detailed capture of the face, eyeballs and even tongue. These features allow for high fidelity, intricately detailed animations. With this as a basis for creating animations, the system can accurately recreate any deficit, from any angle, for any scenario. Unfortunately, in our larger Sample 2, the Facial Palsy Kappa scores were the same as in legacy systems, so further evaluation of this methodology is needed to improve reproducibility on this item.

There are limitations to the present investigation. First, we did not require that Sample 2 users watch the new training module, since all were previously certified; hence we do not know the effect of the digital training animations on scoring success. Another limitation of this investigation is the small sample size. Although comparable to previous clinometric experiments, repeating the same experiment with a larger sample size would be more robust. While the data here speak to reproducibility across users, we cannot assess validity of the scores compared to a live patient, or compared to the old system, which would require a very different experimental design.

In conclusion, a new animation-based application for training and certification to use the NIHSS appears to be reliable. Clinometric properties are nominally similar to legacy versions of training and validation, although this point requires further assessment. Further testing on a larger group of subjects is needed to confirm these preliminary descriptions of reliability.

List of abbreviations

DVD

digital video disk

ICC

intraclass correlation coefficient

NIHSS

National Institutes of Health Stroke Scale

NINDS

National Institute of Neurological Disorders and Stroke

tPA

tissue plasminogen activator