Video training and certification program improves reliability of postischemic neurologic deficit measurement in the rat

Abstract

Scoring systems are used to measure behavioral deficits in stroke research. Video-assisted training is used to standardize stroke-related neurologic deficit scoring in humans. We hypothesized that a video-assisted training and certification program can improve inter-rater reliability in assessing neurologic function after middle cerebral artery occlusion in rats. Three expert raters scored neurologic deficits in post-middle cerebral artery occlusion rats using three published systems having different complexity levels (3, 18, or 48 points). The system having the highest point estimate for the correlation between neurologic score and infarct size was selected to create a video-assisted training and certification program. Eight trainee raters completed the video-assisted training and certification program. Inter-rater agreement (κ score) and agreement with expert consensus scores were measured before and after video-assisted training and certification program completion. The 48-point system correlated best with infarct size. Video-assisted training and certification improved agreement with expert consensus scores (pretraining = 65 ± 10, posttraining = 87 ± 14, 112 possible scores, P < 0.0001), median number of trainee raters with scores within ±2 points of the expert consensus score (pretraining = 4, posttraining = 6.5, P < 0.01), categories with κ > 0.4 (pretraining = 4, posttraining = 9), and number of categories with an improvement in the κ score from pretraining to posttraining (n = 6). Video-assisted training and certification improved trainee inter-rater reliability and agreement with expert consensus behavioral scores in rats after middle cerebral artery occlusion. Video-assisted training and certification may be useful in multilaboratory preclinical studies.

Introduction

Functional outcome is the primary determinant of stroke therapeutic efficacy in clinical stroke trials. Consensus recommendations for preclinical stroke therapeutic development include functional outcome assessment.^1,2 Recently, it was suggested that late stage preclinical therapeutic development includes a multi-institutional study in which a single experimental protocol is deployed across multiple laboratories to confirm efficacy robustness.^3,4 An example of this study design has now been published.⁵ To standardize behavioral outcome assessment across an array of investigators and laboratories, efforts may be necessary to assure inter-observer reliability so as to minimize variability that could obscure a treatment effect. Development of a video-assisted training and certification program (VTCP) modeled after those used in human clinical stroke trials (e.g. the National Institutes of Health Stroke Scale (NIHSS))^6–9 could be useful for this purpose.

This investigation compared three published behavioral scoring systems, representing a range of complexities, designed to assess neurological deficits after middle cerebral artery occlusion (MCAO) in rats. After identifying the system having the highest point estimate for the correlation with cerebral infarct size, we developed a VTCP and tested its effect on inter-rater reliability and agreement with expert consensus scores.

Materials and methods

Procedures using animals including video recording were approved by the Duke University Animal Care and Use Committee, Durham, NC. The Institutional Review Board determined that inclusion of unpaid human volunteers to test the training program was exempt from further review (Pro00059581). Experiments were performed according to the National Institutes of Health guide for care and use of laboratory animals, and ARRIVE guidelines (www.nc3rs.org.uk/arrive-guidelines).

Selection of the behavioral scoring system

Sixteen male Wistar rats (age 10–14 weeks, Harlan Laboratories, Inc., Indianapolis, IN) were anesthetized with isoflurane, mechanically ventilated, and subjected intraluminal MCAO as previously described.¹⁰ Laser Doppler flowmetry was not employed.¹¹ MCAO duration was 60, 75, or 90 min (n = 4–5 rats per MCAO duration) to produce a range of infarct volumes and corresponding neurologic deficit severities.

Three published scoring systems designed to assess behavioral deficits after MCAO were considered for development of the VTCP. The systems have different levels of complexity (Supplemental Table 1). The Bederson system focuses on three functions within a single category (forelimb flexion, resistance to lateral push, and circling) with a total score range of 0–3 points.¹² The Garcia system employs six categories (spontaneous activity, movement symmetry, forelimb symmetry, climbing, and trunk and vibrissae touch) with a total score range of 0–18 points.¹³ The Yokoo system incorporates elements of both the Bederson and Garcia systems but offers a more diverse scoring range (0–48 total points) derived from 14 categories.¹⁴

Fourteen days post-MCAO, three independent expert raters, each having >four years experience scoring post-MCAO neurologic function using all elements of the three systems, simultaneously scored neurologic function in the 16 rats using the three systems while observing each rat being handled by a fourth expert rater. All expert raters were masked to ischemia durations and did not confer with the other expert raters during the scoring process. Tests common to more than one system (e.g. forelimb symmetry) were performed only once.

After scoring, rats were euthanized, brains removed, frozen, histologically sectioned, and stained with hematoxylin and eosin. A different investigator, unaware of neurologic scores, measured cerebral infarct volumes planimetrically.¹⁰ Pearson’s correlation coefficients were calculated for the neurologic scores from each of the three systems and cerebral infarct volume, using values pooled across the three expert raters. We a priori determined that the system having the highest R value point estimate for infarct volume would be used for further study. On this basis, the Yokoo system was selected for the VTCP (see “Results”).

Video recording of neurologic deficits

Forty-one male Wistar rats had intraluminal MCAO with the ischemia duration ranging from 60 to 120 min to provide various deficit severities. Rats were allowed to survive for 14 days for video recording of behavioral testing on post-MCAO days 1, 3, 7, and 14. These 41 rats included the 16 post-MCAO rats from the previous experiment, which were also video recorded on the same days post-MCAO. Because functional deficits improve with time, this allowed us to collect a spectrum of neurologic deficit severity videos for use in the VTCP.

Video recording was performed in a dedicated room with controlled light, sound, and temperature. A flat black impermeable surface (76 cm × 76 cm), disinfected between animals, was positioned 76 cm above the floor. A “studio” was created with black vertical drapes placed approximately 60 cm around the testing surface. Video cameras (Eos D60 and Powershot G11, Canon USA, Inc., NY, and DSC-WX5, Sony Co., Japan) were used for recording. One was mounted 75 cm above the testing surface for a dorsal view. A second camera was mounted on movable tripod adjacent to the testing surface for lateral views. After all behavioral testing was completed, the rats were euthanized and cerebral infarct volumes planimetrically measured.¹⁰

Video segments were selected to obtain examples of each level of deficit in each scoring category, edited using iMovie 9.0.9 (Apple Inc., Cupertino, CA), and superimposed with a scripted narrative.

VTCP

The VTCP contained two sections. The Training Video (Supplemental Video 1) was a tutorial, providing narrated examples of each level of deficit within each category (≈2 h in length). The Certification Video (Supplemental Video 2) provided videos of complete neurologic examinations, for which trainee raters were asked to provide scores for each rat (≈2.5 h).

The Training Video offered five learning resources: 1, demonstration of how to handle rats and evaluate normal behavior; 2, introduction of the neurologic scoring system; 3, examples of each level of post-MCAO deficit within each scoring category; 4, key points for assigning scores; and 5, an examination followed by narrated definition of expert consensus scores.

The Certification Video presented five post-MCAO rats. A video segment was collected reflecting all 14 scoring categories in each rat. The trainee raters were provided with copies of the published scoring criteria¹⁴ and a score sheet for each rat (Supplemental Figure 1). Upon completion of the Certification Video, a single investigator (HT) checked the newly trained raters’ scores. For each category, the trainee rater was required to have the expert consensus score for at least four of the five rats. If the trainee rater failed to achieve this criterion, the investigator asked the trainee rater to review the corresponding segment of the Training Video. The trainee rater was then asked to repeat the corresponding segment of the Certification Video. This process was repeated until the trainee rater scored all five rats in at least 11 of the 14 (79%) of the categories consistent with the expert consensus scores.

To determine expert consensus scores, the same three expert raters involved in selecting the best neurologic scoring system were presented with video clips of complete neurologic examinations (all 14 categories) for the Certification Video and pre- and posttraining tests (pretest and posttest, respectively). Each rater independently scored each rat. The three expert raters then met, discussed each video clip, and obtained an expert consensus score. These expert consensus scores were the basis for grading trainee rater performance in the Certification Video and pre- and posttests.

Evaluation of VTCP effectiveness (Figure 1)

Eight individuals, having no prior exposure to the VTCP or use of the Yokoo system, volunteered to serve as VTCP trainee raters. Seven of these individuals had 1–3 years of experience handling rodents and scoring neurologic function. The eighth had served >10 years in a stroke research laboratory as a neurohistologist. All were fluent in English. Pre- and posttests employed eight different post-MCAO rats, selected to present an array of deficit severities. To assure a similar level of difficulty in the pre- and posttests, two investigators (M.P. and H.T.) scored all rats. From this, pre- and posttests were generated with similar levels of difficulty. Expert consensus scores were never provided to the trainees who were only given copies of the published scoring criteria for the Yokoo system, a score sheet, and simplified directions regarding key points for each behavioral category during the tests (Supplemental Table 2). Given eight video-recorded rats and 14 categories within the scoring system, there were 112 possible scores in both the pretest and posttest to compare against expert consensus scores for each trainee rater.

Trainee raters performed all aspects of training and testing independently. For each trainee rater, the pretest and Training Video were performed on the first day. The Certification Video and the posttest were performed the next day. Both testing and training were performed in the same location using a single computer containing all video content. VTCP content was unavailable during the pre- and posttest. No individuals were allowed to interact with the trainee raters during the pre- and posttest.

Data analysis

The a priori primary measure was change in trainee rater agreement with expert consensus scores between the pre- and posttest, which was compared using a paired t test.

Given the large range of potential scores for a single rat (i.e. 0–48), the chance trainee raters’ total scores corresponded precisely to expert consensus total scores was anticipated to be low. Therefore, in a planned secondary analyses, the data was reanalyzed to determine the frequency that trainee raters derived a total score ±2 points within the expert consensus total score for each rat in the pre- and posttest. Because the data were not parametrically distributed, these frequencies (pretest versus posttest) were compared using the Mann–Whitney U-statistic.

We then determined the frequency with which trainee raters correctly scored each category in the scoring system by counting the number of trainee raters who agreed with the expert consensus scores in six or more of eight rats for each category. These data were qualitatively compared between the pre- and posttest.

Finally, inter-rater agreement was evaluated using the un-weighted κ statistic for multiple raters. Fleiss et al.¹⁵ categorized un-weighted κ coefficients as follows: <0.40 = poor, 0.40–0.75 = moderate, and κ > 0.75 = excellent agreement, respectively. We measured the number of categories with κ ≥ 0.40 and computed 95% confidence intervals (CI) for each of the 14 categories between the pre- and posttest. The number of categories with significant improvement (i.e. no overlap in the 95% CI) was tabulated. Normally distributed values are shown as mean ± standard deviation. P < 0.05 was considered statistically significant.

Results

Selection of the behavioral scoring system

Figure 2 depicts relationships between total neurologic score and infarct volume for each scoring system summarized for the three expert raters (for individual rater values see Supplemental Figure 2(a) to (c)). All systems positively correlated with infarct size and were not statistically different. The Yokoo system had the highest point estimate R value (Yokoo: R = 0.80, 95% CI: 0.67–0.89, p < 0.0001, Garcia: R = 0.76, 95% CI: 0.61–0.86, p < 0.0001, Bederson: R = 0.68, 95% CI: 0.50–0.81, p < 0.0001). The poorest agreement among observers was in the sensory categories (Supplemental Table 3) suggesting need for focused training in measurement of sensory deficit.

Figure 1.

Overview of video training program and effectiveness evaluation procedure. MCAO: middle cerebral artery occlusion.

Figure 2.

Relationship between total infarct volume and total neurologic scores in (a) Yokoo,¹⁴ (b) Garcia,¹³ and (c) Bederson¹² scoring systems by three expert raters. Correlation coefficient formulas were Yokoo: y = 0.0517x−0.9525, Garcia: y = 0.0194x−0.2971, and Bederson: y = 0.0087x−0.1081. Note: 18 = no neurologic deficit in Garcia scoring system.

Evaluation of VTCP effectiveness

All trainee raters completed all components of the VTCP and pre- and posttest. The time required to perform the training and certification videos was 146 ± 34 and 153 ± 62 min, respectively. Thus, the duration for VTCP completion was ≈5 h. Durations for the pre- and posttest were 103 ± 32 and 79 ± 20 min, respectively. The duration for completion of all elements ranged from 340 to 655 min. Total cerebral infarct volumes for the pre- and posttest rats were 228 ± 43 and 221 ± 73 mm³, respectively.

Agreement with expert consensus scores increased between the pre- and posttest in all trainee raters (Table 1), with an average increase of 21 ± 7 points (p < 0.0001, β = <0.001). Figure 3 shows the range of agreement between expert consensus score and assigned total neurologic scores for each rat for each trainee rater in the pre- and posttests. The VTCP increased the median number of trainee raters in agreement with expert consensus total scores (±2 points) from 4 (pretest) to 6.5 (posttest, P < 0.01).

Table 1.

Change in agreement for eight trainee raters with expert consensus score after completion of the video-training program.

	Pretest	Posttest	Score change
Group mean ± SD*	65 ± 10	87 ± 14	21 ± 7
Trainee rater
1	58	89	34
2	69	93	24
3	68	90	22
4	80	98	18
5	65	84	19
6	48	54	6
7	73	97	24
8	62	88	26

Total possible points = 112 (14 categories per rat, eight rats).

SD: standard deviation.

^*P < 0.0001, paired t-test

Figure 3.

Variability in trainee rater (Tr) assigned total neurologic scores in pre- and posttest as a function of expert consensus score (normalized as 0). The y-axis is the difference between the trainee rater assigned score and the expert consensus score for each rat. Shaded area indicates ±2 points from expert consensus scores. (a) Pre-test (b) Post-test.

Table 2 shows the frequency by which trainee raters agreed with expert consensus scores in at least six of eight rats for each category of the scoring system. In the pretest, there was no category for which six or more trainee raters agreed with the expert consensus score. In the posttest, six or more trainee raters agreed with the expert consensus score in 10 of 14 categories.

Table 2.

Number of trainee raters who achieved ≥75% agreement with the expert consensus neurologic score for each behavioral category.

	Number of trainee raters
	Pretest	Posttest
Spontaneous activity	3	7
Body symmetry	0	4
Gait	3	3
Front limb symmetry	1	5
Circling bench top	3	6
Circling holding tail	2	5
Hind limb placement	5	8
Vertical screen climb	1	6
Beam walking	4	7
Forelimb touch	2	7
Hind limb touch	4	7
Trunk touch	2	6
Vibrissae touch	3	6
Face touch	3	7

Total number of trainee raters = 8.

Total number of rats in each category = 8.

Inter-rater agreement is shown in Table 3. The number of categories with κ > 0.4 increased from four in the pretest to nine in the posttest. In six categories, the κ coefficient 95% CI did not overlap between the pre- and posttest. Notably, half of the positive educational effect was concentrated in sensory categories and there was no effect on assessment of gait.

Table 3.

κ coefficients for the eight trainee raters in each neurologic assessment category.

	Pretest		Posttest
	κ	95% CI	κ	95% CI
General status
Spontaneous activity	0.34	0.25–0.43	0.76	0.67–0.86*
Body symmetry	0.11	0.03–0.19	0.35	0.27–0.43*
Gait	0.24	0.15–0.32	0.11	0.01–0.21
Simple motor
Front limb symmetry	0.16	0.07–0.24	0.43	0.35–0.52*
Circling bench top	0.32	0.24–0.40	0.28	0.15–0.41
Circling holding tail	0.27	0.19–0.36	0.24	0.14–0.35
Hind limb placement	0.61	0.50–0.74	0.52	0.39–0.65
Complex motor
Vertical screen climbing	0.59	0.52–0.67	0.46	0.37–0.54
Beam walking	0.51	0.42–0.60	0.71	0.58–0.84
Sensory
Forelimb touch	0.17	0.06–0.27	0.67	0.57–0.77*
Hind limb touch	0.28	0.18–0.39	0.54	0.41–0.68*
Trunk touch	0.06	0.03–0.09	0.39	0.30–0.48*
Vibrissae touch	0.49	0.40–0.57	0.43	0.35–0.51
Face touch	0.22	0.12–0.32	0.43	0.30–0.56

^*= No overlap in 95% confidence interval (95% CI) between pre- and posttests.

Discussion

With the exception of a single report involving two observers,¹⁶ we could identify no efforts to define inter-rater agreement or investigate the value of a training program to improve agreement for neurologic scoring for multicenter experimental studies utilizing acute CNS injury models (e.g. ischemia, traumatic brain injury, subarachnoid hemorrhage). This unexplored issue may be an important source of variability in defining neurologic outcome. When three expert raters assessed post-MCAO function in the same animals, inter-rater variability was present for three different scoring systems (Figure 2, Supplemental Figure 2(a) to (c)). This indicates that clear definitions of the levels of deficit within each scoring category, beyond those offered by published criteria,^12–14 might be necessary to decrease ambiguity and improve reliability. For this purpose, we developed a VTCP for assessment of neurologic function in post-MCAO rats modeled after the training system employed for NIHSS use in human stroke patients.^6–9

Of necessity, a single scoring system needed to be selected for development of the VTCP. We evaluated three systems representing a span of complexity (Supplemental Table 1) and a priori determined that the system having the highest point estimate for the correlation with cerebral infarct size, as defined by three independent expert raters, would be selected. Although the systems did not differ statistically, the Yokoo et al. system¹⁴ had the highest point estimate for the correlation with infarct size and was therefore selected for use in our video training program.

Reliability improved after the VTCP. Using the Fleiss et al.¹⁵ definition of poor, moderate, and excellent agreement based on κ coefficients, agreement among the 14 categories in the current study improved from 10 poor and four moderate to five poor, eight moderate, and one excellent in the pre- and posttests, respectively, with statistically significant improvement in six of the 14 categories. The quality of posttest agreement is not substantially different from that reported in the original investigation of the NIHSS system used in human stroke patients where poor, moderate, and excellent agreement was observed in four, six, and three of the 13 possible categories, respectively.⁶ Video training for human stroke scoring served to improve NIHSS agreement beyond that observed in both our study and the initial NIHSS agreement investigation.⁷ Subsequent study of video-based NIHSS training could not reproduce this level of agreement for human stroke scoring; agreement again was similar to that observed in this rodent study.⁹ Despite that, video-based NIHSS training was deemed reliable.⁹

The VTCP required agreement with expert consensus neurologic scores in 11 of 14 (79%) categories for certification. All eight trainee raters met this criterion. In the pretest, 58% of the 14 categorical responses were in agreement with expert consensus scores, which increased to 78% in the posttest (Table 1), consistent with the VTCP goal. Trainee raters also improved the frequency of obtaining an expert consensus total score (±2 points) from 50% in the pretest to 77% in the posttest (Figure 3). These data reflect the usefulness of the VTCP for improving inter-observer reliability. It remains to be determined whether more intensive training could produce an even higher level of reliability. The interval and frequency of retraining required to maintain proficiency achieved through the VTCP is unknown. This may depend, in part, on how frequently the scoring system is employed by the individual rater. Future work would be of value in determining the frequency of recertification required to maintain competence.

The Yokoo system incorporates essentially all components of the Bederson and Garcia systems, plus additional sensory and motor tests, and is among the most comprehensive scoring systems we identified (Supplemental Table 4). This may explain, in part, its higher R value point estimate, although the 95% CIs overlapped for all three scoring systems. Hence, it cannot be concluded that the Yokoo system is superior to the other systems. Other untested systems (e.g. Supplemental Table 4) may be as good or better than that of Yokoo et al.¹⁴ Further, inspection of neurologic score data for animals with large infarcts (e.g. >250 mm³) indicates large scatter for all systems (Figure 2). Factors in addition to infarct size likely influenced neurologic scores in these animals.

To test generalizability of the VTCP, we post hoc exploited the fact that all elements of the Garcia system are included in the Yokoo system. Pre- and posttest performance of the eight trainees was reanalyzed using the same methodology as described for the Yokoo system using only those items included in the Garcia system. Individual trainee raters and group results from analysis of the Garcia system components are shown in Supplemental Table 5. The VTCP was again effective. The Garcia system employs six categories for scoring. With eight rats examined, a total of 48 scores were obtained for each trainee rater (in contrast to the Yokoo system, which included 14 scoring categories to provide 112 possible scores per trainee rater across eight animals). The eight trainee raters were concordant in 26 ± 6 pretest scores and 36 ± 7 posttest scores (P < 0.001). We also examined trainee rater scores relative to the expert consensus scores. Because the Garcia system had a total possible of 18 points per animal (versus 48 for the Yokoo system), we elected to compute the frequency that trainee raters achieved expert consensus scores within ±1 point (as opposed to ±2 points for the Yokoo system). In the pretest, trainee raters agreed with expert consensus total scores in 36 out of 64 possible scores (Supplemental Figure 3). In the posttest, the trainee raters agreed with the expert consensus total scores in 49 out of 64 possible scores. Posttest trainee rater scores were therefore in agreement with the expert consensus score 77% of the time, corresponding to the 77% agreement observed when analyzing the Yokoo system. However, the Garcia system did not achieve statistical significance (P = 0.08). Whether this lack of significance was due to inadequate statistical power, our admittedly arbitrary decision to require trainee scores to be within ±1 from expert consensus scores, or other reasons, is unknown.

There are potential limitations of this study. There were no control volunteers without exposure to the VTCP in which the pre- and posttests were given at the same times as the eight trainee raters. Exposure to the pretest may have offered some training that, in and of itself, could have contributed to improved posttest performance in the trainee raters. We were limited by availability of trainee raters (the NIHSS video training analysis included 112 raters).⁹ Larger samples of trainee raters may yield a different result. The role of hands-on training with live animals was not explored in this study. This is likely an important factor in learning to perform behavioral assessment; however, testing rating skills in live animals would be complex. Using the same animals for different trainee raters would likely result in differential behavioral responses over time. Potential interactions between individual trainee raters and rats (e.g. rough handling) could influence behavioral responses to various tests performed by subsequent trainee raters. Thus, to provide standardization, the pre- and posttests were performed on video-recorded rather than live rats. Further work may provide insight into whether hands-on training in live animals augments benefits observed from a VTCP and how this would be best implemented across institutions.

In conclusion, a VTCP program was developed and shown to improve inter-rater agreement in assessing neurologic function in rats subjected to MCAO when using two different scoring systems (Garcia et al.¹³ and Yokoo et al.¹⁴). Based on these findings, a VTCP program is likely to be valuable within laboratories and in multi-institutional preclinical trials of stroke therapeutics by improving inter-rater reliability and potentially decreasing the number of animals required for robust efficacy analysis.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Duke University Department of Anesthesiology.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Authors’ contributions

HT: Participated in experimental design, MCAO surgery, video-recording of animals, administration of the VTCP to human volunteers, data analysis and interpretation, and manuscript preparation.

MP: Experimental design, video-recording of animals and editing of the VTCP, and manuscript preparation.

HS: Experimental design, neurologic evaluation of MCAO rats to determine neurologic system used as the basis for the VTCP, and manuscript preparation.

MI: Experimental design, neurologic evaluation of MCAO rats to determine neurologic system used as the basis for the VTCP, and manuscript preparation.

REC: Experimental design, neurologic evaluation of MCAO rats to determine neurologic system used as the basis for the VTCP, and manuscript preparation.

LBG: Experimental design, data interpretation, and manuscript preparation.

DSW: Experimental design, data analysis and interpretation, and manuscript preparation.

Supplementary material

Supplementary material for this paper can be found at http://jcbfm.sagepub.com/content/by/supplemental-data.