Video Review and Clinical Screening of Suspected and Medically Diagnosed Concussions in the National Rugby League Women’s Premiership

Abstract

Background

The National Rugby League Women’s (NRLW) Premiership, created in 2018 as a professional Australian women’s rugby league competition, administers the Sport Concussion Assessment Tool-5th Edition (SCAT5) as part of the game day Head Injury Assessment (HIA). This study aimed to: (i) examine the frequency of video signs of potential concussion in players removed for an HIA, (ii) compare the in-game medical evaluation results between players with concussion and those returned to play, and (iii) report the games missed before returning to match play following concussion in NRLW players.

Methods

Medical personnel and sideline video operators tagged all head impact events requiring medical evaluation via a HIA during three NRLW Premiership seasons (2018–2020). If players were suspected of sustaining a concussion, they were removed from the field for a medical assessment, which included the SCAT5. Two independent reviewers retrospectively coded the video footage of each HIA. Inter-rater reliability of video signs was calculated using Cohen’s kappa (κ) and absolute percentage agreement. Video signs and medical evaluation findings were examined descriptively, and results were compared between concussed players and uninjured players who returned to the field.

Results

During the three NRLW Premiership seasons (2018–2020), there were 24 HIAs, with seven players medically diagnosed with concussion. As such, there was one HIA for every 0.88 matches and one medically diagnosed concussion every 3.00 matches. Inter-rater reliability between the two independent video reviewers was high, with the absolute percentage agreement for all concussions signs at 96.3%. On retrospective review, most NRLW players (n = 20, 83.3%) removed for a HIA did not have any consensus video signs, with three players (12.5%) displaying motor incoordination and two players (8.3%) having a blank or vacant look. On the SCAT5, concussed players had lower cognitive scores (M = 21.8 vs. 27.8, p = .01, d = 1.62) and reported more symptoms (M = 4.3 vs.1.0, p = .02, d = 1.09) with greater severity (M = 9.71 vs. 1.41, p = .02, d = 0.96) compared to those who were not diagnosed with a concussion. All players diagnosed with a concussion returned to play within two weeks of their injury.

Conclusion

Fewer than 20% of women removed from play for HIAs showed consensus video signs of possible concussion. Concussed players obtained lower cognitive tests scores and reported more symptoms, with greater severity, compared with uninjured players. Further studies with larger samples are needed to evaluate the video signs of concussion, with in-game SCAT5 scores, to improve the acute identification and management of concussions in the NRLW Premiership.

Key Points

• This is the first study to examine the observed video signs of concussion and the in-game SCAT5 scores in professional women’s rugby league players removed for a head injury assessment.

• Players who were medically diagnosed with a concussion had significantly lower scores on the cognitive component of the SCAT5, and they reported more symptoms with greater severity, compared to athletes who were not diagnosed with a concussion.

• All players diagnosed with concussion (n = 7) were permanently removed from the match. Five of the seven players returned to their team squads the following week. Two players missed the subsequent match but returned to their team squads two weeks later.

Background

Rugby league is a fast-paced, high intensity collision sport involving numerous tackles [1]. The tackle is the most injurious event in rugby league, with an inherent risk of concussion. The National Rugby League (NRL) is the top level of domestic club competition for men with 17 competing teams in Australia and New Zealand. A professional league for women was created in 2018, the National Rugby League Women’s (NRLW) Premiership. For the first three seasons, the NRLW Premiership had a shorter playing season (7 games) and fewer teams (n = 4), with an incidence of 11 concussions per 1,000 player match hours [2].

As part of the NRLW’s head injury assessment (HIA) processes, players are removed from the field for a medical evaluation if they are suspected of sustaining a concussion. This concussion management protocol has predetermined concussion signs and symptoms that necessitate the immediate and permanent removal of the player from the match (NRLW Category I signs and symptoms) or initiates an off-field medical assessment for concussion (NRLW Category II signs and symptoms) [3, 4]. The Sport Concussion Assessment Tool (SCAT), in its various iterations, has been adopted by the National Rugby League to aid in the HIA medical evaluation process. The Sport Concussion Assessment Tool-5th Edition (SCAT5) has on and off-field components including a symptom questionnaire, assessment of cognitive function via the Standardised Assessment of Concussion (SAC), and balance testing via the modified Balance Error Scoring System (mBESS) [5]. Following a suspected concussion, clinicians may compare post-injury SCAT5 scores to individual players’ pre-season baseline data or to published age, sex, and sport-matched normative reference values [6–8]. In the men’s professional rugby league competition, studies involving video signs, in-game SCAT scores, and the medical decision to permanently remove a player from the game or return them to the field have been published [4, 9, 10].

Studies focusing on the video signs of concussion and the clinical interpretation of game-day SCAT5 scores in women’s rugby league are needed. Past studies from our research group have examined the SCAT normative reference values [11] and the test-retest reliability of the SCAT5 in uninjured professional women’s rugby league players [12]. The three aims of the current study were as follows: (i) to examine the frequency of video signs of potential concussions in players removed for a HIA, (ii) compare the in-game medical evaluation results between players with concussion and those returned to play, and (iii) report the games missed before returning to match play following concussion in professional women rugby league players.

Methods

Participants

Participants were all individuals in the National Rugby League Women’s Premiership who underwent a HIA in the first three seasons (2018–2020), which included a total of 21 NRLW Premiership matches (i.e., 7 games each season). The NRLW medical bunker reviewed and tagged all head impact events live, using in-game video footage. The medical bunker could alert team medical staff on the sideline to review head impact events and assess players accordingly. Tagged video footage of head impact events were collated and included in a database. For example, if a player showed the video sign of no protective action during a fall, the player would be immediately removed from the field for a medical review. At the time of data collection, the NRLW was comprised of four teams competing over four rounds. The mean age of participants was 26.9 years (SD = 5.5, range = 17–41). All players, in accordance with the NRL and Rugby League Players Association Collective Bargaining Agreement, consented a priori to the collection of their deidentified injury data, for the purposes of research. In addition, the study was approved by The University of Newcastle’s Human Research Ethics Committee (H-2012-0344) and conducted in accordance with the standards of ethics outlined in the Declaration of Helsinki.

Procedures

Video analyses of all HIAs were conducted using the STATS Edge™ program, with full access to video provided by the NRLW. A minimum of two camera view videos (25 fps) were available for each HIA, in normal speed and in slow motion. There were 24 HIAs conducted during the 2018–2020 seasons, necessitating either a temporary interchange or permanent removal due to suspected concussion. The HIAs were retrospectively reviewed to determine the presence or absence of video signs of concussion. Two experienced analysts (SM and AG) independently coded all HIA events according to a previously defined matrix used in professional men’s rugby league [13]. The video analysts were able to view each incident multiple times and in slow motion to accurately code the video signs. The video signs were coded as ‘yes’ if observed, ‘no’ if not observed and ‘unknown’ if the footage was unclear to determine if they are present or absent. When there were differences in coding, the two analysts convened to establish a consensus. If there was disagreement between the two analysts, a third analyst (DT) was available to facilitate consensus. The video reviewers were blinded to the outcomes of the sideline assessment, SCAT5 findings, and medical diagnoses of concussion. The reviewers were, however, not blinded to the on-field signaling by the referee and trainers indicating that a player was being removed from play for a HIA.

Frequency of Observed Concussion Signs

The NRLW Category I signs, together with the six international consensus video signs [14], were retrospectively coded by the two independent reviewers (see Table 1). The observed video signs and the game-day HIA SCAT5 scores were subsequently collated. The six international consensus video signs included: lying motionless, impact seizure, tonic posturing, blank/vacant look, no protective action - floppy, and motor incoordination [14]. Three additional video signs, chosen a priori, that are neither consensus signs nor NRL signs and symptoms, were coded based on author interest and experience (i.e., clutching or shaking head, distressed appearance, and slowed movements).

Table 1.

Video signs used to identify possible sport-related concussion

Consensus video sign	Description
Lying motionless	Lying without purposeful movement on playing surface for > 2 s; does not appear to move or react purposefully, respond appropriately to the game situation (including teammates, opponents, umpires or medical staff).
Motor incoordination	Appears unsteady on feet (including loss of balance, struggling to get up/ falling over or staggering and stumbling), or in the upper limbs (fumbling). May occur when rises from the playing surface or when weightbearing.
Impact seizure	Involuntary (clonic) movements with periods of asymmetrical and irregular rhythmic jerking of axial or limb muscles.
Tonic posturing	Involuntary sustained contraction of one or more limbs, usually upper limbs, so that the limb is stiff despite the influence of gravity or player position. This can include cervical, axial, and lower limb muscles; can occur when the player is falling or on the ground.
No protective action - floppy	Falls to the playing surface with no protective action (i.e., not stretching out arms/ hands to break their fall) after a direct or indirect head impact. Loss of motor tone when falling to the ground. Where a player’s arms are being held by a tackling opponent this may be observed in the neck (cervical hypertonia).
Blank/Vacant Look	Player exhibits no facial expression or apparent emotion in response to their environment. This can include a lack of focus/ attention of vision and is best interpreted in reference to the athlete’s normal/expected facial expression

NRLW Category I Signs	Description
Lying motionless/loss of responsiveness (> 5 s)	Lying without purposeful movement on playing surface for > 2 s; does not appear to move or react purposefully, respond appropriately to the game situation (including teammates, opponents, umpires or medical staff).
Motor incoordination/balance disturbance/ataxia	Appears unsteady on feet (loss of balance, struggling to get up/falling over or staggering and stumbling), or in the upper limbs (fumbling).
Tonic posturing or impact seizure	Tonic posturing: Involuntary sustained contraction of one or more limbs, usually upper limbs, so that the limb is stiff despite the influence of gravity or player position. This can include cervical, axial, and lower limb muscles and has previously been described as “no protective action – stiff.” Impact seizure: Involuntary movements involving periods of asymmetric and irregular rhythmic jerking of axial or limb muscles.
No protective action	Falls to the playing surface without protecting themselves (i.e., not stretching out arms/hands to break fall) after a direct or indirect head knock. In the instance where a player’s arms are being held by a tackling opponent this may be observed in the neck (cervical hypertonia)
Dazed/blank or vacant stare	Player exhibits no facial expression or apparent emotion in response to the surrounding environment. This can include a lack of focus/attention of vision and best interpreted in reference to the athletes normal or expected facial expression.

Additional author initiated video signs	Description
Clutching or shaking head	Player is seen to be clutching her head or shaking it after the head impact.
Distressed appearance	Looks visibly distressed (e.g., grimacing), looks in pain or discomfort.
Slowed movements	Moving slowing back to play (e.g., taking short, shuffled steps instead of running into position), no urgency in their return back to their correct position.

Consensus video signs were adapted from Davis et al. (2019) [14]

Measures

As part of the HIA process, NRLW medical staff administered the SCAT5 during their clinical examination to determine the following outcomes: (i) removal and return to play, (ii) removal due to high suspicion of concussion, or (iii) removal for reasons other than concussion. The SCAT5, recommended by the Concussion in Sport Group, is a clinical assessment tool for suspected concussions in athletes aged 13 and over [15]. It has on- and off-field components, with the on-field assessment focusing on the identification of red flags and observable signs of concussion, sports specific orientation questions (Maddocks questions), the Glasgow Coma Scale, and a cervical spine assessment. The off-field components comprise a symptom evaluation, a cognitive screen, and neurological and balance assessments. Each individual symptom on the 22-item symptom checklist is self-rated on a scale from 0 (none) to 6 (severe). The individual symptom scores are added to form an overall severity score (range: 0/132). Based on World Rugby recommendations, the NRLW adopted a change to their symptom checklist, whereby “trouble falling asleep” was replaced with “excessive tiredness” in an in-game context [16]. The cognitive evaluation, known as the Standardized Assessment of Concussion (SAC) assesses orientation to time, immediate memory (ability to learn and recall a 5 or 10-word list), concentration (digit span backwards and months in reverse), and delayed memory (ability to recall the word list after completing all other subcomponents of the SCAT5). A neurological assessment and balance testing via the modified Balance Error Scoring System (mBESS) are also completed. Since the conclusion of data collection for this study, the SCAT has been updated to the Sport Concussion Assessment Tool 6th Edition (SCAT6) [5].

Statistical Analyses

For each video sign, if the two raters agreed that the sign was present on their assessment of the video footage, the video sign was noted as ‘consistent’ (both raters agreed) for that sign on that player. From this, the absolute percentage agreement was calculated for each of the video signs. Additionally, inter-rater reliability (IRR) analyses using Cohen’s kappa (κ) statistics was used to determine consistency between the two analysts for (i) the overall rating of all concussion video signs, and (ii) each of the nine individual signs [17] Cohen’s kappa (κ) is a frequently utilised statistical method for calculating IRR in study designs with two raters. Unlike the total percent agreement, Cohen’s κ provides a more accurate estimate of inter-rater agreement [18] by considering both the observed agreement and expected agreement due to chance [19]. Per McHugh’s interpretations of κ, agreement was categorized as almost perfect (> 0.90), strong (0.80–0.90), moderate (0.60–0.79), weak (0.40–0.59), minimal (0.21–0.39), or none (0-0.20) [19]. All analyses were performed using IBM SPSS Statistics V.29.0. When appropriate, statistical models used two-sided tests for significance at the 0.05 level. Descriptive post-injury characteristics (M, Md, SD, IQR) for concussed and uninjured athletes were calculated. Due to the skewed distributions, Mann-Whitney U tests were utilised to examine the differences in SCAT5 subcomponent scores in participants with and without medically diagnosed concussions. Due to the small sample size and limited power, adjustments were not made for multiple comparisons. Cohen’s d effect sizes were interpreted according to conventional methods: small (d = 0.2), medium (d = 0.5), and large (d = 0.8) [20]. A small number of players who were administered the 10-word list for the Immediate Memory, Delayed Memory, and Total Score of the SAC were removed from these specific cognitive analyses because the majority of players were given the 5-word list.

Results

During the 2018–2020 NRLW Premiership seasons, a total of 154 unique players participated in at least one NRLW match. There was a total of 24 HIAs conducted, with 20 unique players removed from play for HIAs (mean age = 26.9 years; SD = 5.5; range = 17–41 years). Four players were removed twice during the same season, with one player removed twice during a single game. In addition, three players were removed for HIAs during more than one season. Seven of these women were medically diagnosed with concussion and no player sustained more than one concussion within the same season. As such, there was one HIA for every 0.88 matches and one medically diagnosed concussion every 3.00 matches.

Inter-rater Reliability (IRR) of Video Signs on Researcher Review

When video signs were independently assessed by two researchers, the absolute percentage agreement for all concussions signs between the two raters was 96.3%. The IRR for each individual sign (see Table 2) where video signs present were as follows: motor incoordination κ = 0.76 (100% agreement, p < .001), blank or vacant stare κ = 0.85 (91.7% agreement, p < .001), clutching of the head κ = 0.61 (79.2% agreement, p < .001), distressed appearance κ = 0.69 (100% agreement, p < .001), and slowed movements κ = 0.69 (87% agreement, p < .001). The third rater was not required to facilitate a consensus in any of the 24 cases.

Table 2.

Inter-rater reliability calculations for the video signs of NRLW head impact assessments

	Frequency of endorsed sign		Absolute percentage agreement	κ	p
	Rater 1	Rater 2
Lying motionless	0	0	100%	NA	NA
Motor incoordination	2	2	100%	0.76	< 0.001
Impact seizure	0	0	100%	NA	NA
Tonic posturing	0	0	100%	NA	NA
No protective action	0	0	100%	NA	NA
Blank / Vacant stare	3	1	91.7%	0.85	< 0.001
Clutching of the head	15	12	79.2%	0.61	< 0.001
Distressed appearance	15	15	100%	0.69	< 0.001
Slowed movements	14	16	87.5%	0.69	< 0.001

κ kappa, NA not applicable, due to each variable being a constant because neither rater endorsed seeing this video sign; if the two raters agreed in their assessment in individual signs based on video footage, the sign was noted as consistent for that sign on that player. From this, the absolute percentage agreement was calculated for each of the video signs

Frequency of Observed Clinical Signs Based on Game-day Medical Reports

Based on game day notes from team medical staff, none of the 24 players had the Category I signs of impact seizure, tonic posturing, or motor incoordination. A player who had two Category I signs, lying motionless/loss of responsiveness and blank or vacant look, was subsequently diagnosed with concussion. A second player was noted to have no protective action but was not diagnosed with concussion. The observed Category I signs of impact seizure and tonic posturing were missing for two players.

Frequency of Observed Consensus Signs of Possible Concussion on Retrospective Video Review

Based on retrospective video review by our research team, the frequency of observed signs of concussion in players removed from play for an HIA was calculated and is shown in Table 3. Most players undergoing an HIA (n = 20, 83.3%) did not show any consensus video signs. Only four players had video signs of possible concussion identified by our research team. Three players (12.5%) had one consensus video sign (i.e., motor incoordination for two players and blank or vacant look for one player). A single player (4.2%) had two consensus video signs (motor incoordination and blank or vacant look).

Table 3.

Frequency of concussion signs on video analysis and concussion sign combinations for all HIAs

Retrospective video analysis	Yes		No		Missing or unknown
	f	%	f	%	f	%
Consensus – lying motionless	0	0	24	100.0	0	0
Consensus – motor incoordination	3	12.5	17	70.8	4	16.7
Consensus – impact seizure	0	0	24	100.0	0	0
Consensus – tonic posturing	0	0	24	100.0	0	0
Consensus – no protective action	0	0	24	100.0	0	0
Consensus – blank/vacant look	2	8.3	13	54.2	9	37.5
Clutching/shaking head	16	66.7	6	25.0	2	8.3
Distressed appearance	18	75.0	4	16.7	2	8.3
Slowed movements	17	70.8	2	8.3	5	20.8
Number of consensus video signs	f	%	-	-
0	20	83.3	-	-
1	3	12.5	-	-
2	1	4.2

f frequency, % percentage, Missing or Unknown: video signs were not observed but footage was unclear to determine if they were present or absent. Consensus signs per Davis et al. (2019) [14] include six signs: lying motionless, motor incoordination, impact seizure, tonic posturing, no protective action, blank/vacant look

The Category I signs of no protective action, lying motionless, impact seizure, and tonic posturing were not seen in any player. In total, motor incoordination was seen in three players (12.5%) and a blank or vacant look was observed in two other players (8.3%) on retrospective review. For the additional author-initiated video signs, 18 players (75.0%) had a distressed appearance, 17 players had slowed movements (70.8%), and 16 players (66.7%) were observed to be clutching or shaking their head.

SCAT5 Results

Of the 24 HIAs, 15 players were administered the SCAT5 with the 5-item word list while the other nine players received the 10-word memory list on the SAC. The nine players who received the 10-word memory list were excluded from some analyses (i.e., immediate memory, delayed memory, total SAC score) but their data was retained for other analyses of the SAC.

Table 4 shows the differences in SAC, mBESS, symptom severity, and total number of symptoms scores between players with and without medically diagnosed concussions. Players diagnosed with a concussion had significantly worse scores on the SAC (M = 21.8 vs. M = 27.8), with a large effect size (p = .01; d = 1.62). Total symptom severity scores were also significantly higher in concussed players (M = 9.71) than uninjured players (M = 1.41, p = .02, d = 0.96), with a large effect size. Concussed players, on average, endorsed more symptoms compared with players who were not concussed (M = 4.29 vs. M = 1.00, p = .02, d = 1.09). The concussed players made modestly more errors on the mBESS compared to those who were not injured (M = 3.14 vs. 1.94, p = .52, d = 0.57), with a medium effect size, but this difference was not statistically significant. Due to the small sample size, the statistical analyses were under-powered.

Table 4.

Clinical assessment scores stratified by concussion status

Medically diagnosed concussion	Yes			No
	n	M	SD	n	M	SD	U	p	d
SAC score	5	21.8	6.46	10	27.8	1.14	5.0	0.01	1.62
mBESS total score	7	3.14	3.24	17	1.94	1.48	49.5	0.52	0.57
Total symptom severity score	7	9.71	15.76	17	1.41	3.06	177.5	0.02	0.96
Total number of symptoms	7	4.29	4.68	17	1.00	2.06	25.5	0.02	1.09

Given the small sample size, the statistical analyses were significantly underpowered. SAC Standardized Assessment of Concussion; mBESS modified Balance Error Scoring System; day of injury U Mann-Whitney U tests, bold values include a statistically significant difference (p < .05); Effect size estimates were made using Cohen’s d

Symptom Reporting

For the total sample of all HIAs, participants reported a mean of 2.0 symptoms (SD = 3.3; Md = 0.5; IQR = 0-2.75), with a mean symptom severity score of 3.8 (SD = 9.3; Md = 0.5; IQR = 0-2.75). Individual symptom endorsements are summarized in Table 5. Twelve players (50.0%) reported one or more symptoms, and seven players (29.2%) reported two or more symptoms. Players diagnosed with concussion (n = 7) most commonly reported the symptoms of difficulty remembering (n = 3/7, 42.9%), feeling slowed down (n = 3/7, 42.9%), and feeling more emotional (n = 3/7, 42.9%), followed by headache, pressure in head, neck pain, not feeling right, difficulty concentrating, and drowsiness, which were each reported by 28.6% of the sample (n = 2/7). Players removed for a HIA but not diagnosed with concussion (n = 17) most commonly reported fatigue or low energy (n = 5/17, 29.4%), neck pain (n = 3/17, 17.6%), and dizziness (n = 2/17, 11.8%).

Table 5.

Proportion of NRLW players who endorsed each SCAT5 symptom during the head injury assessment

		Entire sample (n = 24)		Medically diagnosed concussion (n = 7)		No medically diagnosed concussion (n = 17)
		f	%	f	%	f	%
1	Headache	2	8.3	2	28.6	0	0
2	‘Pressure in head’	2	8.3	2	28.6	0	0
3	Neck pain	5	20.8	2	28.6	3	17.6
4	Nausea or vomiting	1	4.2	1	14.3	0	0
5	Dizziness	3	12.5	1	14.3	2	11.8
6	Blurred vision	1	4.2	1	14.3	0	0
7	Balance problems	1	4.2	1	14.3	0	0
8	Sensitivity to light	1	4.2	1	14.3	0	0
9	Sensitivity to noise	0	0	0	0	0	0
10	Feeling slowed down	4	16.7	3	42.9	1	5.9
11	Feeling like in fog	1	4.2	1	14.3	0	0
12	Don’t feel right	2	8.3	2	28.6	0	0
13	Difficulty concentrating	3	12.5	2	28.6	1	5.9
14	Difficulty remembering	4	16.7	3	42.9	1	5.9
15	Fatigue or low energy	6	25.0	1	14.3	5	29.4
16	Confusion	1	4.2	0	0	1	5.9
17	Drowsiness	3	12.5	2	28.6	1	5.9
18	Excessive tiredness	0	0	0	0	0	0
19	More emotional	4	16.7	3	42.9	1	5.9
20	Irritability	0	0	0	0	0	0
21	Sadness	2	8.3	1	14.3	1	5.9
22	Nervous or anxious	1	4.2	1	14.3	0	0
	One or more symptoms	12	50	6	85.7	6	35.3
	Two or more symptoms	7	29.2	4	57.1	3	26.7

f frequency, % percentage

Time to Medical Clearance to Return to Play

All players who were diagnosed with concussion (n = 7) were permanently removed from the game (i.e., none of these players returned to play in the same game). Five of the seven players with diagnosed concussions returned to their team squads for their next match, one week following injury (i.e., these five players did not miss a subsequent match). Two players missed the subsequent match but returned to their team squads two weeks later. Fifteen uninjured players removed for a HIA returned to the field on that day. Two players who were removed from the field for another injury (i.e., not a concussion) did not return to the match on that day but returned to their squads for subsequent matches.

Discussion

This is the first study to examine the frequency of potential video signs of concussion and the acute game day SCAT5 scores in professional women rugby league players removed from play for a HIA. Most players did not show any consensus video signs of concussion on retrospective video review. The in-game SCAT5 results revealed that players diagnosed with concussion had a greater number of symptoms with a higher symptom severity score, and significantly worse scores on the SAC, compared with uninjured players. In addition, concussed players made more balance errors on the mBESS (medium effect size), but this finding was not statistically significant likely due to the small sample size.

Inter-rater Reliability of Video Signs

Inter-rater reliability for the two expert reviewers was high (96.3% absolute agreement). Similar to previous studies examining inter-rater reliability of video signs in the NRL, motor incoordination and blank or vacant stare were the two signs that were the more difficult to agree upon [21, 22]. In the Australian Football League (AFL), there was also high inter-rater reliability for all video signs except for blank or vacant look [23]. This is likely due to the limited footage that enables a clear view of a player’s face to accurately and consistently code the video sign of blank or vacant look, and the subjectivity associated with conceptualizing this sign. This results in a high number of cases with “unknown” or “missing” data. To date, this limitation appears to be a consistent finding across different contact and collision sports, potentially limiting the clinical utility of this video sign [10, 14, 23, 24].

Observed Signs of Possible Concussion

In the present study, the only two consensus video signs that were observed were motor incoordination (12.5%) and blank or vacant look (8.3%). In the NRL (men’s) competition, the video sign motor incoordination also was uncommon, seen in 91 of 727 clinical assessments (12.5%) [25]. Studies in the Australian Football League (AFL) and National Hockey League (NHL) have found that players who displayed the video sign of blank or vacant look are subsequently diagnosed with concussion [23, 24]. However, this video sign occurred in only 1–6% of concussions in the NHL and AFL, because it is inherently difficult to identify, requiring a close-up view of a player’s face [23, 24]. That sign was also very uncommon in the present study. On retrospective review, the international consensus video signs/NRLW Category I signs of no protective action, impact seizures, and tonic posturing were not observed in our sample.

The Category I NRLW signs of lying motionless and blank or vacant stare were observed by game-day medical staff in two players, triggering mandatory removal from the game. Of course, one cannot rely on video signs alone to identify suspected concussions. In an NHL study, video signs were not observed in approximately 53% of concussions [24]. Similarly, in the National Football League (NFL), 26% of concussed players did not show video signs of concussion [26]. In the present study, as seen in Table 6, only one of the seven players (14.3%) diagnosed with a concussion showed a consensus video sign of possible concussion on retrospective review.

Table 6.

SCAT5 scores, medical staff documentation, and retrospective video review for 24 head injury assessments (HIAs) conducted with professional women rugby league players

		SCAT5 scores									Retrospective consensus video signs / and post-game team medical staff*						3 additional retrospective video signs
HIA	Conc. Dx.	Madd. Quest.	Symp. Count	Symp. Sev.	SAC Total	Orien.	Imm. Mem.	Concen.	Del. Mem.	mBESS total	Lying motionless	Motor incoord.	Impact seizure	Tonic posturing	No protective action	Blank / vacant look	Clutching, shaking head	Distressed appearance	Slowed movements
1	No	5	0	0	36	5	26	5	0	4	No	No	No	No	No	UK/No	Yes	Yes	Yes
2	No	5	0	0	28	5	14	4	5	0	No	No	No	No	No	Yes/No	No	Yes	Yes
3	No	5	0	0	29	5	15	4	5	3	No	No	No	No	No	UK/No	Yes	Yes	Yes
4	No	5	0	0	29	5	14	5	5	4	No	UK/No	No	No	No	UK/No	Yes	Yes	Yes
5	No	5	0	0	43	5	24	5	9	2	No	No	Yes/No	No	No	UK/No	Yes	Yes	Yes
6	No	5	1	1	27	5	14	4	4	3	No	No	No	No	No	UK/No	Yes	UK	UK
7	No	5	1	2	26	5	14	4	3	1	No	No	No	No	No	UK/No	Yes	Yes	UK
8	No	5	3	5	29	5	15	4	5	0	No	No	No	No	Yes	No	Yes	Yes	Yes
9	No	5	0	0	45	5	26	5	9	1	No	No	No	No	No	No	Yes	Yes	Yes
10	No	5	1	1	26	5	15	3	3	2	No	No	No	No	No	No	Yes	Yes	Yes
11	No	5	0	0	28	5	15	5	3	2	No	No	No	No	No	No	Yes	Yes	Yes
12	No	5	8	12	39	4	23	4	8	1	No	No	UK	UK	No	UK/No	Yes	Yes	UK
13	No	5	0	0	28	5	15	3	5	5	No	UK/No	No	No	No	No	No	UK	Yes
14	No	5	3	3	48	5	30	5	8	2	No	No	No	No	No	No	No	No	No
15	No	5	0	0	44	5	25	5	9	0	No	No	No	No	No	No	Yes	Yes	Yes
16	No	5	0	0	28	5	14	4	5	2	No	No	No	No	No	No	Yes	No	Yes
17	Yes	5	1	1	22	4	15	3	0	0	No	UK/No	No	No	No	No	UK	Yes	UK
18	Yes	5	2	2	36	5	22	4	5	9	No	Yes/No	UK	UK	No	UK/No	UK	Yes	Yes
19	Yes	5	1	2	11	2	7	1	1	0	No	UK/No	No	No	No	UK/No	Yes	Yes	UK
20	Yes	5	13	44	25	5	14	4	2	5	No	No	No	No	No	No	No	Yes	Yes
21	No	5	0	0	46	5	27	5	9	1	No	No	No	No	No	No	No	No	No
22	Yes	5	7	13	45	5	26	4	10	1	No	No	No	No	No	No	Yes	Yes	Yes
23	Yes	5	0	0	28	4	15	5	4	3	No/Yes	Yes/No	No	No	No	Yes	Yes	Yes	Yes
24	Yes	5	6	6	23	3	14	4	2	4	No	No	No	No	No	No	No	No	Yes

*When there are two responses in a box, the first represents retrospective video review (“Retrospective Consensus Video Signs”), and the second represents post-game administrative documentation from the team medical staff (“Post-Game Team Medical Staff”). Conc. Dx. Concussion Diagnosis, Madd. Quest. Maddocks Questions, Symp. Symptom, Sev. Severity, Orien. Orientation, Imm. Mem. Immediate Memory, Concen. Concentration, Del. Mem. Delayed Memory, Incoord. Incoordination, and UK = Unknown. * indicates the athlete received the 10-item word list. Missing data indicates that task was not administered to the athlete

SCAT5 Symptom Ratings, Balance, and Cognition

During the HIA, in the total sample, most professional women rugby league players in this study endorsed few symptoms. The current study revealed that on average, uninjured women rugby league players endorsed 1–2 symptoms, with the most common symptoms being fatigue or low energy (29.4%), followed by neck pain (17.6%) and dizziness (11.8%). In contrast, nearly half of the concussed players reported the symptoms of difficulty remembering, feeling slowed down, and feeling more emotional. A third of the concussed players reported headache, pressure in the head, neck pain, not feeling right, difficulty concentrating, and drowsiness.

Situational factors may play a role in symptom reporting on the sideline during a HIA. When being evaluated on the sideline, it is possible that some players who might be experiencing a concussion, might be focused on whether, or not, they are experiencing “concussion symptoms.” In contrast, for players (women or men) who are confident that they are not experiencing a concussion, they might simply report “any symptom” without considering, before reporting, whether the symptom might be due to a concussion [27, 28]. Past studies in laboratory settings illustrate that symptoms can vary with exertion levels. Following high intensity exertion, previous studies have shown that uninjured participants have differences in their symptom scores. In one study, after an intense bike protocol, women reported changes in headache (22.2% better, 11.1% worse), fatigue (26.6% better, 51.0% worse), noise sensitivity (26.6% better; 0% worse), light sensitivity (15.5% better; 4.4% worse), and mental fog (15.1% better; 13.3% worse) [29]. In another study, after a high intensity fitness test completed by 37 uninjured rugby and wrestling athletes of both sexes, SCAT5 total symptoms and symptom severity scores were elevated at the 0–10 min mark following exertion (d ≥ 0.64) but not at 20 min following exertion (d = 0.06–0.10) [30]. These findings may help contextualize the symptom reporting in the NRLW athletes who underwent a HIA but were not diagnosed with a concussion, suggesting that exertion and individual perception may influence sideline symptom presentation.

Concussed players in the current sample had significantly worse scores on the SAC compared with uninjured players. Our results are similar to previous studies in high school and collegiate men’s football, where the mean SAC scores were lower than uninjured players immediately following a concussion, with a large effect size [31–36]. Therefore, SAC scores obtained during a sideline evaluation, in combination with symptom reports and balance testing, can assist clinicians in formulating a diagnosis and measuring injury severity, or determining if the player can return to the game [37]. With regard to balance, although a medium effect size (Cohen’s d = 0.57) was seen between the groups, this difference was not statistically significant, likely due to the very small sample size.

Clinical Implications Regarding Return to Play

None of the concussed players returned to play on the day of injury. Fifteen uninjured players returned to play following the HIA and two additional players were removed from the field for other injuries. Some of the on-field signs of concussion may be transient and completely resolve before the player can be assessed by the medical team [38]. In addition, symptoms may evolve over time and worsen in the hours and days following a concussion such that a player who was relatively asymptomatic during her in-game HIA may have worse symptoms the following day. Reliance on symptom reports alone is problematic with some players not wishing to disclose their symptoms for fear of being ruled out of competition, letting their team down, or not considering their symptoms worthy of medical attention [27, 28, 39]. Studies have shown that players who are not identified as having a concussion and continue to play are at increased risk a prolonged recovery, compared with those who are identified promptly and are removed from play [35, 36]. This emphasises the importance of early recognition, accurate reporting of symptoms, and the immediate removal of players for further evaluation via concussion management processes.

Limitations

This study has several important limitations. First, the results are influenced and limited by the small sample size and associated power considerations. Second, certain video signs such as blank/vacant look could not be coded on retrospective review, due to the lack of close-up footage of players’ faces or obstruction. Third, although the independent video reviewers conducting the retrospective analysis were blinded to the results of the sideline assessment and medical diagnoses of concussion, they were not blinded to the use of the HIA (i.e., the on-field signalling by a match official identifying an interchange). Fourth, during the in-game review of video signs, there is a potential for confirmatory bias between the observed Consensus or Category I signs with a medical diagnosis of concussion. For example, if the medical team or sideline operators tag a head impact event with a player displaying the video sign of motor incoordination, the player is removed for a HIA. In addition, SCAT5 scores are not independent of the medical diagnosis of concussion, as they form part of the head injury assessment. The NRLW concussion policy mandates that a player who shows any Consensus video signs or Category I signs must be immediately and permanently removed from the field. This process of removing a player from the field may also affect the likelihood of a player being diagnosed with a concussion. Fifth, certain video signs cannot be confirmed beyond doubt on video review. It is likely that a sports trainer or medic (i.e., physician or physiotherapist) on the sideline may be better able to determine the presence or absence of certain signs (e.g., blank or vacant look) when closely monitoring a player. The retrospective video analysis was also conducted without the pressures of in-game decision making and therefore our results may not be representative of live video identification and tagging of head impact events. Sixth, the inter-rater reliability in this study may be inflated because most cases did not show the video signs of concussion and both reviewers agreed that the video signs were absent. Seventh, most players were administered the 5-word list while a small number of players received the 10-word list on the SAC, which reduced sample size comparisons for the SAC. Eighth, the results of this study may not be generalisable to youth or community women’s rugby league players or other football codes. Finally, the current study was conducted during the first three seasons of the NRLW. The NRLW Premiership has subsequently expanded in the number of teams, players, and officials. It is not known whether any of the results in this study would be different if the study were conducted now after the league has evolved.

Summary and Conclusions

This is the first study to consider video signs of concussion in the National Rugby League Women’s Premiership. Video signs of concussion were rated consistently by two independent raters, and fewer than 20% of HIAs showed consensus video signs. Only one of the seven players who was medically diagnosed with a concussion showed a video sign of possible concussion. Players who were medically diagnosed with concussion had significantly lower scores on the SAC and they reported more symptoms with increased severity, compared to uninjured players. Concussed players made more balance errors on the mBESS, but this finding was not statistically significant due to being underpowered. Care must be taken when interpreting this study’s results due to the small sample size and associated power considerations. Future studies with larger samples are required to better understand the association between observable signs of concussion and sideline-evaluation of SCAT5 scores. Considering other factors like exertion, injury biomechanics (measured using impact sensors or instrumented mouthguard technology), clinical outcomes, and time to return to play may also offer insights in improving the identification of suspected concussions and the interpretation of sideline medical evaluations. Given the limited research on women elite and professional athletes, further investigation is needed to understand whether there are sex differences in video signs of injury and acute day-of-injury sideline symptom reporting.

Abbreviations

CI

Confidence interval

f

Frequency

HIA

Head injury assessment

IQR

Interquartile range

IRR

Inter-rater reliability

κ

Kappa

M

Mean

mBESS

modified balance error scoring system

Md

Median

n

Sample size

%

Percentage

NA

Not applicable

NFL

National football league

NHL

National hockey league

SAC

Standardized assessment of concussion

SCAT

Sport concussion assessment tool

SCAT5

Sport concussion assessment tool-5th edition

SD

Standard deviation

SEM

Standard error of measurement

NRL

National Rugby League

NRLW

National Rugby League Women’s

Author Contributors

All authors conceptualized the study. SM and AG collected data and evaluated the video footage of each case. SM conducted the literature review and prepared the first draft of the manuscript. DT conducted the statistical analyses. All authors interpreted the data. All authors reviewed and contributed to the production of the final version of manuscript and approved submission of the final version.

Funding

SM was supported by funding from the Australian Government Research Training Scheme for her PhD studies. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. No entity was involved in study design; the collection, analysis, and interpretation of data; the writing of the report; or the decision to submit the manuscript for publication. No honorarium, grant, or other form of payment was given to anyone to produce the manuscript. AJG and GLI acknowledge philanthropic support from the National Rugby League. GLI acknowledges philanthropic support from ImPACT Applications, Inc., the Mooney-Reed Charitable Foundation, Boston Bolts, and the Schoen Adams Research Institute at Spaulding Rehabilitation.

Data Availability

The data supporting the findings of this study are included in this paper. Additional data relating to individual athletes is not available, and further enquiries can be directed to the corresponding author (Andrew Gardner, andrew.gardner@sydney.edu.au).

Declarations

Ethics Approval and Consent to Participate

All players, in accordance with the NRL and Rugby League Players Association Collective Bargaining Agreement, consented a priori to the collection of their deidentified injury data, for the purposes of research. The study was approved by The University of Newcastle’s Human Research Ethics Committee (H-2012-0344) and conducted in accordance with the standards of ethics outlined in the Declaration of Helsinki.

Competing Interests

SM has no declarations to disclose. DT serves as a scientific advisor for HitIQ. He previously consulted for REACT Neuro, Inc. He has a consulting practice in forensic neuropsychology, including expert testimony, involving individuals who have sustained mild TBIs (including former athletes). He received research funding from Amgen, Inc. and Football Research Inc. GLI serves or has served as a scientific advisor for NanoDX^®, Sway Operations, LLC, and Highmark, Inc. He has a clinical and consulting practice in forensic neuropsychology, including expert testimony, involving individuals who have sustained mild TBIs. He has received past research support or funding from several test publishing companies, including ImPACT Applications, Inc., CNS Vital Signs, and Psychological Assessment Resources (PAR, Inc.). He has received research funding as a principal investigator from the National Football League, and subcontract grant funding as a collaborator from the Harvard Integrated Program to Protect and Improve the Health of National Football League Players Association Members. AJG has a clinical practice in neuropsychology involving individuals who have sustained sport-related concussion (including current and former athletes). He has been a contracted concussion consultant to Rugby Australia. He is a member of the World Rugby Concussion Working Group, and a member of the Australian Football League Concussion Scientific Advisory Committee. He has received travel funding or been reimbursed by professional sporting bodies, and commercial organisations for discussing or presenting sport-related concussion research at meetings, scientific conferences, workshops, and symposiums. Previous grant funding includes the NSW Sporting Injuries Committee, the Brain Foundation (Australia), an Australian-American Fulbright Commission Postdoctoral Award, a Hunter New England Local Health District, Research, Innovation and Partnerships Health Research & Translation Centre and Clinical Research Fellowship Scheme, and the Hunter Medical Research Institute (HMRI), supported by Jennie Thomas, and the HMRI, supported by Anne Greaves. AJG is supported by a National Health and Medical Research Council (NHMRC) Investigator Grant. He acknowledges unrestricted philanthropic support from the Nick Tooth Foundation and the National Rugby League for research in former elite level rugby league players. None of the above entities were involved in the study design, analysis, interpretation, the writing of this article, or the decision to submit it for publication.