Test-Retest Reliability and Agreement Between In-Person and Video Assessment of Facial Mimetic Function Using the eFACE Facial Grading System

Abstract

Importance

Use of a robust high-resolution instrument for grading of facial symmetry would enhance reporting the outcomes of facial reanimation interventions. The eFACE is one such tool.

Objective

To determine test-retest reliability of the eFACE tool over time and agreement between eFACE assessments made in person vs those made using video of facial mimetic function.

Design, Setting, and Participants

A prospective observational study was conducted among 75 consecutive patients with varying degrees of facial palsy who presented between July 1 and December 31, 2014, to an academic tertiary referral hospital. Facial symmetry of all patients was graded in person and via standardized photographic and video documentation of facial mimetic function at the initial visit. Three months after initial presentation, eFACE scores were reassessed by the same raters using the videos of facial mimetic function documented at the initial visit.

Main Outcomes and Measures

Individual and subset eFACE scores assessed by 2 facial reanimation surgeons.

Results

Among the 75 patients in the study (mean [SD] age, 48.18 [16.60] years; 30 men and 45 women), agreement between in-person and video assessments of facial function using the eFACE scale was excellent (static subset score: mean difference, 0.19; 95% CI, −1.51 to 1.88; P = .83; intraclass correlation coefficient [ICC], 0.85; 95% CI, 0.80 to 0.89; dynamic subset score: mean difference, −0.51; 95% CI, −1.72 to 0.71; P = .41; ICC, 0.96; 95% CI, 0.94 to 0.97; synkinesis subset score: mean difference, −1.14; 95% CI, −2.87 to 0.59; P = .20; ICC, 0.90; 95% CI, 0.86 to 0.93; and composite score: mean difference, −0.41; 95% CI, −1.30 to 0.47; P = .36; ICC, 0.94; 95% CI, 0.92 to 0.96). Agreement between repeated eFACE assessments of video of facial function was excellent (static subset score: ICC, 0.93; 95% CI, 0.87 to 0.96; dynamic subset score: ICC, 0.88; 95% CI, 0.80 to 0.94; synkinesis subset score: ICC, 0.92; 95% CI, 0.86 to 0.96; and composite score: ICC, 0.96; 95% CI, 0.93 to 0.98).

Conclusions and Relevance

Strong agreement exists between eFACE scores of facial function assessed in person and from video recordings. Test-retest reliability of eFACE scores is high. The eFACE is a reliable instrument for high-resolution assessment of facial mimetic function.

Level of Evidence

NA.

This study determines test-retest reliability of the eFACE tool over time and agreement between eFACE assessments made in person vs those made using video of facial mimetic function.

Key Points

Question

What is the test-retest reliability of the eFACE facial grading system, and do eFACE assessments made in person agree with those made using video?

Findings

This study demonstrated strong agreement between eFACE scores of facial function assessed in person and those assessed from video recordings. It also demonstrated high test-retest reliability of eFACE scores over time.

Meaning

The results of this study further validate the eFACE instrument as a high-resolution assessment of facial mimetic function and support its use in grading of facial outcomes using video documentation of mimetic facial function.

Introduction

Robust evaluation of facial nerve function is essential to the management of facial paralysis. Although several scales for grading of facial symmetry exist, there is lack of consensus among clinicians regarding which scale should be routinely used. The ideal facial nerve grading instrument would be easily accessible, readily administered, demonstrate high intrarater and interrater reliability, and provide enough resolution to capture subtle yet clinically relevant zonal changes over time and differences between patients. Although the most commonly reported scale in use among members of the Sir Charles Bell Society is the House-Brackmann Scale, it was designed to categorize long-term outcomes following vestibular schwannoma resection. Similar to the House-Brackmann Scale, most existing facial grading scales lack the resolution necessary to report zonal changes in response to targeted reanimation procedures, a critical requirement for accurate communication and comparison of outcomes among health care professionals.

To improve reporting of facial reanimation outcomes, a computer application for clinician grading of facial symmetry in unilateral facial palsy (eFACE) was recently developed. The eFACE consists of a graphical user interface in which the user scores 15 visual analog scales designed to assess zonal facial symmetry at rest, volitional facial movements, and synkinesis (eFigure in the Supplement). Each scale represents the complete spectrum of possible dysfunction for that particular facial parameter, in which a score of 0 represents the worst possible dysfunction or asymmetry and a score of 100 represents completely normal function or symmetry with respect to the contralateral hemi-face. The application links to a database, where results over time are readily displayed in graphical format and subset scores are automatically calculated (eFigure in the Supplement). Although the eFACE has demonstrated high interrater reliability for in-person assessments, to our knowledge, its test-retest reliability has yet to be established. The purpose of this study was to determine test-retest reliability of the eFACE tool over time and between in-person and video assessments of facial mimetic function. Establishment of reliability between in-person and video assessments is important, as management and outcomes reporting in this field necessitate repeated evaluation of facial nerve function over time, which is facilitated by rigorous photographic and video documentation.

Methods

Seventy-five consecutive new patients with unilateral facial palsy of varying duration and severity presenting to our Facial Nerve Center between July 1 and December 31, 2014, were enrolled. Facial symmetry was graded in person at the initial visit for all patients by 2 experienced facial nerve surgeons (C.A.B. and T.A.H.); in addition, all patients underwent standardized photographic and video documentation of facial mimetic function on initial presentation.

For in-person assessment, the patient is seated with his or her arms crossed 3 feet from the examiner, with the examiner’s head positioned directly in line with the patient’s head. The eFACE instrument application is opened on an iPad (Apple Inc). With the patient in repose, the visual analog scales for the 4 static parameters are scored by using touchscreen sliders. Dynamic parameters are scored by asking the patient to perform 7 facial movements (brow elevation, gentle eye closure, full-effort eye closure, gentle smile, full-effort smile, pucker, and lower lip movement with /e/ as in see). Often, the examiner will place 2 fingers on the midline brow to prevent or mitigate the upward pull of the weak side by the healthy side. The patient is then asked to perform the 7 facial movements again while the 4 synkinesis parameters are graded (Video). Video documentation is performed on the same visit; the patient is seated in a chair with a headrest to minimize movement during video acquisition. The patient is asked to perform the 7 standard facial expressions detailed above, in addition to articulation of plosives (eg, “happy birthday”). The video is stored in the patient’s electronic health record for future reference.

Video. Scoring the eFACE Facial Grading Tool.

Download video file ^{(239.8MB, mp4)}

Scoring the eFACE facial grading tool during physical examination of facial function.

Following a minimal delay of 3 months, eFACE scores were reassessed by the same raters (C.A.B. and T.A.H.) using the videos of facial mimetic function documented at the patient’s initial visit. The first frame of the video with the patient in repose was used to score the static parameters of the eFACE. The dynamic and synkinesis eFACE parameters were then scored by playing the video. To assess intrarater reliability over time, videos from 25 patients representing a wide range of facial palsy severity on initial presentation were rated by the same rater on 5 occasions, each separated by at least 2 weeks. The Massachusetts Eye and Ear Infirmary Institutional Review Board approved this study. Patients provided written informed consent.

Statistical Analysis

Agreement between in-person and video-assessed eFACE scores and between raters was assessed using the intraclass correlation coefficient (ICC) (2-way, random, single measures) and paired t tests. Agreement was defined as poor for ICC values less than 0.40, fair for values between 0.40 and 0.59, good for values between 0.60 and 0.74, and excellent for values of 0.75 and above. Bland-Altman analysis, which plots the difference between 2 measures against the mean of the 2 measures, was performed to determine whether linear regression would demonstrate a change in the difference between scores for increasing severity of dysfunction for each eFACE parameter. Agreement between repeated assessments over time was assessed using the ICC. All tests were performed using IBM SPSS Statistics, version 22 (IBM Corp), with a 2-tailed significance level of 0.05.

Results

In-Person vs Video-Assessed eFACE Scores

The mean (SD) age of the patients (30 men and 45 women) was 48.18 (16.60) years. Agreement between in-person and video assessments for eFACE subset scores was excellent; the static subset ICC was 0.85 (95% CI, 0.80-0.89), dynamic subset ICC was 0.96 (95% CI, 0.94-0.97), synkinesis subset ICC was 0.90 (95% CI, 0.86-0.93), and composite score ICC was 0.94 (95% CI, 0.92-0.96) (Table 1). Agreement between individual eFACE parameter scores as assessed in person as opposed to those assessed using video documentation was excellent for 11 parameters, good for 3 parameters (symmetry of nasolabial fold depth at rest [ICC, 0.72; 95% CI, 0.63-0.79], nasolabial fold orientation with smile [ICC, 0.74; 95% CI, 0.66-0.81], and severity of platysma synkinesis [ICC, 0.67; 95% CI, 0.56-0.75]), and fair for 1 parameter (symmetry of the brow at rest [ICC, 0.54; 95% CI, 0.41-0.64]), with most parameters demonstrating an ICC of 0.80 or higher (Table 1). Interrater agreement for subset and individual eFACE parameter scores assessed in person was excellent, with all parameters demonstrating an ICC of 0.75 or higher (Table 1). Interrater agreement for individual eFACE parameter scores assessed using video documentation was lower than scores for parameters assessed in person; while 9 video-assessed eFACE parameter scores demonstrated excellent agreement between observers, 6 parameters demonstrated only good agreement (symmetry of the brow at rest [ICC, 0.67; 95% CI, 0.52-0.78], palpebral fissure width at rest [ICC, 0.66; 95% CI, 0.51-0.78], oral commissure at rest [ICC, 0.71; 95% CI, 0.58-0.81], symmetry of nasolabial fold orientation with smile [ICC, 0.67; 95% CI, 0.51-0.78], lower lip movement with /e/ [ICC, 0.71; 95% CI, 0.57-0.81], and severity of mentalis synkinesis [ICC, 0.69; 95% CI, 0.54-0.79]) (Table 1). Interrater agreement was excellent for eFACE subset and composite scores assessed using video documentation; the static subset ICC was 0.84 (95% CI, 0.75-0.90), the dynamic subset ICC was 0.96 (95% CI, 0.93-0.97), the synkinesis subset ICC was 0.88 (95% CI, 0.82-0.93), and the composite score ICC was 0.94 (95% CI, 0.91-0.96) (Table 1).

Table 1. Intrarater and Interrater Agreement Between In-Person and Video Assessments of Individual eFACE Parameter and Subset Scores.

eFACE Score	Intraclass Correlation Coefficient (95% CI)a
	Within Observers (In-Person vs Video Assessment)	Between Observers (In-Person Assessment)	Between Observers (Video Assessment)
Static subset	0.85 (0.80-0.89)	0.89 (0.83-0.93)	0.84 (0.75-0.90)
Brow at rest	0.54 (0.41-0.64)	0.77 (0.66-0.85)	0.67 (0.52-0.78)
Palpebral fissure at rest	0.77 (0.69-0.83)	0.75 (0.64-0.84)	0.66 (0.51-0.78)
NLF depth at rest	0.72 (0.63-0.79)	0.80 (0.69-0.87)	0.79 (0.68-0.86)
OC at rest	0.79 (0.72-0.84)	0.91 (0.85-0.94)	0.71 (0.58-0.81)
Dynamic subset	0.96 (0.94-0.97)	0.98 (0.97-0.99)	0.96 (0.93-0.97)
Brow elevation	0.94 (0.91-0.95)	0.97 (0.95-0.98)	0.95 (0.92-0.97)
Light-effort eye closure	0.81 (0.74-0.86)	0.97 (0.95-0.98)	0.84 (0.76-0.90)
Full-effort eye closure	0.86 (0.80-0.89)	0.93 (0.90-0.96)	0.90 (0.85-0.94)
NLF depth with smile	0.83 (0.77-0.87)	0.90 (0.83-0.93)	0.84 (0.76-0.90)
OC movement with smile	0.90 (0.86-0.93)	0.95 (0.92-0.97)	0.83 (0.75-0.89)
NLF orientation with smile	0.74 (0.66-0.81)	0.90 (0.83-0.93)	0.67 (0.51-0.78)
Lower lip movement with /e/	0.76 (0.68-0.82)	0.89 (0.84-0.93)	0.71 (0.57-0.81)
Synkinesis subset	0.90 (0.86-0.93)	0.94 (0.92-0.97)	0.88 (0.82-0.93)
Ocular synkinesis	0.87 (0.82-0.90)	0.95 (0.92-0.97)	0.88 (0.81-0.92)
Midfacial synkinesis	0.81 (0.74-0.86)	0.92 (0.87-0.95)	0.83 (0.74-0.89)
Mentalis synkinesis	0.80 (0.74-0.85)	0.75 (0.64-0.84)	0.69 (0.54-0.79)
Platysma synkinesis	0.67 (0.56-0.75)	0.89 (0.83-0.93)	0.77 (0.66-0.85)
Composite	0.94 (0.92-0.96)	0.96 (0.94-0.98)	0.94 (0.91-0.96)

Abbreviations: NLF, nasolabial fold; OC, oral commissure.

^aTwo-way random, single measures intraclass correlation coefficient. P < .001 for all.

Five eFACE parameters demonstrated statistically significant mean differences between in-person and video-assessed scores in paired t tests (nasolabial fold depth at rest, −3.28 [95% CI, −6.04 to −0.52]; brow elevation, 3.38 [95% CI, −0.88 to 5.87]; light-effort eye closure, −3.32 [95% CI, −5.62 to −1.01]; full-effort eye closure, −1.83 [95% CI, −3.19 to 0.47]; and midfacial synkinesis, −4.83 [95% CI, −7.48 to −2.20]); however, such differences were negligible (Table 2). Bland-Altman analysis demonstrated a significant tendency for video-assessed scores to increasingly overestimate the severity of oral commissure asymmetry at rest and mentalis synkinesis and a significant tendency to increasingly underestimate the severity of impairment of light- and full-effort eye closure for increasing severity of dysfunction (Table 2 and Figure 1).

Table 2. Mean Difference and Bland-Altman Analysis Between In-Person and Video Assessments for Individual eFACE Parameter and Subset Scores.

eFACE score	Mean Difference (95% CI)	P Valuea	Bland-Altman Analysis
			95% Limits of Agreement	r2	Slope of Fit Line (95% CI)	P Valuea
Static subset	0.19 (−1.51 to 1.88)	.83	−20.10 to 20.47	0.005	−0.04 (−0.13 to 0.05)	.39
Brow at rest	3.14 (−0.30 to 6.57)	.07	−38.00 to 44.28	0.006	0.08 (−0.09 to 0.26)	.36
Palpebral fissure at rest	−1.21 (−3.91 to 1.49)	.38	−33.53 to 31.11	0.030	0.12 (0.01 to 0.23)	.04
NLF depth at rest	−3.28 (−6.04 to −0.52)	.02	−36.33 to 29.77	0.008	0.07 (−0.06 to 0.21)	.28
OC at rest	2.30 (−0.24 to 4.85)	.08	−28.18 to 32.78	0.055	−0.16 (−0.27 to 0.05)	.004
Dynamic subset	−0.51 (−1.72 to 0.71)	.41	−15.04 to 14.02	0.004	0.02 (−0.03 to 0.07)	.44
Brow elevation	3.38 (−0.88 to 5.87)	.008	−26.54 to 33.29	<0.001	0.001 (−0.06 to 0.06)	.97
Light-effort eye closure	−3.32 (−5.62 to −1.01)	.005	−30.94 to 24.31	0.104	0.21 (0.11 to 0.31)	<.001
Full-effort eye closure	−1.83 (−3.19 to 0.47)	.009	−18.16 to 14.50	0.139	0.21 (0.12 to 0.29)	<.001
NLF depth with smile	−0.77 (−4.00 to 2.47)	.64	−39.52 to 37.99	0.013	−0.07 (−0.17 to 0.03)	.17
OC movement with smile	0.74 (−4.02 to 3.93)	.98	−34.73 to 36.21	0.006	−0.04 (−0.11 to 0.04)	.36
NLF orientation with smile	−0.04 (−2.22 to 3.70)	.62	−47.67 to 47.59	0.003	−0.04 (−0.17 to 0.08)	.49
Lower lip movement with /e/	−1.19 (−4.95 to 2.57)	.53	−46.22 to 43.84	<0.001	0.001 (−0.12 to 0.12)	.98
Synkinesis subset	−1.14 (−2.87 to 0.59)	.20	−21.87 to 19.59	0.003	−0.02 (−0.10 to 0.05)	.54
Ocular synkinesis	1.22 (−1.53 to 3.97)	.38	−31.73 to 34.17	0.008	−0.05 (−0.14 to 0.04)	.29
Midfacial synkinesis	−4.83 (−7.48 to −2.20)	<.001	−36.47 to 26.82	0.018	0.09 (−0.02 to 0.20)	.11
Mentalis synkinesis	2.09 (−0.53 to 4.71)	.12	−29.25 to 33.43	0.090	−0.20 (−0.30 to −0.10)	<.001
Platysma synkinesis	−2.87 (−6.31 to 0.57)	.10	−44.05 to 38.31	0.001	−0.030 (−0.18 to 0.12)	.70
Composite	−0.41 (−1.30 to 0.47)	.36	−11.00 to 10.18	0.002	0.01 (−0.04 to 0.07)	.63

Abbreviations: NLF, nasolabial fold; OC, oral commissure.

^aBold items are significant at P = .05.

Figure 1. Bland-Altman Analysis.

Bland-Altman graphs are plotted for the 4 eFACE parameters that demonstrated a significant trend for increasing differences between in-person and video-assessed scores. A, Symmetry of the oral commissure at rest. B, Impairment of light-effort eye closure. C, Impairment of full-effort eye closure. D, Severity of mentalis synkinesis. The solid reference lines indicate the mean and 95% limits of agreement between the 2 measures. The dotted line represents the line of best fit (linear regression), while dashed lines represent 95% CIs on the line of best fit.

Intrarater Reliability Over Time

Excellent agreement was seen for 13 of 15 eFACE parameters between repeated assessments over time; agreement for eFACE subset and composite scores was also excellent (static subset ICC, 0.93 [95% CI, 0.87-0.96]; dynamic subset ICC, 0.88 [95% CI, 0.80-0.94]; synkinesis subset ICC, 0.92 [95% CI, 0.86-0.96]; and composite score ICC, 0.96 [95% CI, 0.93-0.98]) (Table 3). Agreement was good for repeated video assessment of the asymmetry of nasolabial fold orientation with smile (ICC = 0.67 [95% CI, 0.51-0.81]) and severity of midfacial synkinesis (ICC, 0.72 [95% CI, 0.58-0.85]) (Table 3).

Table 3. Agreement Between Repeated Assessments Over Time for Individual eFACE Parameter and Subset Scores From Videos of Facial Mimetic Function.

eFACE Score	Intraclass Correlation Coefficient (95% CI)a
Static subset	0.93 (0.87-0.96)
Brow at rest	0.87 (0.79-0.93)
Palpebral fissure at rest	0.81 (0.70-0.90)
NLF depth at rest	0.86 (0.77-0.93)
OC at rest	0.90 (0.84-0.95)
Dynamic subset	0.88 (0.80-0.94)
Brow elevation	0.92 (0.86-0.96)
Light-effort closure	0.92 (0.87-0.96)
Full-effort closure	0.93 (0.87-0.96)
NLF depth with smile	0.90 (0.84-0.95)
OC movement with smile	0.87 (0.78-0.93)
NLF orientation with smile	0.67 (0.51-0.81)
Lower lip movement with /e/	0.91 (0.84-0.95)
Synkinesis subset	0.92 (0.86-0.96)
Ocular synkinesis	0.85 (0.76-0.92)
Midfacial synkinesis	0.72 (0.58-0.85)
Mentalis synkinesis	0.94 (0.90-0.97)
Platysma synkinesis	0.86 (0.78-0.93)
Composite	0.96 (0.93-0.98)

Abbreviations: NLF, nasolabial fold; OC, oral commissure.

^aTwo-way random, single measures intraclass correlation coefficient. P < .001 for all.

Discussion

Management of facial paralysis necessitates serial evaluations of facial mimetic function over time. The ideal grading tool for facial nerve function would provide enough resolution to differentiate clinically important zonal changes in response to spontaneous recovery and targeted surgical interventions, while simultaneously demonstrating high interrater and intrarater reliability over time. The ideal grading tool would demonstrate strong agreement between scores assessed in person and those assessed using video documented in the electronic health record. Furthermore, demonstration of high intrarater and interrater reliability for scores assessed using video documentation would permit for use of the tool by blinded observers in the comparison of treatment outcomes within and across centers.

Our study has demonstrated excellent agreement between eFACE subset and composite scores assessed in person vs those assessed from video of facial function. In addition, 11 of the 15 individual eFACE parameters demonstrated excellent agreement between in-person and video assessments. The weakest agreement was seen between in-person and video-assessed scores for symmetry of the brow position at rest (ICC, 0.54). This latter finding is likely the result of the tendency of many individuals to unknowingly elevate the brow, which results in marked brow asymmetry in the setting of hemifacial palsy (Figure 2). During in-person assessments, spurious resting brow asymmetry is readily corrected by reminding patients to fully relax their contralateral healthy brow; such a verbal cue was not offered during routine video documentation of facial function at our center. The weaker agreement seen between in-person and video assessments of eFACE scores for the symmetry of the nasolabial fold depth at rest (ICC, 0.72), nasolabial fold orientation with smile (ICC, 0.74), and severity of platysma synkinesis (ICC, 0.67) may be the result of poor visualization of these zones from video footage owing to suboptimal lighting conditions resulting in shadowing.

Figure 2. Spurious Resting Brow Asymmetry.

A, This patient has left-sided facial paralysis and is unknowingly elevating the contralateral (right) healthy brow at rest, exaggerating resting brow asymmetry. B, Conscious effort at brow elevation raises the right brow further.

Although their agreement was good or excellent, 5 eFACE parameters demonstrated significant differences in mean scores between in-person and video assessments. Video assessments tended to underestimate the severity of resting nasolabial fold depth asymmetry, the degree of midfacial synkinesis, and the impairment in light- and full-effort eye closure and tended to overestimate the severity of brow elevation impairment compared with in-person assessment. However, the mean difference between scores in all 5 cases was negligible, ranging from 1.8% to 4.8%. Similarly, while Bland-Altman analysis demonstrated significantly increasing discrepancies toward the more dysfunctional end of the spectrum between in-person and video-assessed scores for 4 eFACE parameters, such trends were slight and had minimal effect on the mean difference between scores across the spectrum.

This study has demonstrated excellent agreement between repeated assessments using video documentation for most eFACE parameters, indicating that intrarater reliability of this tool over time is high. The slightly weaker agreement seen between repeated assessments of the severity of nasolabial fold orientation asymmetry with smile and midfacial synkinesis might again be explained by shadowing in the region of the nasolabial fold owing to suboptimal lighting conditions during video documentation of facial function, which would render robust assessment of this region more challenging. Furthermore, loss of stereopsis that is inherent to standard video recording techniques might further explain the disagreement between in-person and video assessment of the aforementioned parameters.

Limitations

This study was limited in that both reviewers were already experts on the use of the eFACE scale. Although video documentation of facial function in electronic health records facilitates future blinded assessment of facial function outcomes, its realization in clinical practice is resource intensive.

Conclusions

Excellent agreement exists between in-person and video assessments of facial mimetic function for most eFACE parameters and for all subset scores. Optimization of lighting conditions and verbal cues during video acquisition might further improve agreement. The eFACE facial grading tool demonstrates excellent agreement between repeated measures over time for all subset scores and most individual parameters.

Supplement.

eFigure. The eFACE Facial Grading System