Clinician Facial Expression of Emotion Corresponds with Patient Mindset

Abstract

Background

Mental health has a notable and perhaps underappreciated relationship with symptom intensity related to musculoskeletal pathophysiology. Tools for increasing awareness of mental health opportunities may help musculoskeletal specialists identify and address psychological distress and unhealthy misconceptions with greater confidence. One such type of technology—software that identifies emotions by analyzing facial expressions—could be developed as a clinician-awareness tool. A first step in this endeavor is to conduct a pilot study to assess the ability to measure patient mental health through specialist facial expressions.

Questions/purposes

(1) Does quantification of clinician emotion using facial recognition software correlate with patient psychological distress and unhealthy misconceptions? (2) Is there a correlation between clinician facial expressions of emotions and a validated measure of the quality of the patient-clinician relationship?

Methods

In a cross-sectional pilot study, between April 2019 and July 2019, we made video recordings of the clinician’s face during 34 initial musculoskeletal specialist outpatient evaluations. There were 16 men and 18 women, all fluent and literate in English, with a mean age of 43 ± 15 years. Enrollment was performed according to available personnel, equipment, and room availability. We did not track declines, but there were only a few. Video recordings were analyzed using facial-emotional recognition software, measuring the proportion of time spent by clinicians expressing measured emotions during a consultation. After the visit, patients completed a demographic questionnaire and measures of health anxiety (the Short Health Anxiety Inventory), fear of painful movement (the Tampa Scale for Kinesiophobia), catastrophic or worst-case thinking about pain (the Pain Catastrophizing Scale), symptoms of depression (the Patient Health Questionnaire), and the patient’s perception of the quality of their relationship with the clinician (Patient-Doctor Relationship Questionnaire).

Results

Clinician facial expressions consistent with happiness were associated with less patient health anxiety (r = -0.59; p < 0.001) and less catastrophic thinking (r = -0.37; p = 0.03). Lower levels of clinician expressions consistent with sadness were associated with less health anxiety (r = 0.36; p = 0.04), fewer symptoms of generalized anxiety (r = 0.36; p = 0.03), and less catastrophic thinking (r = 0.33; p = 0.05). Less time expressing anger was associated with greater health anxiety (r = -0.37; p = 0.03), greater symptoms of anxiety (r = -0.46; p < 0.01), more catastrophic thinking (r = -0.38; p = 0.03), and greater symptoms of depression (r = -0.42; p = 0.01). More time expressing surprise was associated with less health anxiety (r = -0.44; p < 0.01) and symptoms of depression (r = -0.52; p < 0.01). More time expressing fear was associated with less kinesiophobia (r = -0.35; p = 0.04). More time expressing disgust was associated with less catastrophic thinking (r = -0.37; p = 0.03) and less health anxiety (GAD-2; r = -0.42; p = 0.02) and symptoms of depression (r = -0.44; p < 0.01). There was no association between a clinicians’ facial expression of emotions and patient experience with patient-clinician interactions.

Conclusion

The ability to measure a patient’s mindset on the clinician’s face confirms that clinicians are registering the psychological aspects of illness, whether they are consciously aware of them or not. Future research involving larger cohorts of patients, mapping clinician-patient interactions during consultation, and more sophisticated capture of nonverbal and verbal cues, including a broader range of emotional expressions, may help translate this innovation from the research setting to clinical practice.

Clinical Relevance

Tools for measuring emotion through facial recognition could be used to train clinicians to become aware of the psychological aspects of health and to coach clinicians on effective communication strategies both for gentle reorientation of common misconceptions as well as for appropriate and timely diagnosis and treatment of psychological distress.

Introduction

There is strong and consistent evidence that mental health accounts for a notable percentage of the variation in symptom intensity related to musculoskeletal pathophysiology. We foresee two components of beginning to address mental health opportunities in musculoskeletal specialty care: First, the ability of clinicians to recognize the verbal and nonverbal signs of distress or unhealthy misconceptions (cognitive bias) and understanding these signs as such. Subsequently, the application of this understanding toward effective clinician communication strategies that build trust and guide people toward a more accurate regard of their symptoms, further helping individuals make a connection between symptoms, thoughts, and emotions.

To aid with implementation of the first component (noticing and appreciating), it may be useful to develop tools that measure signals from physiological monitoring that correlate with misconceptions and psychological distress and use them to foster awareness of the known verbal and nonverbal signs of misconceptions and distress. Several such physiological measuring tools (such as, facial EMG measures or skin conductance) are used in research regarding empathy [30]. Measures of clinician physiology and facial expression can capture unconscious noticing of patient misconceptions and distress. These tools could be used to nurture conscious awareness of this noticing, which can increase specialist confidence identifying distress and misconceptions and acting on them.

Facial recognition software can accurately detect specific facial expressions and is validated to codify emotions (Fig. 1) [2, 7, 16, 43]. Such technologies are based on the foundational work by Ekman [12], that our faces reflect our emotions. Applying this technology in the healthcare setting, we have an opportunity to study a clinician’s facial expressions more systematically as a gauge of his or her emotions during consultations and then determine whether these measures relate to patient mental health and the quality of the patient-clinician relationship. In other words, can a clinician’s face register awareness of patient distress, misconceptions, and experience of the patient-clinician relationship? If there is a relationship, then facial recognition tools could be used to help train clinicians to notice and become conscious of verbal and nonverbal signs of distress and cognitive bias. Along with increased awareness, surgeons can be trained in effective communication strategies for guiding patients away from a limited focus on physical tests and treatments toward a more comprehensive approach to health.

Fig. 1.

Facial recognition software can accurately detect specific facial expressions, as shown in this mock simulation. Printed with permission of Noldus. A color image accompanies the online version of this article.

We therefore undertook a pilot study of facial recognition of clinician emotions to address the following questions: (1) Does quantification of clinician emotion using facial recognition software correlate with patient psychological distress and unhealthy misconceptions? (2) Is there a correlation between clinician facial expressions of emotions and a validated measure of the quality of the patient-clinician relationship?

Patients and Methods

Study Design and Setting

Between April 2019 and July 2019, we performed a cross-sectional study involving 34 patients attending an orthopaedic outpatient clinic within a university health system (Dell Medical School at the University of Texas at Austin) providing integrated multidisciplinary care for upper extremity conditions and sports-related problems. We invited patients to participate and obtained written informed consent for video recording (of the clinician) during their visit. This visit was usually a first encounter with an orthopaedic surgeon, resident, advanced nurse practitioner, or physician assistant who were from different socioeconomic backgrounds. Clinicians were aware that consultations would be video recorded with a camera directed at them rather than the patient during the visit.

We recruited patients pragmatically based on the inclusion criteria (new patient status, age between 18 and 89 years, English fluency and literacy, and the ability to provide informed consent), in a clinic that allowed video recording during clinic schedules that had light-to-moderate patient volumes, to ensure adequate set-up time for video equipment. This way, we were able to ensure the study did not interfere with clinical flow. Although we did not specifically collect data on patients who declined to participate, in fact few declined. The datasets were complete for all patients who participated.

Thirty-four patients were included with a mean age of 43 years ± 15 years. Forty-seven percent (16 of 34) of patients were men, and 50% (17 of 34) were white (Table 1).

Table 1.

Patient and clinical characteristics (n = 34)

Variable	Value
Age in years	43 ± 15
Men	47% (16)
Race/ethnicity
White	50% (17)
Non-white	50% (17)
Marital status
Married or unmarried couple	35% (12)
Divorced, separated, widowed, or single	65% (22)
Work status
Employed	47% (16)
Student, retired, disabled, or unemployed	53% (18)
Got annual doctor's appointment	50% (17)
SHAI-5	11 (9-12)
TSK-4	32 (26-47)
PCS-4	4 (2-8)
PHQ-4
PHQ-2	2 (2-4)
GAD-2	3 (2-5)
PDRQ-9	44 (40-45)

Continuous variables are presented as mean ± SD or the median (interquartile range); discrete variables are presented as the percentage (number); PHQ-4 consists of the two-item Patient Health Questionnaire (PHQ-2) and the Generalized Anxiety Disorder (GAD-2) questionnaire; SHAI = Short Health Anxiety Inventory; TSK = Tampa Scale for Kinesiophobia; PCS = Pain Catastrophizing Scale; PHQ = Patient Health Questionnaire; PDRQ = Patient-Doctor Relationship Questionnaire.

Video Recording and Experimental Process

We placed a high-resolution camera with video recording capability on a side table in the clinic room and positioned it toward the clinician’s face. Research assistants not involved in patient care monitored the video remotely in a nearby private office space, started and stopped the recording, ensured adequate video capture of the clinician’s face through most of the consultation, and were available throughout the consultation to resolve any technical issues. The entire one-to-one consultation was assessed (that is, the total time spent with the clinician).

At the end of the visit, we asked patients to complete a set of questionnaires linked to the recording by a code to protect patient identity and personal information. We completed all questionnaires on a digital tablet device via REDCap (Vanderbilt University), a secure internet-based application for building and managing online surveys and databases [18].

Measures

We analyzed the facial expressions of providers using facial recognition software (Noldus) [31]. The software has undergone substantial testing for validity and reliability using the current reference standard in the field, the Facial Action Coding System [35], and validation against large facial imaging databases (Radbound Faces Database; The Amsterdam Dynamic Facial Expression Set; Warsaw Set of Emotional Facial Expression Pictures) as well as other commercially available classifiers of facial recognition [11, 25, 31, 38, 40]. We engaged extensively with the research and development team at the company on the topic of how such a tool might be applied in orthopaedic specialty practice in the run-up to performing this study and closely reviewed internal and independent external documentation of the performance of this software alluding to its accuracy [31].

We used the proportion of time the clinician spent exhibiting the following measurable expressions of emotion during the consultation: neutral, happy, sad, angry, surprised, scared, and disgusted (Table 2). The degree to which a specific facial expression is exhibited at any given point in time is graded on an algorithm trained using intensity values annotated by human experts. The software assigns each expression a continuous value for intensity between 0 and 100 within fractions of a second. Intensity level 0 means the expression is absent while 100 means it is fully present. A report is provided by the software, using a computer algorithm accounting for intensity, of the proportion of time (given as a percentage) that each of these emotions is expressed over a given period (that is, the duration of the consultation). Based on data detailing software development and external validation by independent researchers, described earlier, we were satisfied as to the accuracy of the outputs.

Table 2.

Proportion of time clinicians spent with each facial expression (n = 34 patients)

Expression	Median (range)
Neutral	63 (16-78)
Happy	7 (2-15)
Sad	4 (1-11)
Angry	2 (0-5)
Surprised	8 (1-25)
Scared	6 (1-16)
Disgusted	7 (2-22)

Report is provided by the software, using a computer algorithm accounting for intensity, of the proportion of time (a percentage) that each of these emotions is expressed over a given time period; that is, the duration of the consultation. Intensity level 0 means the expression is absent while 100 means it is fully present.

After the visit, we invited patients to complete a set of questionnaires in the following order: a demographic questionnaire consisting of age, gender, race or ethnicity, marital status, work status, whether or not the patient typically attends an annual physical (to assess whether they keep an annual doctor’s appointment), the Short Health Anxiety Inventory (SHAI-5), the Tampa Scale for Kinesiophobia (TSK-4), the Pain Catastrophizing Scale (PCS-4), the Patient Health Questionnaire (PHQ-4) (which consists of the two-item Patient Health Questionnaire [PHQ-2] and the Generalized Anxiety Disorder [GAD-2] questionnaire), and the Patient-Doctor Relationship Questionnaire (PDRQ-9).

The SHAI-5 is a five-question measure of heightened illness concern [1, 35]. The total score is between 0 and 15, with higher scores representing a greater tendency to believe that one has a serious problem, despite evidence to the contrary [35].

The TSK-4 quantifies the magnitude of fear of movement (or kinesiophobia) [23]. The questionnaire is answered on a 4-point Likert scale, from 1 = strongly disagree to 4 = strongly agree. The total score ranges from 17 to 68, with higher scores indicating a higher magnitude of fear of movement. A minimum clinically important difference greater than 6 was achieved for this scale after lumbar fusion during a four-week motor and cognitive-behavioral rehabilitation program [29].

The PCS-4 measures catastrophic thinking: less-effective cognitive coping strategies (such as worst-case thinking or helplessness) in response to nociception [5]. The questionnaire contains four questions that are answered on a 4-point Likert scale, from 0 = not at all to 4 = all the time. The total scores range from 0 to 16, with higher scores indicating more catastrophic thinking.

The PHQ-4 measures depression and anxiety using a four-item questionnaire combining the PHQ-2 and the GAD-2, respectively [27]. The four questions are answered on a 4-point Likert-scale, from 1 = not at all to 4 = nearly every day. The score of both questionnaires range from 2 to 8, with higher scores indicating a greater symptom burden.

The PDRQ-9 quantifies the patient-clinician relationship from the perspective of the patient, with the focus on the helping attitude of the doctor [42]. The questionnaire contains nine questions, which are answered on a 5-point Likert-scale from 1 = disagree to 5 = totally agree. The total scores range from 9 to 45, with higher scores indicating better relationships.

These validated measures of distress (depression, anxiety) and unhealthy misconceptions (catastrophizing, heightened illness concern, fear of painful movement) are strong, consistent, and useful correlates of symptom intensity and magnitude of limitations [8, 21]

Primary and Secondary Study Outcomes

Our primary goal was to assess the association of measures of psychological distress (PHQ-4) and unhealthy misconceptions (SHAI-5, TSK-4, PCS-4) with the proportion of time a clinician spends during the one-to-one consultation with the patient expressing basic emotions (measured by emotion intensity using the software).

Our secondary goal was to assess the association of patient experience (PDRQ-9) with clinical facial expression of basic emotions.

Ethical Approval

Ethical approval for this study was obtained from the University of Texas at Austin, Austin, TX, USA (protocol number 2018-12-0030).

Statistical Analysis

Before performing our analysis, we set a target of 30 patients a priori for this pilot study based on pragmatic considerations, including the massive volume of recorded data that needed to be processed to arrive at usable data points, our clinic throughput, time and logistics to set up and perform the recordings and analysis, availability and access to the software package, and configuration and clinical workflows of our musculoskeletal team. Our thinking was that if we could not measure an association with 30 patients, then the relationship is probably not strong enough to be useful. The distributions of continuous variables and assumptions concerning normality were assessed to determine the appropriateness of the statistical tests. None of the variables were normally distributed. Descriptive statistics are presented as the median (interquartile range), and discrete variables are presented as the percentage (number). Bivariate analyses were conducted to test the association between each expression and the other variables. We used the Spearman correlation coefficient for continuous variables and the Kruskal-Wallis test for dichotomous and categorical variables. We planned to include factors with significant associations in the bivariate analysis in a multivariable linear regression model to assess patient factors independently associated with the proportion of time a physician spent on different categories of facial expressions. However, we did not create multivariable models because only the psychological questionnaires were significant. For this reason, we did not need to attend to potential multicollinearity, a common phenomenon that becomes apparent as measures of distress and bias are known to correlate within one another.

Results

Association Between Psychological Distress and Physicians’ Expressed Emotions

The proportion of time a clinician had a neutral facial expression during a visit was not associated with patient factors. More time with a happy expression was associated with less patient health anxiety (measured with the SHAI-5; r = -0.59; p < 0.001) (Table 3) and less catastrophic thinking (PCS-4; r = -0.37; p = 0.03). Less time with a sad expression was associated with employed status compared with student, retired, disabled, or unemployed status (p = 0.05) (Table 3) and less health anxiety (SHAI-5: r = 0.36; p = 0.04), fewer symptoms of generalized anxiety (GAD-2: r = 0.36; p = 0.03), and less catastrophic thinking (PCS-4: r = 0.33; p = 0.05). Less time with an angry expression was associated with greater health anxiety (SHAI-5: r = -0.37; p = 0.03), greater symptoms of anxiety (GAD-2: r = -0.46; p < 0.01) (Table 3), more catastrophizing (PCS-4: r = -0.38; p = 0.03), and greater symptoms of depression (PHQ-2: r = -0.42; p = 0.01). More time with a surprised facial expression was associated with fewer symptoms of anxiety (GAD-2: r = -0.44; p < 0.01) and depression (PHQ-2: r; -0.52; p < 0.01) (Table 3). More time with a scared facial expression was associated with a lower magnitude of kinesiophobia (TSK-4: r = -0.35; p = 0.04) (Table 3). More time with a disgusted facial expression was associated with less catastrophizing (PCS-4: r = -0.37; p = 0.03) (Table 3) and fewer symptoms of anxiety (GAD-2: r = -0.42; p = 0.02) and depression (PHQ-2: r = -0.44; p < 0.01).

Table 3.

Bivariate analyses of factors associated with proportion of time the clinician spent exhibiting each facial expression

Variables	Neutral	p value	Happy	p value	Sad	p value	Angry	p value	Surprised	p value	Scared	p value	Disgusted	p value
Age in years	-0.25	0.15	0.21	0.22	-0.17	0.33	0.12	0.50	0.50	0.003	-0.32	0.06	0.03	0.86
Gender
Men	63 (59-71)	0.55	6 (4-10)	0.60	4 (2-7)	0.89	3 (2-4)	0.11	10 (4-15)	0.41	5 (4-7)	0.29	8 (6-13)	0.55
Women	63 (58-67)		7 (4-10)		4 (2-7)		2 (1-3)		8 (5-9)		7 (3-11)		7 (5-9)
Race/ethnicity
White	63 (54-67)	0.55	7 (4-10)	0.92	5 (2-7)	0.14	3 (1-3)	0.74	7 (3-13)	0.19	6 (3-7)	0.88	9 (6-14)	0.11
Non-white	63 (60-68)		7 (3-9)		3 (2-5)		2 (1-4)		9 (7-11)		6 (4-10)		6 (5-9)
Marital status
Married or unmarried couple	66 (62-69)	0.15	7 (4-11)	0.77	4 (2-6)	0.97	3 (1-3)	0.65	9 (5-11)	0.84	6 (2-7)	0.40	7 (6-9)	0.43
Divorced, separated, widowed, or single	62 (55-65)		7 (3-9)		3 (2-7)		2 [1-4]		8 (5-15)		6 (4-10)		8 (6-14)
Work status
Employed	64 (59-67)	0.86	7 (5-9)	0.64	2 (2-4)	0.047	3 (2-4)	0.28	9 (6-15)	0.15	6 (3-7)	0.62	9 (5-9)	0.90
Student, retired, disabled, or unemployed	63 (60-69)		5 (3-12)		5 (2-7)		2 (1-3)		7 (4-11)		6 (4-8)		7 (6-13)
SHAI-5	0.20	0.27	-0.59	< 0.001	0.36	0.04	-0.37	0.03	-0.24	0.17	-0.03	0.85	-0.27	0.12
TSK-4	0.11	0.54	-0.08	0.66	0.20	0.25	-0.05	0.78	0.22	0.21	-0.35	0.04	-0.18	0.32
PCS-4	0.11	0.54	-0.37	0.03	0.33	0.054	-0.38	0.03	0.11	0.53	-0.15	0.41	-0.37	0.03
PHQ-4
PHQ-2	0.27	0.13	-0.06	0.72	0.22	0.21	-0.42	0.01	-0.52	0.002	0.09	0.62	-0.44	0.01
GAD-2	0.02	0.91	-0.16	0.37	0.36	0.03	-0.46	0.006	-0.44	0.010	0.00	> 0.99	-0.42	0.02
PDRQ-9	-0.14	0.45	-0.09	0.62	0.11	0.53	0.00	0.99	0.11	0.54	0.09	0.62	0.15	0.40

Variables are expressed as the median (interquartile range) or Spearman correlation; bold face indicates statistical significance; SHAI= Short Health Anxiety Inventory; TSK = Tampa Scale for Kinesiophobia; PCS = Pain Catastrophizing Scale; PHQ = Patient Health Questionnaire; GAD = Generalized Anxiety Disorder; PDRQ = Patient-Doctor Relationship Questionnaire.

Association Between Clinician Facial Expressions of Emotions and Measurements of Clinician-patient Relationship

There was no association between clinician facial expressions of emotions and patient experience, based on patient-clinician interactions as measured with the Patient-Doctor Relationship Questionnaire (Table 3).

Discussion

We aimed to understand whether there was an association between the clinician’s facial expressions of emotions and patient distress, unhealthy misconceptions, and experience of their interaction with the clinician. If there is a relationship, then facial recognition technology could be developed as a tool to help train clinicians in effective communication strategies and help them become more aware of verbal and nonverbal signs of distress, cognitive bias, and patient experience of the relationship with the healthcare professional. We found that clinician expressions of happiness, sadness, anger, surprise, fear, and even disgust are associated with psychological distress (symptoms of anxiety and depression) and degrees of faulty thinking (including cognitive biases such as worst-case thinking and fear of painful movement). The ability to measure one person’s mindset on another person’s face using facial recognition technology offers a powerful coaching opportunity for clinicians. This finding could help develop training applications that increase awareness among clinicians that communication effectiveness is a skill that benefits from practice as much as technical skills such as physical examination, injection, and surgery.

Limitations

There are several limitations to this study. First, this study involved only 34 patients and six clinicians providing multidisciplinary care at a university health system. Although the study population is diverse with even numbers of men and women clinicians, the relatively low patient volume, only one nonwhite clinician, and single institution may limit generalizability. Nevertheless, despite the small sample size, the correlations between clinician facial expression and patient mindset were surprisingly strong, fueling our hunch that we are measuring human traits that will prove consistent across various contexts.

Second, we invited anyone present on an enrollment interval in the office and achieved diversity because our patient population is diverse. Further studies are merited involving more complex design, such as purposive sampling of larger patient and clinician cohorts, and analysis discerning the influence of a wider range of contextual factors (such as geography) and patient-clinician phenotypes. Such studies may clarify associations between clinician emotions with patient age and work status as well as the counterintuitive inverse associations found between clinician expressions of fear with kinesiophobia and disgust with catastrophic thinking, anxiety, and depression, or conclude these as spurious. Studies spanning a more diverse patient mix may offer insight into the debate about how accurately this type of tool can code facial expression of emotions in people from different geographical, cultural, and environmental contexts [4]. The authors caution about misinterpreting such data given the complexity of universal human emotional expressions and how subtly these emotions blend. However, a recent study by Cowen et al. [7] actually revealed remarkable similarities in the facial expressions of emotions from different cultures and ethnicities using a using artificial intelligence (deep neural network analysis) and a range of facial expressions of emotions captured from more than 6 million video clips from over 144 countries. The team endorsed the need for more detailed mapping and classification of emotional expressions.

Third, although we did not perform repeat assessments of the video footage for each encounter, we did perform multiple analyses of the footage for one encounter as we familiarized ourselves with the software. This demonstrated the same outputs for the section of footage relevant to our study. Based on this and the validation and reliability data generated by the company and independent investigators to date, we proceeded with a single run of analysis. Future studies involving larger, more diverse populations from different backgrounds could include repeat analysis of video footage to ensure reliability. Fourth, the clinician and patient knew the visit was being filmed. It is possible that clinicians, and to an extent patients, may have altered their behavior knowing they were being studied (the Hawthorne effect), although this is felt to have limited influence on such experiments [28]. No matter the degree of Hawthorne effect, the observed associations are useful. If anything, a stronger effect seems likely in unaware patients and clinicians [36]. Fifth, we did not track patients declining participation. We relied on the diverse demographics of our population to be sufficiently representative. Future larger-scale studies in this field can be more rigorous. Finally, some may consider the heterogeneous set of conditions a limitation. We deem patient variety, and presumably the corresponding variations in clinician facial expressions, important in testing correlations with mental health. Also, pathology type and severity were not explanatory variables, in part due the small scale of this pilot study, and in part given that pathology is shown to have limited correlation with symptom intensity and the magnitude of activity intolerance.

Association Between Psychological Distress and Physicians’ Expressed Emotions

A patient’s mental health can be measured using self-reported questionnaires, detected in the patient’s words (verbal), his or her movements (nonverbal), and now, it appears, the clinician’s face. Clinicians may be sensing and reacting to a patient’s psychological state based on the finding that patient distress and unhealthy cognitive bias are measurable in the clinician’s facial expressions. Given that patients are not always forthcoming in answering mental health surveys [15, 39], measurement of the patient’s mental health from the clinician’s face signals an alternative approach to increasing clinician awareness of mental health opportunities in their patients [37].

Technological advances promise to better account for these complexities, ensure greater accuracy and reliability of facial recognition across contextual variations, including situations where patients find themselves under stress and distress, and even account for verbal cues and exchanged words (including content, style, and tonal analysis), body language, and movement during the course of an interaction [6]. Our study is limited to the association between clinician expressions of emotions and patient mindset and cannot discern directionality. Mapping the course of a consultation and recording patient and clinician actions and reactions, while challenging, may help address the direction of influence and whether clinician facial expressions contribute to misconceptions and greater distress.

Insights and consolidation of this line of work only increases the appeal of coaching clinicians in more effective communication strategies using tools that register and help link facial expressions and their own inner feelings and reactions (emotional self-awareness) with psychological distress and misconceptions. More effective verbal and nonverbal communication strategies are associated with greater trust and perceived empathy, greater satisfaction [3], improved cognitive and physical functioning [20], adherence to physician recommendations [17], lower symptom intensity, fewer patient complaints [22], and lower healthcare costs [19, 22, 26, 34].

Potential real-world clinical applications could borrow from the digital display on a fighter pilot’s helmet that provide visual cues in situ, guiding safer flight. Similarly, clinicians wearing smart glasses (optical head-mounted digital displays) could receive emotion and misconception-related metrics to navigate a more effective consultation [10, 45]. Behavioral nudges (akin to the visual cues on a FitBit or AppleWatch encouraging more steps) could help clinicians, especially those less familiar or less accepting of the strong influence of mental health, become better at noticing distress and unhealthy misconceptions before making these comfortable topics of conversation. Such applications could provide a moral directive for clinicians to better gauge mental health opportunities and attend to the psychological aspects of illness. Simply put, sharing another person’s emotional state and viewing things from their perspective (whether consciously or unconsciously) is central to effective human response and interaction [9, 14, 33].

Association Between Clinician Facial Expressions of Emotions and Measurements of Clinician-patient Relationship

The absence of a correlation between objectively measured clinician facial expressions of emotions and patient experience of the patient-clinician interaction may be the result of the high ceiling effect and inadequate measurement of variations in patient experience. Ceiling effects occur when a high proportion of study participants have maximum scores on an observed variable and are common with measures of patient experience [41, 44]. In our study, the PDRQ-9 demonstrated a median of 44 points with a maximum score of 45 points and a ceiling effect of 41% (Table 1), which is higher than other studies, for example, a mean of 37 in a study among 2275 German patients while consulting with a primary care physician [46] and a mean of 31 among 226 Chinese patients at an outpatient clinic [32]. To better study the relationship between various emotional-visual communication factors and patient experience, we will need to develop experience measures with a greater spread in scores alongside studies that identify important facilitators and barriers to a healthful patient experience. We might also consider qualitative analysis of clinician and patient feedback regarding the finding that technology may notice aspects of the patient-clinician interaction that may not be currently noticed or appreciated. This type of research has been advocated in gaining deeper insights related to the development and study of innovations based on machine learning that may otherwise be challenging to eke out from quantitative work [24]. This line of research could further help prepare strategies for coaching and communication skills development.

Conclusion

Darwin theorized that facial expression of emotion is universal among humans [13]. We demonstrated that technology-enabled measurement of a clinician’s facial expression as a marker of basic emotions indicates they are sensing the patient’s mental health opportunities. In other words, musculoskeletal clinicians are aware (consciously or unconsciously) that patients are experiencing misconceptions, worry, and despair. Although being kind or altruistic is necessary, good care for people with musculoskeletal symptoms also involves learning to recognize, appreciate, and prioritize mental health opportunities before practicing effective communication and care strategies. In-the-moment clinician feedback using facial recognition technology may help facilitate this endeavor and ensure such opportunities are not overlooked. Future studies might include analysis of facial expression of emotion technology to wider populations (from different cultures and geographies) and clinical contexts (such as, decision making around elective surgery and acute settings in orthopaedic trauma care). Qualitative analysis may provide deeper insights into patient and professional experiences using this technology in real-time, while quantitative analysis could further assess the association with outcomes of care, especially after the information from the tool is used for coaching and training or even fed back in real-time to providers and even patients.