Computer Vision-Based Assessment of Motor Functioning in Schizophrenia: Use of Smartphones for Remote Measurement of Schizophrenia Symptomatology

Abstract

Introduction

Motor abnormalities have been shown to be a distinct component of schizophrenia symptomatology. However, objective and scalable methods for assessment of motor functioning in schizophrenia are lacking. Advancements in machine learning-based digital tools have allowed for automated and remote “digital phenotyping” of disease symptomatology. Here, we assess the performance of a computer vision-based assessment of motor functioning as a characteristic of schizophrenia using video data collected remotely through smartphones.

Methods

Eighteen patients with schizophrenia and 9 healthy controls were asked to remotely participate in smartphone-based assessments daily for 14 days. Video recorded from the smartphone front-facing camera during these assessments was used to quantify the Euclidean distance of head movement between frames through a pretrained computer vision model. The ability of head movement measurements to distinguish between patients and healthy controls as well as their relationship to schizophrenia symptom severity as measured through traditional clinical scores was assessed.

Results

The rate of head movement in participants with schizophrenia (1.48 mm/frame) and those without differed significantly (2.50 mm/frame; p = 0.01), and a logistic regression demonstrated that head movement was a significant predictor of schizophrenia diagnosis (p = 0.02). Linear regression between head movement and clinical scores of schizophrenia showed that head movement has a negative relationship with schizophrenia symptom severity (p = 0.04), primarily with negative symptoms of schizophrenia.

Conclusions

Remote, smartphone-based assessments were able to capture meaningful visual behavior for computer vision-based objective measurement of head movement. The measurements of head movement acquired were able to accurately classify schizophrenia diagnosis and quantify symptom severity in patients with schizophrenia.

Introduction

Motor abnormalities have been shown to be a characteristic trait of schizophrenia, are present even in antipsychotic-naive patients, predate the onset of psychosis, and have been demonstrated in at-risk populations and in unaffected relatives [1, 2, 3, 4, 5, 6]. Indeed, psychomotor retardation underlies core negative symptomatology of schizophrenia, such as the demonstration of blunted facial affect and emotional withdrawal [7, 8, 9, 10, 11]. Though there have been many attempts to diagnose schizophrenia and quantify symptom severity using neurobiology and behavior [12, 13, 14], little work has been done to objectively measure motor dysfunction to characterize the disorder in clinical research [15, 16, 17], which could be of particular relevance in the assessment of negative symptoms.

Recent advances in the mechanistic understanding of negative symptomatology have led to a number of promising pharmacological and cognitive treatments for negative symptoms of schizophrenia [18, 19, 20, 21]. Such initiatives are important given the lack of FDA-approved treatments for negative symptoms [22]. However, reliable and change-sensitive measures of negative symptomatology to assess the efficacy of these treatments are sparse [23, 24, 25, 26]. With motor dysfunction being an underlying factor of negative symptomatology, accurate measurement of motor functioning can allow for assessment of treatment efficacy during the evaluation of novel investigational treatments.

Several groups have successfully and accurately measured motor functioning as a characteristic of schizophrenia, though only in laboratory settings [27, 28, 29, 30, 31]. While laboratory-based methodologies increase sensitivity and reduce subjectivity in the measurement of motor functioning compared to traditional assessments, they are limited in their utility as measurement tools in clinical research given the burden imposed on both patients and clinicians.

The use of “digital phenotyping” or measurement of disease symptomatology using digital tools has shown significant promise towards objective, scalable, and remote measurement of central nervous system functioning [32, 33]. In schizophrenia, observable behavior associated with the disorder has been successfully quantified using advancements in machine learning. Examples of this include using digital signal and natural language processing methods to measure changes in verbal prosody and speech characteristics, such as volume, fundamental frequency, pause characteristics, and sentiment, and computer vision to quantify changes in facial expressivity and the presence of blunted facial affect, a core negative symptom of schizophrenia [34, 35, 36, 37, 38]. The use of visual data such as videos of patient behavior presents a promising avenue for digital phenotyping given the ubiquitous availability of smartphone cameras alongside advances in computer vision methodologies that can automate the processing and analysis of visual information. Despite recent advances in head pose estimation [39, 40], there are a lack of studies utilizing these methods to quantify head movement abnormalities in schizophrenia.

In the current investigation, we examined the ability of head movement measured using computer vision from videos recorded during remote smartphone-based assessments to differentiate individuals with schizophrenia from healthy controls and determine disease severity by comparison of head movement with the Positive and Negative Syndrome Scale (PANSS), a commonly used clinical assessment tool for the measurement of schizophrenia severity and the current “gold standard” for assessment of antipsychotic treatment efficacy [41].

Methods

Participants

Patients meeting DSM-5 criteria for schizophrenia at Mount Sinai Health System Outpatient Psychiatry Clinics (n = 18; age µ = 48.1 years, σ = 13.1; 11 females) and healthy individuals from the community (n = 9; age µ = 39.8 years, σ = 10.4; 5 females) were consented to participate in a 2-week observational study under approval of the Mount Sinai Program for the Protection of Human Subjects and its Institutional Review Board. Participants in the schizophrenia group were on a stable regimen of psychotropic medications, had no recent clinically meaningful change in schizophrenia symptomatology, and were clinically stable in that they were not expected to show significant changes in symptomatology over the course of the 2-week observational study.

Data Collection

Positive and Negative Syndrome Scale (PANSS)

The PANSS was administered in-person by the study team on both patients with schizophrenia and healthy control subjects. Healthy individuals were assessed to confirm an absence of schizophrenia symptomatology. From the PANSS, subscale scores were recorded for the positive symptom subscale (P Total), the negative symptom subscale (N Total), and general symptom subscale (G Total) in addition to all individual items within those subscales.

Remote Smartphone-Based Video Assessments

All participants were asked to download the AiCure app (www.aicure.com) on either their personal smartphone or a smartphone provided to them by the study team. They were then trained by the study team on how to use the app to participate in remote assessments. The remote assessments triggered the participants to perform a brief and standardized 1-min task where they were asked open-ended questions to which they responded freely while a video of their response was captured using the front-facing camera on the smartphone. The open-ended questions were neutral in nature and simply meant to elicit a free verbal response (e.g., “What have you been doing for the past few hours?” and “What are your plans for the rest of the day?”). All participants were shown the same prompts and stimuli to ensure standardization and reproducibility of the manner in which all data were collected. Dependent on the length of the participant's responses, the videos collected would have been of different lengths, hence introducing variability. However, all observations reported in this manuscript reflect average movement behavior over the course of those videos, and hence the length of the video is not a factor in any analyses. The assessments were scheduled every day for the 14 days of the study. The videos collected during these assessments were used to quantify head movement during the participants' responses.

Measurements of Head Movement

The software library OpenFace [42, 43, 44] was used to measure framewise head movement from the videos collected through the remote smartphone-based assessments. For each frame, the head's position relative to the camera was calculated using pretrained convolutional neural network-based computer vision models. The software provides a confidence score for every frame of video denoting the likelihood that it is accurately detecting a face; only frames with a confidence score of 80% or higher were used for downstream analyses. Framewise measurements of change in Euclidean head position in the x, y, and z planes were calculated in millimeters as shown in the equation below. From the framewise measurements, the mean head movement was calculated across all videos collected from a participant over the course of the study. All measurements of head movement were normalized between 0 and 1 before subsequent data analysis. Head movement in frame:

where x_n = head position in the x plane during the current frame, y_n = head position in the y plane during the current frame, and z_n = head position in the z plane during the current frame.

Data Analysis

Using head movement data collected from all participants in both groups, a multiple logistic regression with maximum likelihood estimation was conducted to model the probability of an individual having a schizophrenia diagnosis based on measurements of head movement, with age and gender as additional predictors in the regression. Using data only from individuals with schizophrenia, separate linear regressions were conducted to determine the relationship between head movement and the PANSS subscale scores in addition to individual items in each of those subscales, with age and gender as confounding variables. The linear regressions were an exploratory analysis to determine the relationship between head movement and individual items on the PANSS, and hence multiple comparisons correction was not conducted. Additionally, a t test was conducted using the average rate of head movement between the individuals with schizophrenia and healthy controls.

Data Availability

The raw video analyzed in this manuscript is considered protected health information. It was not consented for public use and cannot be published. However, the first level of non-identifiable raw abstractions from that data as well as the derivations used to conduct all analyses will be provided alongside the code.

Code Availability

Code for all methods and analysis presented in this manuscript is publicly available on GitHub (https://github.com/anzarabbas/ms_headmov_scz).

Results

The rate of head movement in participants with schizophrenia (1.48 mm/frame) and those without differed significantly (2.50 mm/frame; p = 0.01), as is shown in Figure 1.

Fig. 1.

The rate of head movement in participants with schizophrenia (SCZ) was significantly less than in healthy individuals (HC).

Logistic Regression

Data from 9 healthy controls and 17 individuals with schizophrenia was used for the logistic regressions, 1 individual being excluded due to missing PANSS scores. Head movement, age, and gender were able to significantly differentiate healthy controls and individuals with schizophrenia (log likelihood = −12.13; Pseudor² = 0.29; p = 0.018) with head movement as the only significant predictor (odds ratio = 0.04, 95% CI 0.00–0.48; z = −2.21; p = 0.027) and age demonstrating a marginal effect (odds ratio = 1.07, 95% CI 0.99–1.18; z = 1.61; p = 0.1; Table 1).

Table 1.

Results from multiple logistic regression of head movement, age, and gender with schizophrenia diagnoses

Variable	Coefficient	SE	z	p value
Constant	−0.456	2.051	−0.222	0.824
Head motion	−5.290	2.393	−2.210	0.027
Age (years)	0.074	0.046	1.611	0.107
Gender	−0.416	1.030	−0.404	0.686

Linear Regression

In the data collected from individuals with schizophrenia (n = 18), separate linear regressions were conducted with head movement, age, and gender as predictors of each individual item on the PANSS. The results from all of these regressions are detailed in online supplementary Tables 1, 2, and 3 (for all online suppl. material, see www.karger.com/doi/10.1159/000512383). A subset of the results from those regressions are outlined in Tables 2 and 3.

Table 2.

Results for the linear regressions conducted with head movement, age, and gender as predictors of PANSS subscale scores

Variable	Predictor	Coefficient	SE	t	p value	F	R2	Adj. R2	p value
P total						2.013	0.301	0.152	0.158
	Constant	0.5152	0.190	2.711	0.002
	Head motion	−0.3032	0.257	−1.181	0.257
	Age (years)	0.0007	0.003	0.204	0.842
	Gender	0.2057	0.092	2.248	0.041
N total						3.055	0.396	0.266	0.0634
	Constant	0.5649	0.211	2.683	0.002
	Head motion	−0.6383	0.284	−2.245	0.041
	Age (years)	0.0037	0.004	0.955	0.356
	Gender	0.2107	0.101	2.078	0.057
G total						6.200	0.571	0.479	0.0067
	Constant	0.6209	0.139	4.457	0.001
	Head motion	−0.5193	0.188	−2.760	0.015
	Age (years)	0.0025	0.003	0.967	0.350
	Gender	0.2338	0.067	3.485	0.004

Table 3.

A subset of the results for the linear regressions conducted with head movement, age, and gender as predictors of individual PANSS items

Dependent variable	Predictor	Coefficient	SE	t	p value	F	R2	Adj. R2	p value
Blunted affect						2.565	0.372	0.227	0.099
	Constant	0.5832	0.264	2.221	0.046
	Head motion	−0.8133	0.356	−2.284	0.040
	Age (years)	0.0038	0.005	0.789	0.444
	Gender	0.2040	0.128	1.590	0.136
Emotional withdrawal						4.849	0.528	0.419	0.017
	Constant	0.5527	0.206	2.681	0.019
	Head motion	−0.7386	0.27	−2.654	0.020
	Age (years)	0.0086	0.004	2.294	0.039
	Gender	0.1933	0.100	1.927	0.076
Poor rapport						1.830	0.297	0.135	0.191
	Constant	0.6081	0.275	2.208	0.046
	Head motion	−0.8332	0.372	−2.241	0.043
	Age (years)	−0.0005	0.005	0.100	0.922
	Gender	0.1107	0.134	0.826	0.424
Motor retardation						4.862	0.529	0.420	0.017
	Constant	0.5972	0.233	2.563	0.024
	Head motion	−1.1118	0.315	−3.534	0.004
	Age (years)	0.0062	0.004	1.447	0.172
	Gender	0.0757	0.113	0.668	0.516
Uncooperativeness						2.034	0.319	0.162	0.159
	Constant	0.7059	0.234	3.023	0.010
	Head motion	−0.6860	0.315	−2.176	0.049
	Age (years)	−0.0019	0.004	−0.438	0.669
	Gender	0.1316	0.114	1.159	0.267
Poor attention					0.044	5.454	0.557	0.455	0.012
	Constant	0.4311	0.194	2.224	0.049
	Head motion	−0.5696	0.262	−2.177	0.976
	Age (years)	0.0001	0.004	0.031	0.004
	Gender	0.3322	0.094	3.523
Lack of judgement and insight						6.892	0.614	0.525	0.005
	Constant	0.5480	0.212	2.587	0.023
	Head motion	−1.0518	0.286	−3.677	0.003
	Age (years)	0.0075	0.004	1.950	0.073
	Gender	−0.1455	0.103	−1.412	0.181

When correcting for age and gender, head movement had a significant negative association with the N Total and G Total scores (p = 0.041 and p = 0.015, respectively) but not the P Total score (p = 0.257; Table 2). Furthermore, when correcting for age and gender, head movement was negatively associated with several individual items on the PANSS (Table 3): blunted affect (p = 0.040), emotional withdrawal (p = 0.020), poor rapport (p = 0.043), motor retardation (p = 0.004), uncooperativeness (p = 0.049), poor attention (p = 0.049), and lack of judgement and insight (p = 0.003).

Discussion

In the current investigation, we sought to test the hypothesis that motor functioning as measured by the amount of head movement during free behavior, quantified through convolutional neural network-based computer vision models, would serve as a digital marker of schizophrenia symptomatology. We further assessed the ability of digital tools, that is, remote smartphone-based assessments that capture video of individuals' behavior, to measure schizophrenia symptomatology in an accurate, scalable, and objective manner.

The findings demonstrate that head movement during free responses to scripted prompts delivered over a smartphone and recorded through the front-facing camera significantly differentiated healthy controls from individuals with schizophrenia (Table 1). Furthermore, the exploratory results suggest that slowed head movement is a marker of negative symptomatology of schizophrenia, including blunted affect, motor retardation, poor rapport, and emotional withdrawal (Table 2). Additionally, we found preliminary evidence that head movement is associated with uncooperativeness, poor attention, and lack of insight. While the interpretation of this relationship is less straightforward, this may reflect underlying cognitive deficits in schizophrenia associated with negative symptomatology. Our findings align with previously reported motor slowing in individuals with schizophrenia [45, 46, 47, 48] and are consistent with the notion that different types of motor symptoms may correlate with different aspects of psychotic symptoms [49].

Furthermore, the results demonstrated that motor functioning can be objectively measured using data captured remotely, in the absence of a clinician or a clinical environment, and without the need for any specialized equipment or hardware beyond a smartphone. Video captured of an individual's behavior during remote smartphone-based assessments was sufficient for measurement of head movement as a behavioral symptom of schizophrenia, and may be sufficient for computationally based measurement of other behavioral symptoms such as changes in facial expressivity, vocal acoustics, and characteristics of speech. With tools emerging that are able to measure all such behavioral symptomatology using computer vision, digital signal processing, and machine learning [34, 35, 36, 37, 38, 50, 51, 52, 53, 54, 55], these behaviors may be measurable in the same manner that movement behavior was assessed in this investigation.

Abnormalities in motor functioning underlie observable symptomatology in several neurological and neuropsychiatric disorders, including major depressive disorder [56, 57], Parkinson's disease [58], and autism spectrum disorder [59, 60, 61], among others. Given the transdiagnostic nature of motor symptomatology, remote and scalable methods for its measurement may have utility for the assessment of treatment response and/or motor side-effects of treatment in both clinical care and clinical research beyond just schizophrenia. With advancements in computationally based measurement of other observable behavioral symptomatology, such as facial expressivity, speech, and physiology, digital measurements could bring immense utility in the monitoring and diagnosis of mental and physical health using remote tools.

It is important to note the key limitations of this study, in particular the multiple comparisons conducted on the modest sample size. With the primary objective of this study being to test the hypothesis that head movement is negatively correlated with schizophrenia diagnosis, the logistic regression provided compelling evidence towards rejection of the null hypothesis. The subsequent linear regressions conducted between head movement and all PANSS items served as a secondary and exploratory aim, allowing for determination of a relationship between head movement and specific schizophrenia symptomatology. With results from the analysis indicating relationships as would be hypothesized based on prior reports, we present them as preliminary findings with the ultimate goal of expanding such an experiment to a wider patient population, allowing for a more fine-grained analysis of head movement as a digital biomarker of schizophrenia symptomatology.

Additionally, the sample size inhibited the ability to age-match the 2 populations, which would have been an important step in alleviating age-related variability in head movement beyond simply correcting for it in the regression. Furthermore, given patients were on a wide range of antipsychotics, it was difficult to assess the effects of individual medications, including antipsychotic-induced motor abnormalities such as tardive dyskinesia, which have the potential to influence head movement measurements. However, involuntary movement behavior would have only increased the head movement intensity observed in the individuals with schizophrenia. Given we were able to reject the null hypothesis, it is possible that antipsychotic-induced head movement did not contribute very strongly as a confounding variable.

Finally, this investigation utilized open-source Python-based software, available to all researchers. As an additional measure, the code that implemented the open-source software for this investigation and subsequent analyses of results have been provided by the authors in the Methods section. This allows for the expansion of the experiment to a wider patient population as mentioned above and the independent validation of the computer vision-based measurement of head movement itself and its implementation in this investigation by other researchers in academic and clinical research, following an open-science framework for the development of digital tools for objective, accurate, and scalable measurement of disease symptomatology in both mental and physical health. Adaptation of such technology will allow for its implementation in larger patient populations, potentially leading to future research that would be able to train machine learning-based classifiers that use head movement along with other quantifiable features to identify the presence and severity of neuropsychiatric disorders such as schizophrenia.

Conclusions

In this investigation, we demonstrated that head movement measured using computer vision from video captured remotely via smartphones demonstrates validity as a marker of schizophrenia and is a promising metric for negative symptom severity. Use of such technology in clinical care and clinical research settings could allow for accurate measurement of disease symptomatology and treatment response in a scalable and accessible manner, which can support the development of novel treatments for schizophrenia and other mental and physical disorders that involve motor symptomatology.

Statement of Ethics

All participants gave their written informed consent for the current study. The study protocol was approved by the Mount Sinai Program for the Protection of Human Subjects and its Institutional Review Board.

Conflict of Interest Statement

Dr. Mercedes Perez-Rodriguez has received research grant funding from Neurocrine Biosciences Inc., Millennium Pharmaceuticals, Takeda, Merck, and AiCure. She is an advisory board member for Neurocrine Biosciences Inc. and a consultant on an American Foundation for Suicide Prevention (AFSP) grant (LSRG-1-005-16, PI: Baca-Garcia). Anzar Abbas, Vijay Yadav, Vidya Koesmahargyo, Li Zhang, Lei Guan, and Isaac R. Galatzer-Levy are employed at AiCure and own shares or are eligible to own shares in AiCure.

Funding Sources

This study was funded by AiCure LLC (Protocol AICVDX01), GCO: 18-0968-00001-01-PD.

Author Contributions

A.A., V.Y., V.K., L.Z., L.G., and I.R.G.-L. contributed towards the conception of the study, development of technology used, data analysis conducted, and reporting of the results. E.S., E.R., S.B.R., C.B., P.R., and M.P.-R. carried out all participant recruitment, technology onboarding, and clinical data collection.

Supplementary Material

Supplementary data