Affect Estimation with Wearable Sensors

Abstract

Affective states are associated with people’s mental health status and have profound impact on daily life, thus unobtrusively understanding and estimating affects have been brought to the public attention. The pervasiveness of wearable sensors makes it possible to build automatic systems for affect tracking. However, constructing such systems is a challenging task due to the complexity of human behaviors. In this work, we focus on the problem of estimating daily self-reported affects from sensor-generated data. We first analyze the intra- and inter-subject differences of self-reported affect labels. Second, we explore different machine learning models as well as label transformation techniques to overcome the individual differences in self-reported responses estimation. We conceptualize three experimental settings including long-term and short-term estimation scenarios. Our experimental results show that the mixed effects model and label transformation can yield better estimation of individual daily affect. This work poses the basis for future sensor-based individualized and real-time affective digital and/or clinical interventions.

Introduction

Self-reported affect has been shown as good representation of mental status. For example, self-reported unhappiness is indicative of scores on clinical measures of depression [11], and self-reported happiness is also strongly associated with health status [2]. Therefore, measuring and understanding self-reported affect plays an important role in clinical applications.

Studies also show that affect could impact the job performance of employees [12]. Hospital workers and students are examples of people working in a complex environment that consists of diverse social interactions, norms, and professional activities. They are also reported as a population at high risk of physical and mental health issues. Some studies pointed out that hospital workers are at high risk of developing occupational burnout [15, 25], which yields symptoms of exhaustion, cynicism, and job inefficacy that develop from excessive and prolonged stressors in work environment. Theses problem may further impact worker’s performance in patient care. Universities also pay attention on college students’ affect status. Depressed emotions are prevalent on college campuses, and are also the most frequent precursor to suicide [36]. For these reasons, the ability of tracking and/or predicting affects yields a number of sensor technologies in recent years it possible to carry out continuous, unobtrusive sensing of individuals including their physiology (e.g., cardio activity), physical activity (e.g., steps taken), social interactions (e.g., amount of speech), and environmental contexts (e.g., outdoor/indoor) [1, 14, 34, 35]. Such data have, in turn, created opportunities for an enhanced understanding of factors contributing to psychological health and well-being. Researchers have also attempted to use sensor data with machine learning techniques to implement systems that could estimate individuals’ self-reported affect [20].

However, physiological signals, perception, and behavior patterns all may be impacted by individual differences, which pose challenges in constructing such a system. First, similar situations may have different impact on people’s affect experiences and displays. Second, self-reported values are influenced by differences in perception across individuals. Different scores may be reported even though two people may have shared the same affect experience. In other words, the perception of external stimuli (e.g., real-world events, personal experiences) varies from individual to individual [4] and consequently self-perceived and expressed affects vary accordingly across individuals.

Existing computational systems for affect detection mostly only consider the extreme cases. They either have binary self-reported labels (e.g., happy or sad) [7, 31] or split the continuous scores into binary labels [21]. However, in reality, individual differences might cause overall high affect score for some people, while relatively lower for others. Thus, the definition of “extreme case” is also open to debate. Simply binarizing the labels will yield losing the observed individual variability of reported affects. Meanwhile, such a coarse approach can easily miss important distinctions relevant to clinical applications.

Another major shortcoming of typical systems is that they attempt to estimate every individual’s affect using the same, one-size-fits-all model [7, 20, 31], ignoring the high degree of individual variation in how a person’s behavior and environment may affect their mood and influence their expressed affect. Personalized models, which train customized models for each individual’s data, also have been explored to overcome the shortcomings. However, it is important to note that these models are subject-dependent, that is, they require to have enough data from each person [21], and enough variability in such data to represent the entire spectrum of positive and negative affects.

In this work, we propose a method to estimate daily self-reported affect labels using wearable sensor data. Our method uses regression techniques to directly estimate the self-reported values thus avoiding the information lost yielded by binarization.

To address the different scoring preference induced by individual variability, we propose a method to rescale the collected self-reported scores. Our method aims to standardize the affect variation across subjects, i.e., affect labels that deviate from the average affect with same distance should have same scores. We will show that such rescaled scores allow machine learning models to better capture inter- and intra-subject affect variation, while exhibiting the desirable advantage that the rescaled scores can be converted back to the original scale without any information loss.

As for estimation model, we adopt a nonlinear mixed effects model, namely mixed effects random forests. Mixed-effect models allow us to capture individual differences as well as to learn the general patterns of affect dynamics from other subjects’ records. Experimental results will show that the mixed-effect models outperform the other types of machine learning models, including the personalized models, especially in presence of less individual data.

The contributions of this paper are as follows:

• We conduct detailed analyses on the distribution and temporal trends of individual self-reported affect labels. We analyze and compare the daily survey labels in two longitudinal studies, then discuss how the properties of low variance and high bias in such type of data impact the affect estimation task.

• We propose the use of mixed effects models and label rescaling methods to capture the inter- and intra-subject variation of affect across subjects. This method is aimed at addressing the challenge induced by the personal bias in affect perception, providing a promising direction for accurately inferring self-reported affect scores.

• We analyze the estimation power of different types of sensory features. This gives the insights on the relationship between daily self-reported affect and the physiological signals, providing a guideline for future study designs.

This study builds upon our previous work on estimating self-reported affect using wearables [37]. This paper focuses on two unique contributions:

• We conduct an in-depth analysis of self-reported affect responses in a longitudinal setting. In addition to the analysis on different affect metrics, we discover the temporal trends of the responses over the period of study.

Related Work

Physical and Physiological Indicators of Affect

There are a large number of studies investigating the relationship between affect states and physical activity. Stephens [33] summarized epidemiological evidence that showed a clear association between activity and several indices of subjective well-being. Fox further has given a summary of the influence of physical activities at the work place [16]. The findings suggest that even single bouts of activity can improve mood, and that people who are more active are much more likely to rate themselves and their sense of mental well-being more positively.

For physiological information, previous studies have shown that heart rate variation (HRV) covaries with emotion regulation [3]. Bhat et al. [6] further showed the relationship between HRV and affect. Positive affect can increase the coherence of heart rate rhythms, i.e., decrease the HRV, while negative affect such as anger and bitterness decreases the coherence of heart rate. Breathing patterns have been shown to be another good indicator of affect. Studies [26] show an implicit influence of respiratory feedback on the induction of emotional feelings.

Affect Estimation

Automatically estimating human affect is an increasingly popular topic in recent years, and there is a growing number of studies that explore the relationship between sensor data and psychological metrics.

Some previous studies have conducted similar studies in real-world settings. They collected multi-modal data from individuals in order to infer psychological and other health states. For example, the 10-week StudentLife study of Dartmouth undergraduate and graduate students used passive and mobile sensor data to study well-being, academic performance, and behavioral trends [34]. SNAPSHOT, a 30-day study on MIT undergraduates, used mobile sensors and surveys to understand sleep, social interactions, affect, performance, stress, and health [32]. RealityMining, a 9-month study of 75 MIT Media Laboratory students, used mobile sensor data to track the social interactions and networking [14]. The friends-and-families study collected data from 130 adult members of a young family community to study fitness intervention and social incentives [1].

Different approaches also have been explored on the data from these studies for the automatic estimation of individuals affect states. Bogomolov et al. [7] used mobile phone, weather, and personality data to predict binary stress labels. Sano et al. [31] proposed a method to separate happy-sad mood group from sleep parameters, using 30 days of wearable sensor and self-reported data from undergraduate students. They also analyzed the factors impact the affect states. Jaques et al. [20] developed a binary classification model with multi-modal information, including physiology, phone log data, mobility, and survey data, to distinguish between happy and unhappy mood of college students, which only considered the top 30% scores (positive days) and bottom 30% scores (negative days).

Most of these aformenetioned studies build a generic model for all participants, although several of them have discussed the importance and effect of personalization. For example, Canizian and Musolesi [9] found that for detecting depressed mood, the generic model yielded worse sensitivity and specificity scores. LimKamWa et al. [23], who used a smartphone app to infer subjects’ mood based on communication history and usage patterns, demonstrated that their generic model yielded 66% accuracy while the person-specific classifier reached 93% accuracy. Jaques et al. [21] extended their work using Domain Adaptation and Multitask Learning techniques. They proposed principled ways for building personalized models that can leverage the data of all individuals. However, their model still requires adequate data from each individual, which may not be possible in real-world applications.

In this paper, we propose a method for estimating self-reported affect with wearable sensors to directly predicts the self-reported affect scores instead of binary classification. Our method is based on mixed effects model, which can both leverage the information from all individuals and address the limitation of previous work.

Study and Dataset

To demonstrate the generalizability of the proposed method, we conduct our experiments on two different types of cohort populations: hospital workers and college students. The datasets come from two longitudinal studies that share significant similarities: the TILES dataset [8, 22] captures the activity and affect states of hospital workers; StudentLife [34] analogously characterizes a college student population. The two studies have similar design, including comparable study duration, collected physiological signals, and psychological surveys.

Hospital Workers

We designed and carried out a study named Tracking Individual Performance with Sensors (TILES) [8, 22]. Over the course of 10 weeks, we tracked more than 200 hospital worker volunteers’ well-being and behavioral trends in a large Los Angeles hospital. Two hundred twelve participants are recruited with three rolling-recruitment periods in a period between March and May of 2018 that respectively enrolled 52, 116, and 44 volunteer participants. Bio-behavioral signal data were collected from continuous sensing of garment-based OMsignal1 sensor and wristband-based sensor Fitbit Charge 2.2 Daily affect labels are collected from daily surveys delivered via a mobile application installed in their smartphones.

Sensor Data

Bio-behavioral signal data were collected from continuous sensing of garment-based OMsignal sensor and wristband-based sensor Fitbit Charge 2. Participants were asked to wear their Fitbit on a 24/7 basis (limiting off-line time as much as possible and for reasons such as battery recharging) for the whole study. In addition to Fitbit, subjects were instructed to wear the OMsignal garments exclusively during work hours. The OMsignal garments (shirt for male participants, sport bra for female subjects) are equipped with a “sensing box” that measures biometric data in real time—collected data-streams are sent via Bluetooth Technology to the participant’s smartphone, each equipped with the OMsignal app, for persistent data recording.

OMsignal provides physiological information including heart rate, heart rate variability (HRV), breathing, and accelerometer. Fitbit Charge 2 is designed to measure heart rate, step count, heart rate zones, and sleep statistics. Heart rate zones are based on a percentage of maximum heart rate that is estimated for each subject. For example, Out of Zone is the total number of minutes when the heart rate falls below 50% of the estimated maximum heart rate. Out of Zone is also an indicator of the absence of physical exercise. Sleep statistics include the asleep minute and the duration of light, rapid eye movement (REM), and deep sleep. The calculation of sleep statistics and the inference of sleep stages is done automatically by the Fitbit software using machine learning [5].

Affect Labels

Affect was measured by using the short form for Positive and Negative Affect Schedule (PANAS-Short) [24], which only takes less than 1 min to complete. PANAS-Short presents affect-related terms (e.g., enthusiastic, irritable), and participants respond using a Likert scale ranging from 1 (very slightly or not at all) to 5 (extremely). The PANAS-Short consists of 10 items and is scored by summing a participant’s responses to the 5 items pertaining to positive affect (range 5–25) and the 5 pertaining to negative affect (range 5–25). Thus, the PANAS-Short yields two scores: PANAS-Short Positive Affect Score and PANAS-Short Negative Affect Score. Higher scores indicate stronger affect. PANAS-Short survey will be sent to the participants daily and ask them to rate how they feel at the moment.

Compliance Rate

The bio-behavioral signal data were from two types of sensors: Fitbit Charge 2 and OMsignal garment. Fitbit sensors were meant to be worn 24/7 during the study. In our dataset, over 80% of the participants exhibited over 90% compliance rate of Fitbit, and only 5% participants had less than 60% compliance rate. This means that most participants wore their Fitbit for the majority of the 10-week periods. OMsignal garments were only meant to be worn during the working hours, which means we can only collect OMsignal data from around 30% of the days during which the 10-week long studies occurred, due to the participants’ working schedules.

The overall compliance rates are 92% for Fitbit, 32% for OMsignal, and 74% for daily survey data.

Students

Students’ data come from the 10-week StudentLife study [34] of Dartmouth undergraduate and graduate students. Forty-eight students joined this study during Spring term in 2013. They were asked to carry smartphones with them throughout the day. Automatic sensing data were collected without any user interaction and uploaded to the cloud when the phone was being recharged and under WiFi. During the collection phase, students were asked to respond to various ecological momentary assessment (EMA) questions as they use their phones.

Sensor Data

The automatic sensing of StudentLife study relies on the StudentLife app. It automatically infers activity (stationary, walking, running, driving, cycling), sleep duration, and sociability (i.e., the number of independent conversations and their durations). The app also collects accelerometer, proximity, audio, light sensor readings, location, co-location, and application usage. The inferences and other sensor data are temporarily stored on the phone and are efficiently uploaded to the StudentLife cloud when users recharge their phones under WiFi.

Affect Labels

StudentLife uses Photo Affect Meter (PAM) [27] as the measurement of daily affect. PAM is an unobtrusive measurement of affect, presenting the user with a randomized grid of 16 pictures from a library of 48 photos. Each picture represents a 1–16 score, mapping to the PANAS. The user selects one picture that best fits their mood. PAM score keeps with the conceptualization of PANAS positive affect as ranging from low pleasure and low arousal to high pleasure and high arousal at the high end. PAM is strongly correlated with PANAS (r = 0.71,p < 0.001) for positive affect.

Compliance Rate

The trend of data dropping off as the term winds down. The overall compliance of collecting automatic sensing data from all users over the term is 82%. On average, 5.5 EMAs were received per day per student across the whole term. As the term progresses, there is a drop in both administered EMAs and responses.

Affect Label Analysis

Distributions

Figure 1 shows the distribution of self-reported affect labels. In particular, Fig. 1a, Fig. 1b, and Fig. 1c respectively show the distributions of the mean of positive affect, negative affect, and PAM; Fig. 1d, e, and f instead show the distributions of the variance of positive affect, negative affect, and PAM responses, respectively. These data show that the self-reported affects exhibit very different distributions.

Fig. 1.

Participants’ self-reported affects exhibit different distributions. a–c respectively show the distribution of individual average scores. d–f show the distribution of individual responses variance

PAM responses are most close to the normal distribution. However, for positive affect, the resulting distribution of the mean, shown in Fig. 1a, is centered around 12.8 exhibits the full range of scores (with a tendency to lower scores being over-represented with respect to higher scores). The within-subject standard deviation is on average 3.2, as reflected by the distribution of positive affect variance—see Fig. 1b.

For negative affect, the distribution of the mean, shown in Fig. 1c, is significantly right skewed at around 5, and the within-subject standard deviation is on average 1.82, as shown by its distribution of the variance illustrated in Fig. 1d, which is right skewed as well.

Temporal Trends

Figure 2 examines within-subject temporal variation, with data partitioned in five temporal groups of increasing length, namely 2-week, 4-week, 6-week, 8-week, and all 10-week periods. Looking at the distribution of affect labels across temporal partitions, we can observe that the mean and variance of self-reported affect labels are stable over time. This suggests that self-reported affect did not change dramatically over this 10-week study period, and only few samples deviated from the mean of all observations.

Fig. 2.

Distribution of self-reported affect for a sample individual (SD1003/u00), over different length time windows in five temporal groups of increasing length, namely 2-week, 4-week, 6-week, 8-week and all 10-week periods. The mean and variance of self-reported affect labels are quite stable over time

More interestingly, in TILES dataset, the individual variances of self-reported responses decrease over time. Figure 3 illustrates the change of the individual average and standard deviation of responses. The self-reported affects have larger variance at the beginning of the study (e.g., the first 2 weeks), while participants tend to have more similar responses as the study goes.

Fig. 3.

Distribution of the mean/standard deviation of all participants’ self-reported affect over different time windows in 10 weeks

Feature Set Construction

Both physiology and physical activity features are extracted from the collected sensor data. To create the feature matrix, statistics (e.g., mean, standard deviation) of each time series was calculated over a 24-h time window, the same day the affect surveys was reported by participants.

In general, the features in TILES dataset are directly extracted from the physiology signals, e.g., heart rate, steps, while StudentLife dataset provides more high-level features such as daily activity and conversation patterns. We summarize the features into three categories for each dataset.

Table 2 provides the summary of the extracted features for both datasets. We categorize the features into different feature classes. TILES dataset includes four feature types: heart rate, physical activity, sleep, and context. StudentLife also provides the audio relevant features collected from the smartphones. The full feature set is available in the Appendix.

Table 2.

Summary of extracted features

TILES
Heart rate (7 features)	Heart rate
	Heart rate variability (HRV)
Physical activity (67 features)	Step count
	Breathing rate/depth
	Accelerometers
	Body postures
	Exercise zones
Sleep (16 features)	Sleep duration
	Sleep efficiency
	Sleep duration change
Context (3 features)	Location
	Interaction
	Activity
StudentLife
Physical activity (13 features)	Stationary duration
	Walk duration
	Run duration
	Exercise duration
Audio (11 features)	Conversation duration
	Conversation frequency
	Silence duration
	Silence frequency
Context (8 features)	Number of unique locations
	Number of nearby devices
	Class time

Heart rate.

Both Fitbit and OMsignal collect heart rate data with sampling rate ranging from 1 to 15 s. We calculate statistical features such as the mean and standard deviation of heart rate time series during the day of the affect report. Additionally, Fitbit also provides summary information of resting heart rate and heart rate zones. Resting heart rate refers to the heart rate measured when subjects are awake, calm, comfortable, and have not recently exerted efforts—it can be an important indicator of the health and predictor of mood changes. Heart rate zones split the heart rate values of a full-day into different stages. Four zones are calculated by Fitbit: peak zone (greater than 85% of maximum heart rate), cardio zone (70 to 84% of maximum), fat burn zone (50 to 69% of maximum), out of zone (below 50% of maximum).

Additionally, OMsignal also calculates the heart rate variability (HRV) from the electrocardiogram (ECG) signal. HRV metrics are calculated from a 5-min time window moving over daily ECG recordings. Two widely used HRV features are provided by the OMsignal sensor: SDNN (standard deviation of normal RR intervals) and RMSSD (root mean square of SDNN).

Sleep.

Fitbit provides three types of sleep information: sleep efficiency, sleep duration, and duration of sleep stages. Sleep efficiency is described as the ratio of sleep duration to the duration of lying on the bed. Fitbit estimates the sleep stages using a combination of movement and heart rate patterns. We also compare the change of duration of sleep patterns compared with the previous day as new sleep features to measure the regularity of sleep. According to [5], three algorithmically inferred sleep stages are recorded in Fitbit sleep log: light sleep, deep sleep, and rapid eye movement (REM) sleep. Light sleep promotes mental and physical recovery. Deep sleep promotes physical recovery and aspects of memory and learning. This stage has also been shown to support the immune system [17]. REM sleep has been shown to play an important role in mood regulation, learning, and memory, as the brain processes and consolidates information from the previous day so that it can be stored in long-term memory [28].

Physical activity.

Fitbit and OMsignal both record the step count over time. OMsignal also provides breathing, accelerometers, and body postures time series with 5 min resolution. We further aggregate these time series by extracting the mean and standard deviation of each series of each day associated with affect labels.

Fitbit and OMsignal both record the step count over time. OMsignal also provides breathing, accelerometers, and body postures time series with 5 min resolution. We further aggregate these time series by extracting the mean and standard deviation of each series of each day associated with affect labels.

Context.

Participants were asked to provide contextual information when answering the survey. This information was encoded as categorical values. Table 1 provides details about these categorical features.

• Location: The location where the participant took the survey.

• Activity: The activity the subject was engaged with immediately before starting the survey.

• Interaction: The interaction (if any) the subject was engaged with immediately before starting the survey.

Table 1.

Context features

Location	Activity		Interaction
1 - Home	1 - Working	7 - Caring	1 - In person
2 - Work	2 - Communication	8 - Personal care	2 - Electronically
3 - Indoor	3 - Leisure	9 - Educational	3 - No interaction
4 - Outdoor	4 - Purchasing	10 - Organizational
5 - Vehicle	5 - Eating	11 - Travel
6 - Other	6 - Household	12 - Other

In StudentLife dataset, context features are computed as the distance a student travels inside buildings during the day using WiFi scan logs. WiFi scan logs including all encountered BSSIDs, SSIDs, and their signal strength values. BSSIDs and signal strength are used to determine if a student is in a specific building.

Audio.

StudentLife implements two classifiers on the phone for audio and speech/conversation detection: an audio classifier to infer human voice and a conversation classifier to detect conversation. The classifier pipeline aims to infer if the user is “around conversation.” The output of the classification pipeline captures the number of independent conversations and their duration. Audio features include the frequency and duration of conversations around a participant as a measure of sociability.

Comparison of the Two Cohort Datasets

TILES [8, 22] and StudentLife [34] are two longitudinal studies that collected similar sensor signals and self-reported affect labels (Table 2). In this section, we compare the two selected datasets.

• Study design: both studies lasted for 10 weeks, and the self-reported affect labels were collected through smartphones on a daily basis.

• Sensor: TILES collected physiological signals through wearable sensors (Fitbit and OMsignal), while StudentLife only relied on the sensors on smartphones.

• Affect labels: both TILES and StudentLife collected affect labels through daily surveys sent out to the subjects’ smartphones. TILES used a widely applied psychometric, PANAS-Short. StudentLife used PAM as the measurement of daily affect, which is strongly correlated with PANAS for positive affect.

• Feature set: both studies collected physical activity and context information. The TILES dataset has more physiological signals including heart rate and sleep. StudentLife also collected audio features through smartphones.

• Compliance rate: TILES has overall higher compliance rate (approx. 90%) in terms of both sensing data and surveys, and the compliance rate is relatively more stable over time.

We refer the reader interested in additional details to consult the study design papers for TILES [8, 22] and StudentLife [34].

Methods

Most approaches proposed in the literature treat affect prediction as a classification problem: They either infer binary class labels (e.g., high positive affect and low positive affect) [7, 31] or split the continuous scores to superimpose binary labels [20]. Another aspect worth noting is that previous work evaluate their estimation methods in a mixed samples approach.

In our experimental setting, we focus on a more realistic scenario. We hypothesize an artificial scenario in which we have collected sensor data and self-reported affect labels from participants enrolled in a study for a number of weeks. We aim to use these data to train models that can estimate the subjects’ daily self-reported affect in the subsequent future. Such an artificial setting can be easily achieved a posteriori, with the data from the 10-week long study at our disposal, by splitting data in training and test sets, each of arbitrary suitable duration, and thus using training data to learn the models that will run on test data to generate estimations. The advantage of such strategy is that we will have ground truth assessments available for the test data as well, which will allow us to systematically estimating the accuracy of the models. Furthermore, by tuning the temporal length of training data, we will be able to test the robustness of our method depending on different conditions and data constraints. Meanwhile, we aim to exploit regression techniques to directly estimate the self-reported values.

Baseline: Stability of Daily Self-Reported Affect

From the observation we have drawn in Fig. 1, participants have low individual standard deviation of their daily self-reported affect scores. Figure 4 further illustrates the relationship between daily affect scores and the average of individuals’ scores. The frequency of that value in historical data is represented by the size and color of each dot. Larger and lighter dots are the self-reported values that have higher frequency to present in the dataset. It is clear that most of the self-reported responses are very close to the mean of each participant’s scores.

Fig. 4.

Relationship between individual average affect and daily self-reported scores. The size and color of dots represent the frequency of that score showed up in individual records. Larger size and lighter color represent higher frequency

For further analysis, we investigate the individual patterns of daily self-reported affect. The positive affect scores of two randomly selected subjects are taken to illustrate this phenomenon. In Table 3, we can observe that the individual average affect scores (μ) are similar over time. The %(M + σ) stands for the ratio of future affect scores within σ-distance from μ. For the two participants considered in this example, almost 30% of future scores only deviate from μ by 1, and more than 40% future scores have the distance less than 2.

Table 3.

Individual patterns of affect score labels

Time duration	Average affect scores (μ)	%(μ ± σ)
		σ = 1	σ = 2
(a) SD1003
2 weeks	12.142	0.352	0.529
4 weeks	11.428	0.351	0.648
6 weeks	11.238	0.217	0.608
8 weeks	11.267	0.333	0.777
(b) SD1074
2 weeks	9.692	0.254	0.418
4 weeks	9.320	0.279	0.488
6 weeks	8.307	0.275	0.620
8 weeks	8.245	0.200	0.666

A systematic analysis with all participants in our data is also provided. Again, for illustration purposes, we refer to the same setting as above with 6 weeks of training data. As Fig. 5 shows, in this setting, half of the affect scores from more than 100 participants fall in the range within 1 distance from individual average scores.

Fig. 5.

Ratio of future affect scores within σ-distance from individual average scores

Given the low variation of self-reported affect values among our participants, we can hypothesize that the average of historical self-reported affect labels can be a possibly accurate estimate of individual affect in the future. We implement this idea as our Baseline model, which simply calculates the mean of all available self-reported labels in the training data for each participant. Those values are used as the estimated labels for the corresponding participants. The accuracy of this Baseline model is reported in Table 4. Noteworthy, the baseline model’s results are generally promising, in particular for positive affect; however, they exhibit very significant fluctuations depending on the size of the training data, suggesting that the occurrence of external events that can influence subjects’ mood have a strong impact on the accuracy of the model.

Table 4.

Estimation performance of baseline model

Construct	Time duration	Baseline
		R2	Corr.	MAE
Positive affect	2 weeks	0.390	0.651	3.252
	4 weeks	0.543	0.767	2.737
	6 weeks	0.640	0.815	2.410
	8 weeks	0.705	0.845	2.223
Negative affect	2 weeks	0.085	0.387	1.444
	4 weeks	0.005	0.368	1.502
	6 weeks	0.147	0.435	1.377
	8 weeks	0.219	0.480	1.393
PAM	2 weeks	0.184	0.431	2.045
	4 weeks	0.279	0.544	2.022
	6 weeks	0.373	0.630	1.963
	8 weeks	0.449	0.737	2.071

Rescaling Affect Labels

Although the baseline model shows a promising performance, it cannot capture the daily variation of participants’ affects. Two major problems with self-reported affect scores are self-bias and perception differences across subjects (a.k.a. intra- and inter-subject variability). Some individuals perceive mood changes in a more or less extreme fashion than others, and therefore self-report affect scores in a larger or smaller ranges, respectively. Furthermore, even within the same subject, context or other sources of individual variability may bias affect responses making affect scores and their variations inconsistent over time. Hence, self-reported affect scores may not be immediately comparable across subjects, or within the same subject across time or contexts.

In order to better estimate the affect variations, we apply two rescaling methods in our estimation pipeline.

Z-score transformation

Z-score (a.k.a standard score) captures sample variance, measuring how distant a given observation (i.e., a self-reported affect score of a subject) is from the sample mean.

$$z\text{-score}=(y-\mu )/\sigma ,$$ 1

where y is the original self-reported score, μ represents the historical individual average affect scores, and σ is the individual standard deviation of affect scores.

After getting the estimated rescaled value $\widehat{z}$, it is easily to convert it back to the original range via

$$ŷ=\widehat{z}\times \sigma +\mu \mathrm{.}$$ 2

R-score transformation

Z-score transformation subtracted the population mean from the original scores, which neglects the individual differences of the self-reported labels. To maintain the differences across subjects, we propose another transformation method: revised z-score (r-score).

$$\begin{array}{rcl}r\text{-score}& =& \mu \times (1+z\text{-score}(y\left)\right)\\ & =& \mu \times (1+(y-\mu )/\sigma ),\end{array}$$ 3

where y is the original self-reported score, μ represents the historical individual average affect scores, and σ is the individual standard deviation of affect scores.

The reversed transformation method of r-score is

$$ŷ=\widehat{r}\times \frac{\sigma }{\mu }-\sigma +\mu \mathrm{.}$$ 4

Figure 6 shows the distribution of affect scores before (Fig. 6a) and after (Fig. 6b) rescaling. This rescaling does not change the mean of the original affect scores; it only rescales the deviation of each sample from the mean, such that two subjects with the same sample mean will also have the same standard deviations. There are three significant advantages of rescaling methods:

• After rescaling, participants will have affect scores mapped into the same range, irrespectively of the range of the original affect score distributions.

• The two scaling functions are reversible, which means we can map back the rescaled affect scores into the original scale as long as the mean and variance of a given sample (i.e., a subject’s affect scores) is preserved.

• Compared with z-score, r-score transformation does not change the mean of individual affect scores, which furture preserves the differences (or similarities) in affect perception across individuals.

Fig. 6.

Distribution of original and rescaled labels. a shows the distribution of two synthetic samples with same mean and different standard deviations. b is the distribution of the rescaled values. After rescaling, we remove the difference of standard deviation, and thus, the two samples have similar distribution

Estimation Models

Previous studies have explored different types of machine learning models in affect estimation. In this work, we select four most popular and effective types of models [18, 29, 30]: bagging model, boosting model, personalized model, and mixed effects model.

Bagging Model

Random forest model is a typical example of bagging models. We built the random forest model based on the implementation in scikit-learn library with n_estimators = 300 and max_depth = 5. Hyper parameters including n_estimators and max_depth are tuned with 10-fold cross validation.

Boosting Model

Boosting is an ensemble method for improving the model predictions by training weak learners sequentially and each trying to correct its predecessor. XGBoost [10] is an implementation of gradient-boosted decision trees designed for speed and performance. We built the boosting model based on XGBoost with a similar configuration n_estimators = 250 and max_depth = 3.

Personalized Model

Previous work has discussed the performance of personalized models in affect prediction [9, 21, 23], noting that personalization is highly reliant on the availability of adequate data from each participant. Recently, [21] showed that their Personalized Model yields state of the art accuracy in affect prediction in settings similar to TILES. For such a reason, we decided to implement a model inspired by the concept of Personalized Model, where a separate model is learned for each subject participant. We implemented a version of personalized model that leverages each subject’s entire history of affect scores (i.e., all self-reported affect scores available in the training data) for learning purposes, and adopts random forest for prediction of future individual affect scores.

Our personalized model is based on the standard random forest model. We train a separate random forest model for each participant in our dataset based on their individual historical affect scores.

Mixed Effects Model

Mixed effects models take into account both the fixed effects (i.e., the variation that is explained by the features) and the random effects, namely the individual effects that are uncorrelated with the features and are better explained by individual variability.

Using the mixed effects modeling approach, Hajjem et al. [19] extended the random forest algorithms to the case of clustered data, formalizing the mixed effects random forest (MERF) model we use in our present work. Similarly to linear mixed effect models, MERF assumes a generative model of the form

$$\begin{array}{rcl}& & y_i=f\left(X_i\right)+Z_i{b}_i+{𝜖}_i,\\ & & b_i\sim \mathcal{N}(0,D),{𝜖}_i\sim \mathcal{N}(0,R_i),i=1,\mathrm{...},n,\end{array}$$

where y_i is the vector of responses for the observations in cluster i, X_i is the feature matrix of fixed effects covariates, Z_i is the matrix of random effects covariates, b_i is the unknown vector of random effects for cluster i, 𝜖_i is the vector of errors. D and R_i = σ²I_n are the covariance matrix of b_i and 𝜖_i, respectively.

A MERF model first starts with default values for b_i, σ², and D, i.e., b_i = 0, σ² = 1, and D = I, and it calculates the fixed part of the response variable using a standard random forest algorithm. Second, the MERF model takes bootstrap out-of-bag prediction samples from the training set to build a forest of trees: b_i, σ², and D are updated based on the estimates of the residuals. This algorithm keeps iterating until convergence.

To predict the response for a new observation j that belongs to a cluster i among those used to fit the MERF model, the predicted result is the combination of both its corresponding population-averaged random forest (RF) prediction, $\widehat{f}\left(x_{ij}\right)$, and the predicted random effect part corresponding to its cluster, $Z_i\widehat{b_i}$. For a new observation that belongs to a cluster not included in the sample used to estimate the model parameters, the random forest (RF) prediction $\widehat{f}\left(x_{ij}\right)$ is considered the prediction result.

Results

In order to empirically demonstrate the advantages of the mixed effects model and label transformation methods, we compare the estimation results of four types of models (Section 4.3) and the results with and without label transformation. Three metrics are used to assess regression performance: coefficient of determination (R²), Pearson correlation (Corr.), and mean absolute error (MAE).

We hence lay out three experimental settings:

1. Long-term estimations: In Section 5.1, by training the four models on time windows of different lengths, we estimate the subjects’ long-term affect states over the entire test period; we show that the proposed MERF with label transformation method systematically outperforms the others.

2. Short-term estimations: In Section 5.2, we carry out short-term estimations of subjects’ affect by training the models on 2 weeks of data and then estimating on the two subsequent weeks, sliding the train/estimation windows by 1 week at the time, to cover the entire study duration.

3. Long-term estimations informed by short-term estimations: In Section 5.3, the estimations of short-term models from Section 5.2 are used as training inputs for the long-term estimation models from Section 5.1, showing that long-term estimations can be satisfactory even in absence of training on ground truth data.

Long-term Estimations

Estimations with Original Affect Scores

Table 5 reports the results from the four types of models above with original affect labels in the datasets. “Time duration” defines the amount of data used in the training process: for example, “2 weeks” means that we only take all participants’ first 2 weeks worth of data to train the models, and use the remainder of data (in this example, 8 weeks) as test data to assess regression performance.

Table 5.

Performance evaluation with original scores: random forest vs. XGBoost vs. personalized model vs. MERF

Construct	Time duration	R2	Corr.	MAE	R2	Corr.	MAE
		Random forest			XGBoost
Positive affect	2 weeks	0.090	0.346	4.217	0.082	0.387	4.177
	4 weeks	0.275	0.573	3.698	0.233	0.511	3.730
	6 weeks	0.372	0.651	3.449	0.323	0.581	3.533
	8 weeks	0.457	0.717	3.360	0.429	0.667	3.395
Negative affect	2 weeks	− 0.070	0.152	1.766	− 0.154	0.167	1.727
	4 weeks	− 0.110	0.157	1.750	− 0.139	0.179	1.728
	6 weeks	− 0.047	0.185	1.698	− 0.048	0.220	1.647
	8 weeks	0.084	0.305	1.633	0.056	0.279	1.553
PAM	2 weeks	0.030	0.219	2.004	− 0.048	0.244	2.164
	4 weeks	0.156	0.410	2.059	0.016	0.380	2.099
	6 weeks	0.110	0.424	2.687	0.123	0.522	2.637
	8 weeks	0.219	0.685	3.448	0.178	0.567	3.538
		Personalized model			MERF original
Positive affect	2 weeks	0.365	0.642	3.296	0.364	0.627	3.384
	4 weeks	0.525	0.759	2.787	0.550	0.767	2.800
	6 weeks	0.619	0.801	2.475	0.634	0.806	2.525
	8 weeks	0.676	0.827	2.346	0.711	0.847	2.277
Negative affect	2 weeks	0.038	0.376	1.501	0.049	0.322	1.629
	4 weeks	− 0.088	0.371	1.610	− 0.005	0.332	1.634
	6 weeks	0.009	0.379	1.491	0.084	0.382	1.509
	8 weeks	0.184	0.489	1.460	0.182	0.447	1.476
PAM	2 weeks	− 0.123	0.320	2.078	0.059	0.332	1.941
	4 weeks	0.242	0.552	2.006	0.348	0.594	1.868
	6 weeks	0.466	0.714	1.996	0.447	0.713	2.037
	8 weeks	0.723	0.961	2.154	0.578	0.975	2.592

The best results in model comparison are shown in italics

Personalized random forest and MERF significantly outperform the other two models across all experimental settings. When compared with personalized models, MERF tends to yield better results in terms of R², especially when we have less data in the training set (e.g., using only 2 weeks worth of data to predict the next 8 weeks’ responses). MERF exhibits much better performance for the construct with less within-subject variability, i.e., negative affect. MERF outperforms the personalized model when the training set has fewer than 8 weeks’ data, and yields an 8-fold improvement in R² when the time duration is 6 weeks.

Comparing with the results in Table 4, even the best results of the four models cannot outperform the baseline in some cases. The gap between our results and baseline tends to be larger when in presence of short training data windows and the target variable shows a highly skewed distribution (e.g., negative affect).

Estimations with Label Transformation

Next, we aim to improve the modeling performance with the assistance of label transformation. We integrate the two transformation methods we presented in Section 4.2 with the MERF model. For each subject in the training data, we generate the subject’s distribution of self-reported affect scores and calculate its mean and standard deviation; then, we rescale the affect scores as described earlier (cf. Section 4.2). These will be used as ground truth labels for training purposes. In the estimation (i.e., prediction) stage, the estimated values need to be converted back to the original scale. The rescaling only relies on the mean and standard deviation of each participant’s historical responses. This assumption is safe, as we previously empirically showed (cf. Fig. 4) the stability of the distribution of individual affect scores over time. So the estimated values can be converted back to the original scale using the statistic metrics from the past without any significant loss of accuracy.

The pipeline of the estimation process depicted in Fig. 7 can be summarized as follows:

• Step 1: For each subject in the training data, we extract all features from sensors and surveys.

• Step 2: For each subject in the training data, we generate the subject’s distribution of self-reported affect scores and calculate its mean and standard deviation; then, we rescale the affect scores according to Eqs. 1 and 3. These will be used as ground truth labels for training purposes.

• Step 3: We fit the MERF model using the extracted features and the rescaled affect scores as input training data.

• Step 3: For each subject, we convert the estimated MERF’s output affect scores back to the original subject’s scale according to that individual’s mean and standard deviation (calculated in Step 2 on training data).

• Step 4: We output the values obtained after Step 3, and evaluate the accuracy of predictions.

Fig. 7.

Estimation pipeline with label transformation

Comparing against the results with original score in Table 5, all results based on rescaled scores exhibit significantly higher R², as shown in Table 6. The estimations for PAM have the most significant improvement across all three constructs. The R² boosts from 0.059 to 0.181 when training with 2 weeks, which is more than a twofold improvement. A thorough analysis of Table 6 reveals two additional insights:

• For all three affect labels, the r-score shows better estimation performance when using less data for training; for example, the MERF model with r-score can estimate PAM with R² 0.181 (resp., 0.449) comparing with 0.173 (resp., 0.409) when training with 2 weeks (resp., 4 weeks) of data. This insight suggests that the r-score transformation allow us to obtain a better estimation when less information is available for model training.

• According to the analysis above, models without label transformation cannot accurately estimate negative affect. However, MERF with the r-score transformation shows significant improvement for negative affect estimations compared with z-score. When the training set has 8 weeks worth of data, the r-score transformation can yield 28% and 22% increase in R² with respect to the baseline and z-score transformations.

Table 6.

Performance evaluation with rescaled scores: r-score vs. z-score

Construct	Time duration	MERF r-score			MERF z-score
		R2	Corr.	MAE	R2	Corr.	MAE
Positive affect	2 weeks	0.408	0.665	3.277	0.404	0.668	3.221
	4 weeks	0.570	0.789	2.763	0.592	0.794	2.638
	6 weeks	0.648	0.822	2.484	0.661	0.826	2.399
	8 weeks	0.706	0.850	2.338	0.724	0.856	2.198
Negative affect	2 weeks	0.110	0.355	1.523	0.093	0.376	1.491
	4 weeks	0.139	0.404	1.458	0.052	0.399	1.508
	6 weeks	0.193	0.447	1.386	0.129	0.454	1.432
	8 weeks	0.274	0.525	1.390	0.228	0.509	1.420
PAM	2 weeks	0.181	0.446	1.772	0.173	0.439	1.793
	4 weeks	0.449	0.672	1.714	0.409	0.644	1.790
	6 weeks	0.543	0.757	1.857	0.550	0.745	1.790
	8 weeks	0.749	0.996	1.965	0.766	0.997	1.923

The best results in model comparison are shown in italics

Short-term Estimations

In Section 5.1, we illustrated the performance of the proposed models on long-term estimations. Due to the possible temporal variance of self-reported labels, in this section, we further analyze the estimation performance focusing on the short term: we use 2 weeks of data to train the models and then estimate on the two subsequent weeks. To generate estimations that cover the entire study duration, the train and estimation windows are slid by 1 week at the time. For example, the model is trained on the first 2 weeks’ worth of data and then it estimates the affect states for week 3 and 4; afterwards, we train the model with data from week 2 and 3, then estimate week 4 and 5; and so on, until the entire 10-week period is covered.

Table 7 reports the average performance over all different train/estimation windows. Similar to the long-term estimation, MERF and personalized model outperform other models based on original scores. The two types of models exhibit similar performance and have R² twice as high as random forest and XGBoost models.

Table 7.

Performance evaluation for short-term estimation

Construct	Random forest			XGBoost
	R2	Corr.	MAE	R2	Corr.	MAE
Pos. Aff.	0.28 ± 0.09	0.54 ± 0.08	3.64 ± 0.23	0.24 ± 0.09	0.51 ± 0.07	3.65 ± 0.23
Neg. Aff.	− 0.04 ± 0.08	0.17 ± 0.07	1.66 ± 0.07	− 0.03 ± 0.07	0.22 ± 0.06	1.61 ± 0.09
PAM	− 0.20 ± 0.15	0.05 ± 0.23	2.52 ± 0.25	− 0.38 ± 0.31	0.14 ± 0.29	2.79 ± 0.28
Construct	Personalized model			MERF original
	R2	Corr.	MAE	R2	Corr.	MAE
Pos. Aff.	0.53 ± 0.08	0.75 ± 0.04	2.68 ± 0.31	0.55 ± 0.09	0.75 ± 0.05	2.67 ± 0.32
Neg. Aff.	0.10 ± 0.18	0.45 ± 0.11	1.38 ± 0.11	0.12 ± 0.10	0.38 ± 0.10	1.49 ± 0.07
PAM	− 0.11 ± 0.25	0.43 ± 0.10	2.21 ± 0.21	− 0.14 ± 0.24	0.29 ± 0.09	2.44 ± 0.16
Construct	MERF r-score			MERF z-score
	R2	Corr.	MAE	R2	Corr.	MAE
Pos. Aff.	0.58 ± 0.07	0.77 ± 0.04	2.60 ± 0.27	0.61 ± 0.07	0.79 ± 0.04	2.45 ± 0.29
Neg. Aff.	0.20 ± 0.12	0.46 ± 0.10	1.41 ± 0.04	0.21 ± 0.14	0.50 ± 0.10	1.34 ± 0.09
PAM	0.10 ± 0.42	0.51 ± 0.28	1.94 ± 0.24	0.15 ± 0.40	0.52 ± 0.27	1.87 ± 0.25

The best results in model comparison are shown in italics

As expected, the short-term estimations yield better performance than the long-term ones (cf., Tables 5 and 6 in Section 5.1). MERF with both label transformation methods exhibits the better performance for all three constructs; specifically, MERF with z-score transformation shows the best performance in terms of R².

Long-term Estimations Informed by Short-term Estimations

In real-life applications, each participant usually has few affect labels (e.g., the first 2 weeks) for model training. In this section, we further discuss the possibility of informing long-term estimations by short-term estimations.

In this experimental setting, we obtain the first 2 weeks’ affect scores for each participant and train the initial model with those data. The model then is updated in a manner in which the estimations become available in a sequential order: we generate short-term estimations of affect scores for the two subsequent weeks (i.e., week 3–4) from the initial model, and then treat these estimations as ground truth to re-train the model; we repeat the above process to get the short-term estimations and update the model for different time durations.

Table 8 shows the performance of informed long-term estimations. Comparing with the baseline model, the long-term estimations informed by short-term estimations achieve better performance for negative affect and PAM. However, the estimation errors are propagated in the updating process. The greater the number of estimations in the training, the more noisy the estimations might be. The estimation performance stops improving or becomes worse when more noisy labels are involved in model training.

Table 8.

Performance evaluation for informed long-term estimation

Construct	Time duration	MERF r-score			MERF z-score
		R2	Corr.	MAE	R2	Corr.	MAE
Positive affect	4 weeks	0.427	0.692	3.201	0.447	0.707	3.123
	6 weeks	0.413	0.679	3.257	0.431	0.693	3.185
	8 weeks	0.444	0.696	3.322	0.479	0.719	3.187
Negative affect	4 weeks	0.131	0.389	1.493	0.059	0.384	1.515
	6 weeks	0.175	0.437	1.482	0.132	0.445	1.485
	8 weeks	0.173	0.421	1.546	0.147	0.433	1.540
PAM	4 weeks	0.163	0.448	2.099	0.228	0.507	2.025
	6 weeks	0.394	0.640	2.007	0.452	0.686	1.941
	8 weeks	0.579	0.833	2.004	0.596	0.864	2.064

The best results in model comparison are shown in italics

Analysis

Feature Importance

We investigate the importance of different features and the role they play in the estimation of daily affect scores. For MERF with both label transformation methods, we calculated the feature importance based on Gini index, which is used as the splitting measure.

For sake of illustration, in Fig. 8, we present the fractions of top 15 features in each classes showed in Table 2. Additionally, the top features vary with different time duration and the label transformation the model use. We summarize the common top features of models with different time duration and label transformation for each affect label in Table 9.

Fig. 8.

Feature importance analysis. Stacked barplots showing how different feature classes are represented among the top 15 features

Table 9.

Common top features with different time duration and label transformation

Construct	Common top features	Feature class
Positive affect	Location-home	Context
	Std heart rate	Heart rate
	Sleep minutes stage light change	Sleep
Negative affect
	Mean SDNN	Heart rate
PAM	Phone screen dark duration	Audio
	Running duration	Activity
	Number of WiFi location	Context

Physical activity features are good indicators for all three constructs, especially for negative affect, physical activity features compose almost 70% of the top 15 features. Heart rate features also frequently appeared among top features for both positive and negative affects. StudentLife also collected audio features, which are 40% of the top features. There are also trends of how sleep features help the estimation of positive and negative affects. As the time duration expand, the fraction of sleep features decline from 20 to 6% with the positive affect estimation with both r-score and z-score transformation. While negative affect showS an opposite direction, the fraction of sleep features increases as the time duration expand.

Within the common top features in Table 9, context features such as location are associated with positive affect and PAM, while Mean SDNN, one of HRV measurements, is the only common top feature for negative affect estimation. Figure 8 c and d also indicate that context features are not informative for negative affect.

Feature Analysis

Based on the feature importance, we further analyze the correlation between top features and the target variables.

Context

Context features, such as location and interaction features, are good indicators for positive affect. Moreover, location-home has the highest feature importance in all settings. Figures 9 and 10 compare the distribution of affect labels in different context groups. The results show that home (location), working (activity), and no interaction (interaction) are the three main context features that correlate with affect labels. Specifically, participants who are at home, or who are engaged in personal care, or who have not had any social interaction prior to taking a survey, have significantly lower reported positive affect labels compared with other groups. For negative affect, lower labels are reported during leisure and personal care. Dunn’s tests [13] are also conducted to verify the significance of our observations. Home, working, and no interaction are all significantly different from other groups, having p values less than 0.0001. Similarly, context features (e.g., number of unique locations) also positively correlated (p value < 0.0001) with PAM responses.

Fig. 9.

Context vs. positive affect labels with Dunn’s test results. Two groups linked by the horizontal line are significantly different from each other, p < 0.0001. No interaction, home, work are groups that are significantly different from other groups for positive affect

Fig. 10.

Context vs. negative affect labels with Dunn’s test results. Two groups linked by the horizontal line are significantly different from each other, p < 0.0001. No interaction, home, work are groups that are significantly different from other groups for negative affect

Sleep

Sleep information is available in TILES dataset measuring by Fitbit. Fitbit provides three types of sleep information: sleep efficiency, sleep duration, and duration of sleep stages. We also compare the change of duration of sleep patterns compared with the previous day as new sleep features to measure the regularity of sleep. Within the top features for both positive affect and negative affect, the regularity of sleep patterns is informative for affect variation, for example, sleep minutes stage light change for positive affect and sleep start time change for negative affect.

Physical Activity

StudentLife provides more high-level features compared with TILES, while the top features show consistency in how physical activity impact daily affect in both datasets. Top features like Running Duration and Stationary Duration in StudentLife can be infered from wearable sensor signals such as heart rate and steps, which also appear in top features for positive/negative affects for TILES dataset. For example, positive affect is negatively correlated with Std Heart Rate (r = − 0.08, p < 0.01), and Running Duration also share a negative correlation with PAM (r = − 0.26, p < 0.01).

Discussion

The method we proposed in this work uses mixed effects model and label transformation to address the problem of inter- and intra-subject differences. This method provides a more accurate way to estimate affective states. It can be applied as an unobtrusive approach to track mental health states for early intervention and self-management. Colleges or companies may also leverage this method to try improving the work/study environment, etc.

Compared with other methods, ours yield better performance and need fewer training samples. This suggests a desirable property: our method is particularly amenable when in presence of sparsity in the training data, a condition that often occurs in real-world applications. Another interesting aspect of our work is its experimental design: here, we focused on estimating individual future affect by leveraging historical sensor data and self-reported scores. This setting can be easily replicated in real-world applications for individual affect monitoring or tracking, using wearable sensors and smartphone apps.

Our method has two major limitations. First, our mixed effects model is based on the subject-based clusters, which means it only can generate predictions for the subjects with some historical data has been collected. In other words, it cannot produce predictions for held-out samples. Second, the label transformation relies on the stability of self-reported affect. For other constructs with higher within-subject variability, label transformation might decrease the performance.

Conclusions and Future Work

The pervasiveness of wearable sensors and other sensing technologies allows for unprecedented opportunities to model, quantify, and predict human behavior in natural settings. We recently carried out one such study, called TILES, where we collected wearable sensor and survey data from over 200 hospital worker volunteers over the course of 10 weeks. This paper focuses on the problem of affect estimation from sensor-generated data. Ground truth data, in the form of self-reported daily affect scores, was available to us and it allowed for the design and evaluation of various affect prediction models. For sake of generality, we also expand the cohort to student populations with StudentLife dataset.

We conducted both distribution and temporal analysis on the self-reported affect scores. Even in presence of known self-bias and individual variability in perception of emotions and mood states, our analysis illustrated that individual self-reported affect scores are mostly stable over time and within an individual, with few exception that significantly deviate from the average observations—these deviations, however, may be the most important to estimate due to the fact that they may be associated with external shocks related to out-of-the-ordinary events.

Motivated by the properties of real-world self-reported affect, we propose the method combining mixed effects random forest (MERF) model and label transformation for estimations tasks. For evaluation, we lay out three experimental settings to test the utility of our method in different application scenarios: long-term estimations, short-term estimations, and long-term estimation informed by short-term estimations.

We explore different types of machine learning models, such as boosting method, personalized models, and mixed effects models. We further propose a baseline model that predicts future affect scores as simply the average of historical affect scores. Estimation results show that this baseline model yields reasonable predictions due to the stability of affects. The MERF model outperforms other models in majority experimental settings.

When combining MERF with label transformation, the experimental results suggest that label transformation is able to improve the estimation individual variation and fluctuation of daily affect. In presence of short training data periods, e.g., 2 or 4 weeks, and for highly skewed distributed affect labels (e.g., negative affect), our proposed r-score can yield more accurate affect estimates. When more training data is available, e.g., 8 weeks, the performance of MERF with z-score transformation has better estimation performance.

Our investigation and the contributed model pose the basis for future sensor-based individualized and real-time affective digital and/or clinical interventions. For future work, we will extend this work to a more general application scenario, where no historical data are required to estimate the affect state of the participant of interest (so called subject hold-out tasks). Also, additional features from multi-modal sensors, such as speech interaction patterns, will be explored to further improve the model.

Appendix

Table 10.

Extracted features

TILES	Class	Feature name
	Heart rate	Heart_Rate_mean-Fitbit	Heart_Rate_std-Fitbit
		RMSSD_mean-OMsignal	RMSSD_std-OMsignal
		SDNN_mean-OMsignal	SDNN_std-OMsignal
		Resting_Heart_Rate-Fitbit
	Sleep	Sleep_Efficiency-Fitbit	Sleep_Efficiency_change-Fitbit
		Minutes_Awake-Fitbit	Minutes_Awake_change-Fitbit
		Minutes_Asleep-Fitbit	Minutes_Asleep_change-Fitbit
		Minutes_In_Bed-Fitbit	Minutes_In_Bed_change-Fitbit
		Minutes_Stage_Deep-Fitbit	Minutes_Stage_Deep_change-Fitbit
		Minutes_Stage_REM-Fitbit	Minutes_Stage_REM_change-Fitbit
		Minutes_Stage_Light-Fitbit	Minutes_Stage_Light_change-Fitbit
		Minutes_Stage_Wake-Fitbit	Minutes_Stage_Wake_change-Fitbit
	Phys. Act.	Number_of_Steps-OMsignal	AvgBreathingDepth_mean/std-OMsignal
		Steps_Count_mean/std-Fitbit	AvgBreathingRate_mean/std-OMsignal
		Cadence_mean/std-OMsignal	StdBreathingDepth_mean/std-OMsignal
		Intensity_mean/std-OMsignal	StdBreathingRate_mean/std-OMsignal
		Sitting_mean/std-OMsignal	AngleFromVertical_mean/std-OMsignal
		Supine_mean-OMsignal	Supine_std-OMsignal
		AvgXAccel_mean/std-OMsignal	StdXAccel_mean/std-OMsignal
		AvgYAccel_mean/std-OMsignal	StdYAccel_mean/std-OMsignal
		AvgZAccel_mean/std-OMsignal	StdZAccel_mean/std-OMsignal
		AvgGForce_mean/std-OMsignal	StdGForce_mean/std-OMsignal
		Peak_max-Fitbit	Peak_max_change-Fitbit
		Peak_min-Fitbit	Peak_min_change-Fitbit
		Peak_minutes-Fitbit	Peak_minutes_change-Fitbit
		Peak_caloriesOut-Fitbit	Peak_caloriesOut_change-Fitbit
		Cardio_max-Fitbit	Cardio_max_change-Fitbit
		Cardio_min-Fitbit	Cardio_min_change-Fitbit
		Cardio_minutes-Fitbit	Cardio_minutes_change-Fitbit
		Cardio_caloriesOut-Fitbit	Cardio_caloriesOut_change-Fitbit
		Fat_Burn_max-Fitbit	Fat_Burn_max_change-Fitbit
		Fat_Burn_min-Fitbit	Fat_Burn_min_change-Fitbit
		Fat_Burn_minutes-Fitbit	Fat_Burn_minutes_change-Fitbit
		Fat_Burn_caloriesOut-Fitbit	Fat_Burn_caloriesOut_change-Fitbit
		Out_of_Range_max-Fitbit	Out_of_Range_max_change-Fitbit
		Out_of_Range_min-Fitbit	Out_of_Range_min_change-Fitbit
		Out_of_Range_minutes-Fitbit	Out_of_Range_minutes_change-Fitbit
		Out_of_Range_caloriesOut-Fitbit	Out_of_Range_caloriesOut_change-Fitbit
	Ctx.	Location, activity, interaction
Std.Life	Phys. Act.
		Stationary frequency	Stationary duration
		Walking frequency	Walking duration
		Running frequency	Running duration
		Working duration	Relaxing duration
		Exercise duration	Activity status change
	Audio	Silence frequency	Silence duration
		Voice frequency	Voice duration
		Noise frequency	Noise duration
		Conversation frequency	Conversation duration
		Phone dark frequency	Phone dark duration
		Audio status change
	Ctx.	Number of nearby devices	Number of unique devices
		Number of locations	Number of unique locations
		Number of WiFi	Number of unique WiFi
		Class experience	Class time

Funding Information

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No 2017-17042800005.