Imputation of missing time-activity data with long-term gaps: A multi-scale residual CNN-LSTM network model

Abstract

Despite the increasing availability and spatial granularity of individuals’ time-activity (TA) data, the missing data problem, particularly long-term gaps, remains as a major limitation of TA data as a primary source of human mobility studies. In the present study, we propose a two-step imputation method to address the missing TA data with long-term gaps, based on both efficient representation of TA patterns and high regularity in TA data. The method consists of two steps: (1) the continuous bag-of-words word2vec model to convert daily TA sequences into a low-dimensional numerical representation to reduce complexity; (2) a multi-scale residual Convolutional Neural Network (CNN)-stacked Long Short-Term Memory (LSTM) model to capture multi-scale temporal dependencies across historical observations and to predict the missing TAs. We evaluated the performance of the proposed imputation method using the mobile phone-based TA data collected from 180 individuals in western New York, USA, from October 2016 to May 2017, with a 10-fold out-of-sample cross-validation method. We found that the proposed imputation method achieved excellent performance with 84% prediction accuracy, which led us to conclude that the proposed imputation method was successful at reconstructing the sequence, duration, and spatial extent of activities from incomplete TA data. We believe that the proposed imputation method can be applied to impute incomplete TA data with relatively long-term gaps with high accuracy.

1. Introduction

Individuals’ daily activities occur at multiple locations and different time periods, which in turn results in extreme complexity in their travel-activity patterns (Hanson & Hanson, 1981). As a means of addressing the complexity, some researchers collected detailed information on the time and location at which individuals’ activities occurred and improved understanding of day-to-day travel patterns. For example, Hanson and Hanson (1981) and Pas and Koppelman (1986) have used daily travel records collected from about 150 study participants to examine individuals’ daily variability in out-of-home travel-activity patterns. More recently, Long, Zhang, and Cui (2012) and Egu and Bonnel (2020) used smart card data combined with household travel survey to understand the commuting patterns of urban dwellers. A number of recent studies have also used individuals’ daily travel-activity patterns gleaned from individuals’ GPS-based time-activity (TA) data to assess individuals’ exposure to environmental hazards, such as atmospheric pollutants (Dias & Tchepel, 2018; Glasgow et al., 2016; Nyhan, Kloog, Britter, Ratti, & Koutrakis, 2019), pesticides (Elgethun, Fenske, Yost, & Palcisko, 2003), and noise pollution (Duncan et al., 2017; Ma, Li, Kwan, Kou, & Chai, 2020).

Traditionally, TA data have been collected using travel surveys (Gerike, Gehlert, & Leisch, 2015), which still remains as a dominant means of collecting day-to-day travel demands despite their drawbacks. One of the limitations is associated with the cost and burdens to conduct surveys, which often restricts the size of the sample and shortens the study period. Second, respondents’ willingness to fill out a survey tends to get deteriorated over time due to survey fatigue. In addition to these two issues that often negatively affect the accuracy and reliability of responses, missing data problems have also been recognized as a serious challenge, particularly for long-term surveys (Badoe & Steuart, 2002; Sammer, Gruber, Roeschel, Tomschy, & Herry, 2018).

Some limitations inherent in survey-based TA data collection have been addressed by adopting location-aware devices and technologies, such as mobile phones and Global Positioning System (GPS). Specifically, mobile phone data, including call detail records (CDR), can help address the issue of small sample size as they typically contain the location information of millions of mobile phone users over months to years. However, missing data problems still remain with both mobile phone- or GPS-based TA data due to limited battery capacity and individuals’ privacy concerns (Barnett & Onnela, 2020; Kung, Greco, Sobolevsky, & Ratti, 2014; Li, Gao, Lu, & Zhang, 2019; Yoo, Roberts, Eum, & Shi, 2020).

Incomplete TA data (or TA with missing data) can be problematic if they were to be used as a primary data source from which travel-activity patterns are inferred. As Chen, Viana, Fiore, and Sarraute (2019) reported, individuals’ long-distance travel patterns inferred from TA data with missing records likely affect the conclusions drawn on observed mobility patterns. Similarly, the number of trips and day-to-day variability in the individuals’ travel patterns are likely to be underestimated if the TA data are incomplete (Pendyala, 1999; Yoo et al., 2020). Furthermore, these inaccurate representations of travel-activity patterns adversely affect the subsequent analyses, such as segmenting travelers (Crawford, 2020) and forecasting travel demand (Dharmowijoyo, Susilo, & Karlström, 2017; Raux, Ma, & Cornelis, 2016).

As a means of addressing missing data problems in TA data, one may pay attention to the regularity and predictability of activity patterns (Zhao, Jonietz, & Raubal, 2021). The high degree of spatio-temporal regularity in daily activity and travel patterns has long been recognized in studies either using traditional travel survey or mobile phone data (Gonzalez, Hidalgo, & Barabasi, 2008; Hanson & Huff, 1988; Huff & Hanson, 1986; Pappalardo et al., 2015; Schneider, Rudloff, Bauer, & González, 2013). Rooted in the regularity of daily mobility patterns, Zhang et al. (2014) and Liu, Wu, Wang, and Tan (2016) predicted individuals’ next whereabouts by taking into account the sequential patterns of visited locations. Similarly, Burkhard, Ahas, Saluveer, and Weibel (2017) identified the presence of prototypical patterns in daily activities and reconstructed individuals’ trajectories based on these observed patterns.

Clearly, the regularity and predictability of travel-activity patterns also enabled researchers to impute some spatial and temporal gaps in TA data. For example, Barnett and Onnela (2020) replaced missing GPS records for up to 30 min with a weighted resampling of observed time-location data, and Yoo et al. (2020) proposed an imputation strategy for less than 24 h of gaps in mobile phone-based GPS data using spatial and temporal contextual information of the data. Similarly, Han and Sohn (2016) imputed missing activity from a single trip chain inferred from smartcard data for two specific days using a hidden Markov model. It is worth noting that all these studies focused only on a single trajectory or TA data with relatively short-term gaps (i.e., a few minutes and hours). Therefore, they are not readily applicable to TA data with long-term gaps that are typically over 24 h. Alternatively, recent studies captured temporal dependency in trajectory data using recurrent neural network (RNN)-based models without strict assumptions (e.g., a multivariate normal distribution), and achieved a greater prediction accuracy than statistical models, such as Markov models (Feng et al., 2018; Gao et al., 2019; Liu et al., 2016; Zhou, Yue, Trajcevski, Zhong, & Zhang, 2019). However, they have not explored the presence of multi-scale temporal dependencies (both short- and long-term temporal dependency across days or weeks) in TA patterns nor used such information for imputation except Feng et al. (2018).

Lastly, individuals’ daily travel-activity patterns are complex as each individual engages in different activities and travels to attempt to satisfy individuals’ unique needs and desires (Dharmowijoyo et al., 2017), but also characterized by a high degree of predictability due to the spatial and temporal relationships between daily activities (Song, Koren, Wang, & Barabási, 2010). The interrelations among the daily activities need to be considered to effectively fill the missing records in the individuals’ TA data. This can be achieved by transforming the TA data into a low-dimensional representation that preserves the complicated dependencies among the activities based on their spatiotemporal context (Cao, Xu, Sankaranarayanan, Li, & Samet, 2019; Gao et al., 2017; Xu, Cao, Legg, Liu, & Li, 2019).

In the present study, we propose a novel imputation approach for long-term missing TA data over multiple days. The proposed TA imputation method explicitly takes into account the regularity present in daily activity patterns. Specifically, we developed a numerical representation system for each daily TA datum to represent the temporal, spatial, and contextual information of activities. Based on the numerical representation, we developed a neural network model that integrates multi-scale residual convolutional neural network (CNN) and stacked long short-term memory (LSTM) models to learn both short-term and long-term temporal dependencies across a series of historical daily TA observations and predicted the missing TA. The performance of the proposed imputation method was evaluated using mobile phone-based TA data.

2. Methods

The proposed imputation approach is based on a deep neural network model that predicts missing daily time-activity from multi-scale temporal dependencies learned from daily observations. As most deep learning architecture requires, the proposed model needs numerical inputs and outputs for the prediction, which take the form of numerical representation of daily time-activity data. Thus, the proposed imputation process operates in two steps. First, daily TA data are converted into numerical vectors that represent time, place, and type of activities for each day, using an embedding approach. Second, we used the numerical representation of daily time-activity to train the proposed deep neural network model that captures multi-scale temporal dependencies across the daily observations, and predicted missing daily TA based on the trained model.

2.1. Numerical representation of daily time-activity

We generated a sequence of time-activities for each individual on a daily basis as follows. First, we divided each day into time units of equal length and allocated to each unit the corresponding temporal, spatial, and contextual information as illustrated in Fig. 1(a). The length of time units can be determined based on the temporal resolution of TA data. For example, daily mobile phone-based trajectory can be converted into a sequence of 48 TA units (i.e., 30-min interval), denoted as A_u = (u,γ,α), u = 1, …, 48. Here u, γ, and α represent the index of time slot within a day, the place identifier, and the activity type, respectively.

Fig. 1.

(a) Numerical representation of daily TA data; (b) The architecture of CBOW word2vec neural network model.

From these sequences of time-activities, we extracted numerical features (also known as embedding vectors) that preserve sequential relationships between TA units using a continuous-bag-of-words (CBOW) word2vec model (Mikolov, Yih, & Zweig, 2013). The CBOW model was developed in the domain of natural language processing to obtain an efficient representation of words. Recognizing that a word in a sentence can be predicted by its context, such as its neighboring words, the CBOW model maps each word into a vector space based on the contextual meaning within the sentence through the learning process.

Based on an analogy with natural language processing, we used the CBOW model to obtain a numerical representation of each TA unit, denoted as w(A_u), taking into account the context TA units (θ(A_u)). We considered a sequence of daily time-activities as a sentence and the combinations of time ID, place ID, and activity type as a word in the sentence. In Fig. 1(a), for example, the target word was coded as (37, 1143, dining). The context of the target word was determined by what activities occurred one hour before and after the target TA occurred. For the target word in the example above, four context words were defined as: (35, 570, work), (36, 570, work), (38, 1143, dining out), and (39, 182, Home). If any short-term gaps are found in a daily TA sequence, we split the TA sequence into multiple sub-sequences based on the gaps (i. e., before and after the gaps). By doing so, we established the pairs of target-context time-activity units free of the short-term gaps. It is worth noting that the context TA units beyond the gaps were also incorporated in the sub-sequences if those TA units were within the range of context (i.e., one hour).

The CBOW word2vec model is a neural network that consists of an input, a hidden, and an output layer as illustrated in Fig. 1(b), and the pair of target-context words were used as the input and output of the embedding model. The input layer takes a dimension of N_{θ ·} V, where N_θ is the number of context TA units, and V denotes the size of unique TA units. The dimension of the hidden layer, denoted as m, is predefined as a model parameter. The input layer is connected to the hidden layer by a (V × m) of weight matrix, denoted as W_I hereafter. We multiply W_I to each context TA unit, λ _{∈ θ}(A_u), and obtain a vector representation, which is denoted as w(λ). Because there are multiple context TA units, the vector representation of context TA units were averaged to get the hidden layer, which can be written as: Φ(θ(A_u)) = Σw(λ)/N_θ. The (m × V) of weight matrix, denoted as W_O, transforms the hidden layer to the output layer. Lastly, we applied the softmax function and obtain the conditional probability of generating the target TA unit given the context TA units as:

$$p\left(A_u\mid \theta \left(A_u\right)\right)=\frac{exp\left(w\left(A_u\right)\mathbf{\Phi }\left(\theta \left(A_u\right)\right)\right)}{{\sum }_{A\in \text{Ω}}exp(w(A)\mathbf{\Phi }(\theta (A)))}$$ (1)

with Ω for the set of all TA units. To maximize the conditional probability, we updated model parameter values, such as embedding matrix elements, through a gradient descent algorithm iteratively. As a result, each TA unit was mapped into a low-dimensional vector space where TA units in a similar context are located close to each other. For a daily TA sequence representation, we summarized TA units within a day by averaging the embedding output vectors. The averaging approach was chosen based on its effectiveness in sentence embedding by averaging word-level embedding vectors that are included in the sentence as reported in the literature (Adi, Kermany, Belinkov, Lavi, & Goldberg, 2016; Wieting, Bansal, Gimpel, & Livescu, 2015).

2.2. Imputation of missing TA data

Fig. 2 presents the proposed deep neural network architecture to model both long- and short-term temporal dependencies from the numerical representation of daily TA observations. To effectively capture temporal patterns within different range of TA history, we used both two weeks of historical daily TA sequences (X_t−14, …, X_t−1) and one week of historical observations (X_t−7, …, X_t−1) to predict the daily TA embedding vector at time t (X_t). For both historical TA inputs, the same network structure was shared with multi-scale residual CNN and stacked LSTM network for the prediction, and the model outputs were integrated using trainable weights. The proposed model also extracts features from external factors, such as day-of-week effects, temperature, and precipitation, and further integrates them for the model prediction.

Fig. 2.

The proposed deep neural network architecture.

2.2.1. Multi-scale residual CNN

The multi-scale residual CNN model was developed to extract features varying across days at different temporal scales from observed daily TA sequences. The central idea of residual learning is to incorporate additive features with respect to the input (He, Zhang, Ren, & Sun, 2016). Similarly, we extracted hidden features and trends that exist in the adjacent days of TA sequences at multiple scales and integrated them into the original TA sequences. The temporal scale is determined by the kernel size of the one-dimensional convolutional layer, denoted as s, and we used the three convolutional kernels each with s=1, 2, and 3. We also added non-linearity by using LeakyReLU (LReLU) activation functions for each kernel output. The multi-scale residual function outputs were concatenated with the identity mapping of historical TA sequences in order to main features at both low- and high complexity levels.

2.2.2. Stacked LSTM network

In the present study, we developed a deep LSTM model that stacks bidirectional and uni-directional LSTM layers to effectively capture temporal dependence in the features extracted by multi-scale residual CNN, and predict the next-day TA from the historical observations Cui, Ke, Pu, and Wang (2018). The LSTM network is a type of recurrent neural network (RNN) model, and was developed to address vanishing gradient problem in the initial architecture (also known as vanilla RNN) using a gated structure that allows for relevant information to be forwarded throughout the long chain of sequences (Hochreiter & Schmidhuber, 1997). A deeper LSTM architecture can be accomplished by stacking multiple LSTM layers on top of each other, increasing the network capacity. Consequently, the stacked LSTM has shown superior performance to a single LSTM layer model, although the theoretical background is yet to be evident (Goldberg, 2016; Hermans & Schrauwen, 2013).

In the proposed model, the bi-directional LSTM layer enables the model to effectively learn the temporal relationships across daily TA embedding vectors by handling sequence data using two LSTM layers in forward and backward directions, separately. More specifically, the forward LSTM layer output is sequentially updated using the inputs from the most distant to the nearest history, whereas the backward LSTM layer uses the inputs in reverse order. The layer output information from both forward and backward layers are averaged. The model also stacks a uni-directional LSTM layer to predict X_t based on the forward dependency in the higher-level features of historical daily TA data learned from the bi-directional LSTM layer. Lastly, the model can capture the relationship across the dimensions of the embedding vector from the fully-connected (FC) layer on the top of the LSTM layer (Yoon, Zame, & van der Schaar, 2017).

2.2.3. Fusion of networks

The fusion layer in the model combines the two multi-scale CNN-stacked LSTM model outputs from both long- and short-historical TA sequences as ${\widehat{X}}_{t,f}$, as follows:

$${\widehat{X}}_{t,f}=W_1\circ {\widehat{X}}_{t,1}+W_2\circ {\widehat{X}}_{t,2}$$

where ∘ denotes Hadamard product, and W₁ and W₂ are trainable weight parameters that assigns different weights to the predictions from two historical TA observations, denoted as ${\widehat{X}}_{t,1}$ and ${\widehat{X}}_{t,2}$. The fused layer output is denoted as ${\widehat{X}}_{t,f}$.

2.2.4. External factors

Individuals’ daily TA behaviors can be affected by external factors such as day-of-week effects and weather conditions. In our model implementation, we considered the day-of-week, temperature, and precipitation as external factors, and used two stacked FC layers to include their effects (Zhang, Zheng, & Qi, 2017). We used one-hot encoded day-of-week factor, and standardized the temperature and precipitation variables in the range between 0 and 1 using Min-Max normalization. A FC layer extracts features from the external factors, followed by a LReLU activation layer to introduce non-linearity. We stacked another FC layer to transform the output from previous layer to have the identical shape with ${\widehat{X}}_{t,f}$, and added the layer output to ${\widehat{X}}_{t,f}$ in order to predict the final numerical representation of TA sequences at time t.

2.2.5. Model prediction

We used the trained model to predict the numerical representation of missing daily TA in the embedding vector space. The predicted feature values cannot be directly used to infer missing daily TA because the meaning of each embedding vector dimension is not readily understandable. In response, we assumed that daily TA sequences are highly routinized and form a regular pattern as evidenced in existing literature (Gonzalez et al., 2008; Huff & Hanson, 1986; Pappalardo et al., 2015). For each predicted numerical representation, we identified the most similar observation day that has the minimum Euclidean distance in the embedding space. We used the observed TA information to impute the missing daily TA.

3. Experiment and results

3.1. Data

We used mobile phone traces collected from 180 study participants from October 24, 2016 to May 22, 2017. Study participants consisted of residents in the Erie and Niagara counties in western New York who were recruited by letter, flyers, social media, and local news. The participants provided home and workplace addresses through the baseline survey when they enrolled in the study. Participants used their own mobile phones in the study and voluntarily shared their daily mobile phone traces via the application developed by our research team. They also reported the names and addresses of up to five places where study participants frequently visited each week. The study was approved by the Institutional Review Board of the university authors are affiliated with.

The collected mobile phone trace data do not contain contextual information about individuals’ activities, such as the place and the purpose of trips. To acquire such information on individuals’ activity patterns, we preprocessed the raw data so that we could extract basic information about individuals’ activities prior to the imputation. First, we identified the places where each individual’s daily activities occurred, hereafter referred to as activity places, using the location information (address) that the participant provided. They include the location of home, workplace, and up to five frequently visited places each week. Each study participant reported about 25 unique places (mean of 24.7 and the standard deviation of 18.9), and these reported places were matched with their mobile phone traces. Other activity places, which were not directly reported, were inferred from individuals’ trajectories. First, we identified places where participants were static for a prolonged time period (i.e., 30 min). We also considered mobile phone traces to be static if the instantaneous speed at the time of tracking was less than 1 m per second. The static records were aggregated if they were located within the same parcel boundary (http://gis.ny.gov/gisdata/inventories/details.cfm?DSID=1300). We incorporated positional uncertainty of mobile phone GPS-based data by applying a spatial buffer around the parcel boundaries with a distance of 100 m. Finally, we measured the duration of stay at each parcel from the aggregated records, and defined it as a place of activity if the individual was static for more than 30 min.

Next, we classified each individual’s observed activity places into one of 11 ‘representative’ activity places. For the present study, we defined a total of 11 representative activity places based on their frequency of visits (i.e., how commonly found in individuals’ observed activity places). These activity places consist of home, work, shopping, social, recreational, educational, dining out, religious, healthcare, personal, and others. For the linkage, we used the three databases as a complementary data source, at which the following definition of data and type of places were used: Google Places database that classifies place types (https://developers.google.com/places/supported_types), New York State GIS parcel data that provides a statewide uniform property type classification codes (https://www.tax.ny.gov/research/property/assess/manuals/prclas.htm), and infoUSA mailing list data that include name, address, and NAISC industry code of 33,912 businesses and 390,140 residences within the study region.

Based on the inferred contextual information of time-activity data, we converted each participant’s daily mobile phone traces into a sequence of 48 TA units. We selected 30 min as the unit of TA sequences because the average time interval of mobile phone traces was 31.4 min. If multiple activities existed in a single 30-min time unit, we assigned the activity where the duration of stay was the longest within the time unit. Meanwhile, the time-activity sequences were based on the activities at home, work, and visited places, but the activities associated with travel behaviors were not included.

We also estimated the daily average temperature and precipitation at each activity place in order to incorporate them in the modeling of daily time-activity as external factors. We obtained the meteorological data for 53 weather stations in the study area from Climate Data Online (https://www.ncdc.noaa.gov/cdo-web/), and matched each activity place with the closest weather station to estimate the daily weather conditions. The daily minimum temperature and precipitation ranged − 17.8–27.8 °C and 0–74.7 mm, respectively, during the study period.

3.2. Setup of experiment

We evaluated the performance of the proposed imputation approach using a 10-fold out-of-sample cross-validation method. Specifically, for each participant, we selected a subset of TA data that did not contain serious missing records (i.e., 6 h of no record) for 98 consecutive days. The first 14 days of TA data were not included in the cross-validation because the proposed deep neural network calls for two weeks of historical data for model training. We randomly partitioned the remaining 84 days of TA data into 10 groups for the cross-validation. We held out each group (i.e., 8 or 9 days of TA data) in turn as a test set, and used the remaining groups (i.e., 77 or 78 days) for training the model (a training set). In summary, we trained the proposed imputation model with training sets and validated the model prediction (imputed TA records) with the ground-truth TA information in the test sets. We quantified the imputation error by calculating the gap between imputed TA records against the observed TA records via multiple metrics, and any missing units in the ground truth were excluded from calculating imputation errors.

The two-step imputation method was developed in R (R Core Team, 2020), utilizing python modules via the reticulate R package. We trained the word2vec model using the gensim python module, and developed the deep neural network model using the tensorflow version 2.3 and Keras deep learning API version 2.4.3.

During the deep neural network model fitting, the training data set was further split into 80% for training and 20% for validation. The model loss function was defined as the mean standard errors between the embedding vector of observed daily TA sequences in the validation set and the predicted values. We used the Adam optimizer with learning rate of 0.001 to minimize the model loss and train the proposed deep neural network because the optimizing algorithm combines the advantages of precedent gradient descent-based optimizers and outperforms them (Prilianti, Brotosudarmo, Anam, & Suryanto, 2019). We allowed 200 iterations to train the model and used the early stopping mechanism to avoid overfitting, which monitors the model loss and stops training if the validation loss did not decrease.

3.3. Numerical representation of daily time-activity

As the first step of the imputation, we converted each study participants’ observed daily TA data into a low-dimensional numerical vector. This low-dimensional representation in an embedding vector enabled us to compare and classify observed daily TA information efficiently. For the purpose of illustration, we randomly selected a study participant and obtained one’s TA for 10 days. We converted the observed TA data into a three-dimensional embedding numerical vector and presented the result in Fig. 3.

Fig. 3.

A three-dimensional embedding vector of a randomly selected participant’s time-activity data for 10 days.

Through the embedding process, the daily time-activity sequences are represented with numerical vectors of the dimension m. A high dimension of embedding space is typically associated with greater details of the representation. In the present study, we used m to refer to the dimension of embedding vectors but also to specify the number of features used in the subsequent deep learning prediction model if the embedding vectors of daily TA sequences were used as an input. Typically, over-fitting is a problem in machine learning, because it harms the predictability of the model, and it often occurs when the number of features is too large. To avoid this problem by selecting a low but efficient dimensionality for representing TA sequences, we conducted a sensitivity analysis and selected an optimal value of m.

We evaluated the effectiveness of the embedding process using a sequential alignment algorithm that has been widely used to quantify the similarity between two sequences with respect to both the frequency and order of components in the sequence (Joh, Arentze, Hofman, & Timmermans, 2002; Kwan, Xiao, & Ding, 2014; Wilson, 1998). Specifically, we calculated edit distance (also known as Levenshtein distance) between the pairs of daily time-activity sequences, measured by the minimum number of processes (i.e., insertions, deletions, and substitutions of elements) to align one sequence to the other. For the calculation of edit distance, we compared the daily sequences of place ID and activity type, which follows the order of time index. The details on the edit distance calculation is provided in Appendix A. We focused on the sequential order of TA units as an evaluation criterion for numerical representation based on the assumption that daily TA sequences with a similar sequential pattern of TA units is likely to be located nearby in the embedding space because the embedding model takes into account TA units that occurred at a similar time as context. Thus, we compared the edit distance from daily TA sequences and the Euclidean distance from embedding outputs by calculating the correlation between them. We repeated the calculation for any possible pair of daily TA sequences. The correlation coefficient was used to evaluate the performance of numerical representation. Fig. 4 shows the boxplot of correlation values from all study participants by the dimension of embedding vectors m. Given that each participant had 719 unique TA units (i.e., combinations of time ID, place ID, and activity types) on average with a standard deviation of 255 TA units, we considered a wide range of values for m = {10,…,150}. We observed a high correlation between the two distance metrics, and we selected m = 60 where the 1st quartile of correlation coefficient values was greater than 0.9.

Fig. 4.

Correlation between the two distance metrics for each pair of daily time-activity, calculated from the inputs and outputs of the embedding process, by different size of the embedding vector space. Blue dot represents mean value. Correlation of 0.9 is indicated by a dotted horizontal line. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

3.4. Imputation of missing daily time-activity

Based on the numerical representation of daily TA sequences, we imputed missing data in the cross-validation test data using the multi-scale CNN-stacked LSTM model. In total, 725,760 missing TA units (180 participants × 84 days × 48 units/day) were imputed for the entire data set. The prediction accuracy of the imputation was 84%, which was calculated as the percentage of correctly imputed TA units over the total number of missing TA units. The imputation allowed recovering useful information about activity patterns, such as time allocation for different activity types. Table 1 shows the activity type of each missing TA unit in the cross-validation test data and imputation results. Overall, the distribution of activity types from the imputation was similar to that of the test data, but did not perfectly match. In particular, the proportions of home and work activities in the imputed TA data were 2.7% and 0.8% higher than the same information in the test set, respectively, while other activity types were lower with less than 0.5% of difference. The discrepancies between the imputation results and test data were considered errors in the imputation results, and they were used to evaluate the performance of imputation in terms of reconstructing activity patterns.

Table 1.

Activity type of missing and imputed time-activity units.

Activity type	Time-activity unit (%)		Evaluation (Δ%p)
	Missing	Imputed
Home	527,222 (72.3)	544,129 (75.0)	+2.7
Work	98,682 (13.6)	104,421 (14.4)	+0.8
Shopping	18,326 (2.5)	14,773 (2.0)	−0.5
Social	26,189 (3.6)	22,196 (3.1)	−0.5
Recreational	13,589 (1.9)	10,751 (1.5)	−0.4
Educational	8,410 (1.2)	6,403 (0.9)	−0.3
Dining out	9,995 (1.4)	7,183 (1.0)	−0.4
Religious	2,728 (0.4)	1,428 (0.2)	−0.2
Healthcare	8,211 (1.1)	5,138 (0.7)	−0.4
Personal	8,530 (1.2)	5,536 (0.8)	−0.4
Others	3,837 (0.5)	1,409 (0.2)	−0.3

3.5. Performance evaluation

In the model performance evaluation, we focused on three aspects of activity patterns: the sequential patterns of time-activities, the time allocation per activity type, and the averaged travel distance. For each case, we quantified the differences between predicted and observed activity data using three metrics. Specifically, we measured the differences between sequential patterns by calculating the edit distance (ε_Q), the total gap between the time spent for each activity type using ε_T, and the differences in the travel distance measured with the radius of gyration, denoted as ε_S. The full description of performance evaluation metrics can be found in Appendix B. For each evaluation criteria, we calculated the imputation error as the difference between observed and imputed time-activity sequence per day, and averaged the imputation errors for the entire data set.

To examine the robustness of the evaluation, we compared the performance of the proposed imputation method with seven different imputation methods as a baseline for benchmarking (See Appendix C). The numerical representation of daily TA was not required to implement the last-observation-carried-forward (LOCF), last same-day-of-week (DOW), and most frequent (MF) methods. However, the other sequential data imputation methods, such as multiple imputation by chained sequence (MICE), LSTM, and stacked LSTM models, used the embedding output to predict the numerical representation of missing daily TA.

The evaluation results are summarized in Table 2 that indicated that the proposed imputation method outperformed all other baseline models across all the performance evaluation criteria. For example, replacing missing TA data with the observation from the day before (or the latest day being observed) in LOCF yielded higher imputation errors of 54%, 41%, and 25% than the proposed model in terms of ε_Q, ε_T, and ε_S, respectively. Meanwhile, when the missing daily TA was replaced with the TA of the same day-of-week in previous weeks (DOW), the imputation errors were reduced. We found that imputing the missing TA for each time unit in the daily sequence with the most frequently observed TA from the historical observations using the MF method yielded the highest accuracy. The reconstructed based on the most frequent historical record instead of a single day record substantially reduced the discrepancy in the edit distance (ε_Q).

Table 2.

Performance evaluation.^*

Imputation models	Average imputation error
	εQ	εT (mins.)	εS (km)
Proposed model	8.4	417	2.79
LOCF	13.0	591	3.48
DOW	11.5	529	3.25
MF	9.9	554	3.21
MICE	10.2	490	3.10
RNN	16.2	756	4.02
LSTM	9.6	470	3.03
Stacked LSTM	8.9	450	2.91

^*ε_Q: edit distance in activity sequence; ε_T: total absolute difference in activity duration; ε_S: absolute difference in ROG.

The sequential data imputation methods that used embedding outputs to predict the numerical representation of missing daily TA also improved the performance of imputation. Specifically, the MICE imputation algorithm performed better than the aforementioned approaches except the ε_Q from MF. The LSTM model further improved the imputation performance, whereas the vanilla RNN model performed the worst among the baseline imputation methods due to the failure of capturing long-term dependency. Lastly, the proposed model performed better than other RNN-based models in all three performance evaluation criteria.

We tested the significance of the difference between the imputation errors from proposed imputation approach versus baseline models using the Kolmogorov-Smirnov test. The result showed that the imputation errors from the proposed method was significantly smaller than all the baseline methods (p < 0.05).

3.6. Sensitivity analysis

One of the premises of the proposed imputation method is the regularity of travel-activity patterns. However, the degree of regularity may vary per individual, and the performance of the proposed imputation method can be affected. To investigate the effect of regularity in individuals’ daily TA on the imputation performance, we conducted a sensitivity analysis. First, we measured the regularity of each individual’s TA patterns using an entropy index (Goulet-Langlois, Koutsopoulos, Zhao, & Zhao, 2017; Song, Qu, Blumm, & Barabási, 2010; Xu, Belyi, Bojic, & Ratti, 2018) (See Appendix D). Smaller values of the entropy index depict less randomness, and consequently higher regularity in TA. The minimum entropy measure is zero when the TA pattern is determined by only a single activity place with no uncertainty. We also quantified the association between individuals’ regularity of daily TA and the performance of imputation using a simple linear regression analysis. We fitted a regression model for each performance evaluation metric.

The degree of regularity in each participant’s TA patterns was characterized by calculating the entropy index, which ranged from 0.04 to 1.15 with a mean of 0.64. Using the entropy index, we assessed if there exists a linear association between the regularity in individuals’ daily TA patterns and imputation performance using a regression analysis. As summarized in Fig. 5, the result indicated that imputation errors are associated with the regularity in TA patterns.

Fig. 5.

Entropy index versus imputation errors of ε_Q (a), ε_T (b), and ε_S (c), overlaid with a fitted regression line and 95% confidence interval. Marginal distribution of each variable is also displayed on each axis.

4. Discussion

We proposed a novel imputation approach to address missing data problems in time-activity sequence data, particularly with long-term gaps. The proposed imputation method is based on both an embedding model and a deep neural network that combines a multi-scale residual CNN and a stacked LSTM network. The proposed multi-scale deep learning approach enables researchers to impute long-term gaps in TA sequences with high accuracy, while addressing the limited applicability of existing imputation methods that focus only on short-term gaps. In fact, this is the first imputation method that explicitly takes into account both the long- and short-term temporal dependencies present in individuals’ daily routine activities.

Specifically, we designed a numerical representation system to obtain a meaningful representation of individuals’ daily TA patterns from their mobile phone-based GPS tracking data with a high granularity of the location and temporal information. Word embedding model-based representation systems have been used in a few recent studies, although they were primarily used for the evaluation of the similarity between spatial locations (Crivellari & Beinat, 2019; Zhu et al., 2019) or extraction of location-specific features (Xu et al., 2019; Zhang et al., 2021). More importantly, the contextual information of TAs, including time and type of activities, was not incorporated, mainly because they have used TA data used CDR or check-in data-based TA data that contain less accurate (approximated) time-location information at coarser temporal and spatial resolutions than mobile phone GPS-based TA data. It should be noted that CDR provides location information of phone users based on the nearest cell tower location and location data were collected only if mobile phones were in use for active communications, such as calls or texts (Çolak, Alexander, Alvim, Mehndiratta, & González, 2015; Yoo et al., 2020). In contrast, mobile phone GPS-based traces collected at high spatio-temporal granularity provides rich contextual information about TAs, including when and where the activities occurred and what type of activities they were. The proposed imputation model takes into account this detailed information in the process of numerical representation.

We also developed a deep neural network that handles the unique challenges associated with modeling temporal dependencies across daily time-activity at multiple time scales (Liu et al., 2018), integrating multi-scale residual CNN and stacked LSTM network. A similar approach has been used in financial modeling (Guo, Lei, Ye, & Fang, 2021), although we extended the model by incorporating multi-scale temporal dependencies found in the varying lengths of TA history (e.g., a greater effect of recent history on the next time-activity prediction than a long-term history; see Barbosa, de Lima-Neto, Evsukoff, and Menezes (2015)). Moreover, our proposed model exploits the high network capacity of stacked LSTM layers to capture temporal dependencies more effectively. Lastly, the deep neural network explicitly takes into account the effects of external factors such as temperature, precipitation, and day-of-week for the prediction of daily TA sequences. The multi-scale residual CNN-stacked LSTM network allowed us to capture temporal dependencies present in TA sequences across days and weeks, and consequently achieved superior imputation performance in comparison to baseline methods.

The proposed imputation method has practical implications for future studies that utilize individuals’ TA data as a primary data source to infer individuals’ daily travel-activity patterns. Although the imputation results were inclined to more frequently observed activities such as home and work, we found that the proposed imputation method was able to reconstruct the sequences, duration, and spatial extent of activities from TA data that contain long-term gaps with high accuracy. Thus, our proposed method could reduce potential errors attributable to missing data in the inference of individual spatial and temporal activity patterns from TA data. Further investigations can be performed to evaluate the effect of missing TA data imputation on mobility-based analysis, such as assessment of personal exposure to air pollution using the location of activity places and duration of stay at each place (Glasgow et al., 2016; Park & Kwan, 2017; Yoo, Pu, Eum, & Jiang, 2021; Yoo, Rudra, Glasgow, & Mu, 2015). Meanwhile, we believe that imputation yielded a relatively small number of types of activities instead of a wide range of different activities because of our use of the CBOW word2vec model. A CBOW word2vec model has been widely used in the field of natural language processing for data classification, event detection, and emotion analysis, but Mikolov et al. (2013a) have noted that a CBOW word2vec model tends to represent frequent words more effectively than other models, such as skip-gram word2vec model. Our imputed results are consistent with previous findings, given that TA units associated with infrequent activity types were not well-represented. We believe that alternative embedding models that provide a better representation for less frequent TA units might address this under- or over-imputation of activities in future studies.

It is worth noting that the framework of the proposed imputation model was ground on the premise of high regularity present in daily activity patterns. As it was designed for, the proposed imputation method performed substantially better for the missing data from individuals with highly regular TA patterns. Because the performance of the proposed imputation method depends on the degree of regularity in activity patterns, caution should be taken in the interpretation of the results.

Related to that, the present study has several limitations. First, the TA sequences used in this study likely contain potential errors associated with mapping original mobile phone GPS-based data into daily time-activities. Second, the performance of the proposed imputation method was evaluated by relying on a single data set. Further investigation is needed to generalize the usability of the imputation method for other TA data sets, such as multi-week travel surveys. Lastly, the performance of our imputation method might vary by the granularity of the daily TA sequence. In the present study, we selected 30 min as a time unit to assign spatial and contextual information of activity, taking into account the average time intervals of location data available. As we included a single activity (i.e., the longest duration of stay) for each time unit, the information on other activities that might have occurred at the same time unit was lacked. In other words, the 30 min of time interval might not be enough to maintain detailed activity information during a day, thus underestimating daily TA variability. On the other hand, excessively fragmented TA units using a short time interval would prevent the imputation model from capturing meaningful routines of daily activity patterns and temporal dependency across them. Thus, if TA data are available with a finer temporal resolution, examining the effect of TA unit segmentation on the imputation performance could provide additional insights.

5. Conclusion

We developed an imputation method to predict missing daily TA information, taking into account a high degree of regularity in daily activity patterns. To take into account sequential relationships embedded in activities within a day and multi-scale temporal dependence across days, we applied the word embedding model and developed a multi-scale CNN-stacked LSTM network architecture. We found a superior imputation performance of the proposed method to existing imputation approaches. Our imputation method performed more effectively for the individuals who had higher regularity in activity patterns.

Appendix A. Edit distance

By design, each daily TA sequence has an equal length of 48 elements. Consider two daily TA sequences a and b. We denote d_(i,j) as the edit distance between the first i elements of a and the first j elements of b. We used a dynamic programming algorithm to compute d_(i,j). Here, d_(i,j) is first computed for small values of i and j, and subsequently for larger i and j based on the previous computation results. The computation algorithm is initialized with d_(i,0) = i and d_(0,j) = j. For 1 ≤ i, j ≤ 48, we compute d_(i,j) as:

$$d_{(i,j)}=\{\begin{array}{l}{d}_{(i-1,j-1)}\text{for}{a}_i=b_j\hfill \\ 1+min\{\begin{array}{c}{d}_{(i-1,j)}+1\\ d_{(i,j-1)}+1\\ d_{(i-1,j-1)}+2\end{array}\text{for}{a}_i\ne b_j.\hfill \end{array}$$ (A.1)

As a result, d_(48,48) is the edit distance between a and b.

Appendix B. Performance evaluation metrics

We evaluated the performance of imputation by comparing the imputed time-activity with the TA sequences in the test data set based on the following three metrics: ε_Q, ε_T and ε_S. First, ε_Q quantifies the difference in sequential patterns of time-activities. Specifically, we calculated the edit distance between the TA sequences in the test data set and the imputation results based on Eq. (A.1). Second, we examined a temporal aspect of imputed TA by comparing the time allocated for different types of activities. The total time gap between imputed and observed TA data was calculated as ${\epsilon }_T={\sum }_{k=1}^{11}\left|T_k-{\widehat{T}}_k\right|$, where T_k and ${\widehat{T}}_k$ denote the duration time for the kth activity type in the ground-truth and imputed TA, respectively. Lastly, we evaluated a spatial aspect of imputed TA by measuring the difference in radius of gyration (ROG) values derived from the imputed and observed TA (ε_S). The ROG quantifies a characteristic distance traveled by an individual. Radius of gyration can be calculated as:

$$\sqrt{\frac{1}{T}\sum _{l=1}^L{t}_l{\left({\mathit{c}}_l-{\mathit{c}}_{\mu }\right)}^2}$$ (A.2)

Here, c_l denotes the geographical coordinates of the lth activity place, c_μ is the mean center of c_l, and L denotes the total number of activity places. The geographical coordinates of activity places were weighted by the time spent at each location, denoted as t_l, and $T={\sum }_{l=1}^L{t}_l$ is the total activity duration time.

Appendix C. Baseline methods

We compared the performance of the proposed imputation method to the following 7 baseline methods:

• last observation carried forward (LOCF): the missing daily TA was replaced with the TA data of the most recent observation day (Mavridis et al., 2019);

• last same day-of-week (DOW): the TA data in the closest same day-of-week were selected to replace missing TA based on the cyclical pattern of activity patterns (Csáji et al., 2013);

• most frequent (MF): for each time unit, the most frequently observed TA from the historical data replaced the missing TA;

• multiple imputation by chained sequence (MICE): a principled statistical multiple imputation algorithm for time-series data was used to impute the numerical representation of missing daily TA (van Buuren & Groothuis-Oudshoorn, 2011);

• RNN: the forward sequential transitions of daily TA was modeled by a vanilla RNN model;

• LSTM: a single LSTM layer model was employed to capture long-term dependency across the numerical representation of historical TA data;

• stacked LSTM: a bi-directional LSTM is used to learn temporal patterns in both forward and backward directions, and a uni-directional LSTM layer is stacked for a prediction (Cui et al., 2018).

Appendix D. Entropy index

We used an entropy index, denoted as H, to measure the regularity of each individual’s TA patterns. For the jth individual, the entropy index H_i was calculated as

$$H_j=-\sum _{l=1}^{L_j}{p}_{jl}log\left(p_{jl}\right).$$ (A.3)

Here, p_jl denotes the proportion of time spent at the lth activity place over the total activity duration time at L_j places.