Mining the Home Environment

Abstract

Individuals spend a majority of their time in their home or workplace and for many, these places are our sanctuaries. As society and technology advance there is a growing interest in improving the intelligence of the environments in which we live and work. By filling home environments with sensors and collecting data during daily routines, researchers can gain insights on human daily behavior and the impact of behavior on the residents and their environments. In this article we provide an overview of the data mining opportunities and challenges that smart environments provide for researchers and offer some suggestions for future work in this area.

1. Introduction

Modeling and analyzing the behavior of humans is a valuable tool for computational systems. The modeling of human behavior provides insights on human habits and their influence on health, sustainability, and well being. Many automated systems that focus on the needs of a human require information about human behavior. Researchers are recognizing that human-centric technologies can assist with valuable functions such as home automation, health monitoring, energy efficiency, and behavioral interventions. The need for the development of such technologies is underscored by the aging of the population, the cost of health care, and the rising concern over resource usage and sustainability.

Until recently, gathering data about human behavior meant conducting surveys or placing observers in the sphere of other humans to record observations about human behavioral habits. Since the miniaturization of microprocessors, however, computing power has been embedded in familiar objects such as home appliances and mobile devices; it is gradually pervading almost every level of society. Advances in pervasive computing have resulted in the development of unobtrusive, wireless and inexpensive sensors for gathering information in everyday environments. When this information is analyzed using data mining techniques, this power can not only be integrated with our lives but it can provide context-aware, automated support in our everyday environments. One physical embodiment of such a system is a smart home. In a smart home, computer software plays the role of an intelligent agent that perceives the state of the physical environment and residents using information from sensors, reasons about this state using artificial intelligence techniques, and then selects actions that can be taken to achieve specified goals [1].

During perception, sensors that are embedded into the home generate readings while residents perform their daily routines. The sensor readings are collected by a computer network and stored in a database that the intelligent agent uses to generate more useful knowledge, such as patterns, predictions, and trends. Action execution moves in the opposite direction – the agent selects an action and stores this selection in the database. The action is transmitted through the network to the physical components that execute the command. The action changes the state of the environment and triggers a new perception/action cycle.

Because of their role in understanding human behavior and providing context-aware services, research in smart environments has grown dramatically in the last decade. A number of physical smart environment testbeds have been implemented [2]–[10] and many of the resulting datasets are available for researchers to mine. The wealth of data that is generated by sensors in home environments is rich, complex, and full of insights on human behavior. In this article we highlight advances that have been made in data mining smart home data and offer ideas for continued research.

2. Smart Home Data

One reason why the vision of ambient intelligence is powerful is that it is becoming very accessible. Sensors are available off-the-shelf to localize movement in the home, provide readings for ambient light and temperature levels, and monitor interaction with doors, phones, water, appliances, and other items in the home (see Figure 1). There are two broad categories of smart home sensors based on where the sensors are placed, namely environmental and wearable sensors. Environmental sensors are embedded in the environment and can detect changes (and types of changes) to the environment as a result of human interactions. Examples of these sensors include video cameras, passive infra-red (PIR) motion detectors, temperature sensors, magnetic reed switches, etc. On other hand wearable sensors are located on the smart home residents themselves and monitor changes into measurement values resulting from human motion and location. Sensors such as accelerometers, gyroscopes, magnetometers and Wi-Fi strength detecting sensors, to name a few, are examples of wearable sensors. These categories of sensors measure specific nuances of human interactions in the smart home and thus are used for different applications. The sensors provide an awareness of the resident context (location, preferences, activities), the physical context (lighting temperature, house design), and time context (hour of day, day of week, season, year). Sensors are used that provide an awareness of the resident context (location, preferences, activities), the physical context (lighting temperature, house design), and time context (hour of day, day of week, season, year).

Figure 1.

Example smart home sensors including 1) a passive infrared motion detector [4], 2) a magnetic reed switch door sensor [4], 3) a pressure sensor to detect if a chair or bed is occupied [18], 4) a temperature sensor to detect if the stove is being used [4], 5) a water usage sensor [19], 6) an electricity consumption sensor to detect appliance usage [20], 7) a microphone array [21], 8) a smart phone [22], and 9) a video camera [23].

Video camera based approaches (commonly referred to as vision based approaches) extract information regarding the interactions and changes taking place in the home by employing a cascade of techniques such as motion segmentation, human detection, action recognition and activity extraction [11]–[14]. These techniques analyze the captured visual data and generate human-interpretable information about activities and interactions in the home. While video cameras and microphone arrays provide a rich source of data, they are not always widely accepted as a means of monitoring human activities [15].

One type of sensor that is more readily accepted and is commonly found in smart homes is an infrared motion sensor. Wireless motion sensors such as those shown in Figure 1 contain a PIR sensor that measures nearby heat-based movement, a small chip that generates a message to send when the change is sufficiently large, and a radio to transmit the message to a central computer.

Other wireless sensors generate messages when different types of events are detected. Multiple motion sensors are embedded in the home environment and the sequence of motion sensor events resulting from heat-based (human) movement is used to extract information about interactions in the smart home, such as mobility and resident activities [16]. In addition, magnetic door closure sensors indicate when the magnetic circuit is completed or broken indicating door close or open status and shake sensors are often attached to objects and indicate when vibration is sensed, possibly signifying interaction with the object. Another popular home-based sensor relies on Radio Frequency Identification (RFID) technology [17]. RFID tags are affixed to objects of interest and register their presence within the range of an RFID reader. These tags can be used to detect some types of manipulation with the objects.

In addition to sensors that are embedded in the environment, data can be collected in the home by other means. A number of researchers and companies are developing technologies to recognize when devices are being used through their energy consumption [20], and similar means can be used to detect the use of water and gas [19]. Data collected from these sensors can be mined and used to develop applications that promote the design of energy-efficient appliances and sustainable resident behaviors.

Wearable sensors placed on the home residents can provide more fine-grained data through accelerometers that measure body movement [24]–[28]. Applications such as ambulation monitoring and tracking body energy expenditure process the data obtained from the wearable sensors. These sensors are positioned directly or indirectly on the resident’s body and track movements and physiological states of the resident. If residents carry smart phones with them in the home, a vast amount of data can be collected via the phone’s microphone, video, accelerometer, gyro, and magnetometer [29]–[33]. This data can be correlated to the different contexts associated with the resident such as location and activity.

The proliferation of sensors in the home results in large amounts of raw data that must be analyzed to extract relevant information. Data mining plays a pivotal role in the process of seeking bits and pieces of information that provide useful observations on resident behavior and the state of the home. Here we focus on a subset of data mining research on smart environment data that includes activity recognition and discovery, multiple resident tracking, correlation of behavior with other parameters of interest, and mining behaviors across a population.

3. Behavioral Complexity

In order to understand human behavior in a home environment, researchers need to be able to describe observations using a common vocabulary. In most cases, the vocabulary consists of commonly-understood activities that individuals perform in their home. However, there exists quite a disparity between the types of behaviors that are discovered, modeled, recognized and analyzed using data mining techniques.

Figure 2 characterizes the continuum of activity complexities that have been explored in the literature, from the perspective of the difficulty of sensing, detecting, and performing the activities [33]. At the lowest level is the recognition of basic body poses or gestures. This has most widely been explored in the computer vision literature [34][35][36], but success has also be achieved with wearable sensors [26] and the resulting technologies have been used to control devices found in smart homes [37].

Figure 2.

Continuum of activity classes for recognition.

The next level of activity complexity is ambulation. In this case sensors (most commonly wearable sensors such as accelerometers) are used to recognize single-movement or cylic-movement ambulation activities such as walking, running, sitting, standing, climbing, lying down or falling [27][38]. More recently, researchers have been exploring the use of smart phones equipped with accelerometers and gyroscopes to track ambulatory movements and gesture patterns as well [29][30]. Most of these approaches have been able to model such simple activities and recognize them in real time primarily by characterizing the constituent movements [39] using a sliding window protocol. Since the movement information related to ambulatory activities is typically well represented within a window of the data from accelerometers with a high sampling rate, a sliding window based approach is appropriate for real-time processing of these behavior classes.

As we move along the continuum, the classes of activities become more complex. For example, researchers are very interested in using smart home technologies to monitor the ability of residents to perform important Activities of Daily Living (ADLs) and instrumented ADLs (iADLs) because of their role in health management and living independently [40]–[44]. These activities include common daily tasks such as cooking, grooming, and taking medicine. Unlike the previous activity classes, ADL activities are complex, containing sequences and cycles of subtasks. These activities include common daily tasks such as cooking, grooming, and taking medicine. Unlike the previous activity classes, ADL activities are complex, containing sequences and cycles of subtasks. These activities are an expression of the interaction between different objects and humans through complex movements. A denser sensing infrastructure is required to detect and recognize these activities. In fact, some researchers explore representing such activities using formal grammars [33], [45], [46]. The individual symbols in these grammars actually represent entire activities from a lower complexity class such as gestures and ambulation; ADL activities can be described as a combination of such activity classes.

Finally, we reach one of the most complex activity classes, that of social interaction. In smart homes, monitoring social activities and interactions has received less attention but is widely recognized as being fundamental to human health. Social signals are recognized as determinants of human behavior [47], indicators of work productivity [48], and essential for cognitive and physical health [49], [50]. Recognizing interactions between individuals and group activities again relies on the ability to recognize the less complex classes of activities on the continuum. In addition, new types of subtasks are introduced in multi-person settings that introduce additional data mining challenges [51].

4. Activity Recognition

Activity recognition is a challenging and well researched problem. The goal of activity recognition is to map a sequence of sensor observations to a label indicating the activity class to which the sensor sequence belongs. Many data mining and machine learning techniques have been designed and evaluated for their ability to recognize activities in settings that range from scripted activities in laboratories to real-time detection of activities in everyday homes with multiple residents.

The typical framework for activity recognition algorithms is illustrated in Figure 3. The activity recognition problem is typically posed as a supervised learning problem with the goal of learning the mapping that associates features extracted from the sensor data to the underlying human activities that resulted in the sensor data in the first place.

Figure 3.

Commonly used framework for activity recognition.

Wearable sensor-based activity recognition approaches extract various statistical and spectral features such as mean, variance, maximum, spectral entropy, and dominant frequency component to characterize the sensor data [27], [30], [38], [39]. Typically fixed length window sizes (each containing a few seconds of data) are employed to sample the wearable sensors such as accelerometers. The data windows are then transformed into features. In applications such as tracking human ambulation, this is a robust approach as the information required to describe an activity is captured within a single window. The features are then used in conjunction with the activity labels for training supervised learning techniques.

Environmental sensor based activity recognition follows a similar track of transforming the raw sensor data into a richer representation of features. When dealing with PIR sensors, example features include the count of the sensor firings within the duration of an activity, contextual features such as location (for example, Bedroom, Dining room and Couch) of the PIR sensors, the sequence of sensor firings, motion trajectories estimated from videos as well as object interaction information extracted from images and RFID data [16], [52]. These features are then used to build activity models using supervised learning approaches.

The supervised learning that have demonstrated success for the task of learning activity models can be broadly categorized into template matching or transductive techniques, generative, and discriminative approaches. Researchers have explored these approaches for both wearable and environmental sensor based activity recognition. Template matching techniques employ a kNN classifier to label sensor sequences based on either the distance that is computed between a test window and training window based on Euclidean distance in the case of a fixed window size [53] or dynamic time warping in the case of a varying window size [34]. Template matching techniques have focused on a static set of spatial-temporal features [34] as well as description of resident trajectories [54].

The simplest generative approach is the naïve Bayes classifier, which has been utilized with promising results for activity recognition in smart homes [11][27][55][52]. Naïve Bayes classifiers model all sensor readings as arising from a common causal source: the activity, as given by a discrete label. Using a bag-of-words approach, the dependence of the sensor observations on activity labels is modeled as a probabilistic function that can be used to identify the most likely activity given the set of observed sensor features. Despite the fact that these classifiers assume conditional independence of the features given the activity class, they are robust when large amounts of sample data are provided. Nevertheless, naïve Bayes classifiers do not explicitly model any temporal information, which is usually considered important in activity recognition.

The hidden Markov model (HMM) is arguably the most popular generative approach that models temporal relationships. A HMM is a probabilistic model consisting of a set of hidden states coupled in a stochastic Markov chain, such that the distribution over states at one point in time depends only on the state values for a finite number of preceding points in time. Given a sequence of observed sensor values, computational methods can be used to compute the hidden state sequence that most likely corresponds to the observations. When used in activity recognition, hidden states represent the activities, as is shown in Figure 4. To increase the accuracy of activity recognition using HMMs, the model can be trained incorporating prior knowledge of the activity classes.

Figure 4.

HMM for an activity recognition task with four hidden states (activities) and a set of observable nodes that corresponding to observed sensor events.

HMMs can also be coupled to recognize multiple simultaneous activities [56]. Other structures of dynamic Bayesian networks have been tested for resident tracking [57] and to model activities together with other parameters of interest such a emotional state [58]. In one case, a three-level HMM represented as a DBN was used to infer activities from GPS sensor readings [59]. HMMs have played the role of a post-processor [60] to smooth out activity recognition results generated by an AdaBoost classifier.

In contrast to generative methods, discriminative approaches explicitly model the boundary between different activity classes. Decision trees such as C4.5 model activities based on properties of the activity sensor-based features [55]. Meta classifiers based on boosting and bagging have also been tested for activity recognition [61][62]. Support vector machines have demonstrated success at real-time recognition of activities in complex everyday settings [16], and discriminative probabilistic graphical models such as conditional random fields have been effective in home settings [52][63].

An important aspect of the activity recognition is the settings in which the data has been collected and the activity models are tested. Most of the activity recognition approaches experiment with data that is collected in scripted laboratory settings [8], [39][64]. This is the most constrained setting where the subjects perform the activities under the supervision of the researchers. The markers for the beginning and ending of the activity are very clear. Activity recognition algorithms focus on learning these clean pre-segmented activity sequences. The next step taken by the researchers is to recognize activities in unscripted settings, an example of which is a smart home while residents perform their daily routine. This setting provides the natural setting for the residents to perform the activities. However the activity algorithms use an offline annotation process to segment the required data for building the activity models [63], [65]. Recently, there has been an increasing focus on activity recognition algorithms that work directly on senor streams, offering a practical solution for real-world settings. This represents the most complicated yet realistic setting for training and testing activity recognition algorithms.

One of the challenges for data mining researchers working with smart home data is the lack of uniformity with which activities are recognized and labeled even among human observers and annotators [66]. Another challenge is the similarity between activities – if an observer is given a limited time window of information, cooking and cleaning activities could appear identical. Knowledge-driven activity recognition techniques attempt to overcome these challenges by making use of external sources of information to model, recognize, and analyze activity data. Examples of external information sources are activity ontologies and domain knowledge [67] and semantic activity information that is mined from publically available sources [8][68][69]. Activity recognition across generalized settings has also been investigated by a number of researchers [70]. Examples of the generalized settings include different sensor layouts, activity recognition across different sensors and different residents.

5. Behavioral Pattern Discovery

While recognizing predefined activities often relies on supervised learning techniques, unsupervised learning is valuable for its ability to discover recurring sequences of unlabeled sensor activities that may comprise activities of interest in smart homes. Methods for behavior pattern discovery in smart homes build on a rich history of unsupervised learning research, including methods for mining frequent sequences [71][72], mining frequent patterns using regular expressions [3], constraint-based mining [73], mining frequent temporal relationships [74], and frequent-periodic pattern mining [75].

More recent work extends these early approaches to look for more complex patterns. Ruotsalainen et al. [77] design the Gais genetic algorithm to detect interleaved patterns in an unsupervised learning fashion. Other approaches have been proposed to mine discontinuous patterns [73], in different types of sequence datasets and to allow variations in occurrences of the patterns [78]. Discovered behavioral patterns are valuable to interpret sensor data, and models can be constructed from the discovered patterns to recognize instances of the patterns when they occur in the future.

In contrast to approaches that mine patterns in an unsupervised fashion from sensor data, topic model-based daily routine discovery [76][79] builds a hierarchical activity model in a combination of supervised and unsupervised methods. The less complex activities (see Figure 2) are recognized using a supervised learning algorithm. Next, the model’s higher level discovers combinations of lower-level activities that comprise more complex activity patterns. A mixture of activities is modeled as a multinomial probability distribution where the importance of each feature is also modeled as a probability distribution. Figure 5 (top) shows an intuitive description of activities noted by a human subject and Figure 5 (bottom) shows the activities that were automatically discovered using the topic model.

Figure 5.

Ground truth and discovered activities using topic models [76].

Behavior pattern discovery and activity recognition are not always pursued as separate directions within data mining research. Pattern discovery methods identify patterns of interest that can later be tracked to monitor a smart home resident’s daily routine and look for trends and abrupt changes [78]. In addition, unsupervised discovery can help analyze data that does not fall into predefined patterns. In fact, some researchers found that activity recognition accuracy is actually boosted when unsupervised learning methods are used to find patterns in data not falling into a predefined activity class [80]. The discovered patterns can themselves be modeled and recognized, allowing for characterization and tracking of a much larger portion of a resident’s daily routine.

In addition to characterizing and recognizing common normal activities that account for the majority of the sensor events that are generated, smart home residents are also very interested in abnormal events. These abnormal events may indicate a crisis or an abrupt change in a regimen that is associated with health difficulties. Abnormal activity detection, or anomaly detection, is also important in security monitoring where suspicious activities need to be flagged and handled. Anomaly detection is most accurate when it is based on behaviors that are frequent and predictable. There are common statistical methods to automatically detect and analyze anomalies including the box plot, the x chart, and the CUSUM chart [81]. Anomalies can be captured at different population scales. For example, while most of the population may perform activity A, one person carries out activity B, which pinpoints a condition needing further investigation [82]. Anomalies may also be discovered at different temporal scales, including single events, days, or weeks [83].

Little attention has been devoted to anomaly detection in smart homes. This is partly because the notion of an anomaly is somewhat ill-defined. Many possible interpretations of anomalies have been offered and use cases have even been generated for ambient intelligent environments [84]. Some algorithmic approaches have been suggested that build on the notion of expected temporal relationships between events and activities [85]. Others tag events as anomalies if the occur rarely and they are not anticipated for the current context [86].

6. Monitoring and Promoting Behavioral Influence

As we have discussed, data mining techniques can be used to recognize predefined classes of behaviors as well as identify interesting or anomalous patterns from smart home sensor data. Once resident behavior is characterized in this way, it can be further analyzed to achieve several goals. First, behavior can be correlated with parameters of interest to the resident or to researchers. Second, the mined patterns can be use to proactively automate control of the home. Third, home-based interventions and modifications can be identified through data mining techniques that will promote desired resident behaviors.

Researchers who have analyzed smart home data realize even more profoundly that people do not live in a vacuum. They are influenced by their environment and other residents in the environment, and they have a profound impact on the other residents as well as their physical environment. The ability to measure the correlation between smart home behavior and other parameters of interest is both a unique opportunity and a unique challenge for data mining researchers. Such correlations have been identified between smart home-based behaviors and well being, energy usage, and air quality.

The possibilities of using smart environments for health monitoring and assistance are perceived as “extraordinary” [87] and are timely given the aging of the population [88][89]. A smart home is an ideal environment for performing automated health monitoring and assessment, because the resident can be remotely observed performing their normal routines in their everyday settings. As an example, Pavel et al. [90] observed using smart home data that changes in home-based mobility patterns are related to changes in cognitive health. Lee and Dey [91] also designed an embedded sensing system and presented information to older adults to determine if this information was useful to them in gaining increased awareness of their functional abilities. Hodges et al. [92] found a positive correlation between sensor data gathered during a home-based coffee-making task and the first principal component of standard neuropsychological scores for the participants. In a large study with hundreds of participants, Dawadi et al. employed data mining techniques to automatically group participants into cognitive health categories based on sensor features collected while they performed a set of complex activities in a smart home testbed [93].

The ability of a smart home to provide insights on well being can also be evidenced in the research performed to gain insights on healthy habits. As an example, Wang et al. analyzed smart home data to estimate exercise levels and energy expenditure for residents [94]. Helal et al. not only estimate overall energy expenditure but also monitoring eating habits in the home to provide feedback on healthy behaviors for diabetes patients [95] (see Figure 6).

Figure 6.

Feedback on data mined healthy behaviors detected in a smart home [95].

In addition, data mining techniques can be used to investigate the relationship between behavioral patterns and sustainability. Households and buildings are responsible for over 40% of energy usage in most countries [96], yet many residents still receive little or no detailed feedback about their personal energy usage. On the other hand, previous studies have shown that home residents actually reduce energy expenditure by 5%–15% as a simple response to acquiring and viewing related usage data [97]. By designing data mining techniques to process smart home data as well as whole-home energy usage data, researchers have been able to provide detailed feedback to residents [98] and to find correlations between behavior and energy usage and to predict energy usage based on activities detected in smart homes [99]. Similarly, using data mining techniques detailed feedback on air quality is provided to residents [100] and correlations have been identified between behavior and indoor air quality [101].

It is important to note that the combination of smart home and data mining techniques is not only useful for monitoring behavior and behavior-based correlation, but can also be used as the basis for providing proactive assistance from the home itself. As an example, Komatsuzaki et al. designed a technique to find items in a room using smart home sensors, intelligent reasoning and robotics [102]. While data mining techniques can identify automation strategies that relieve residences of the burden of interacting devices [10], they can also be directed to improving usage of the home. As examples of this idea, Scott et al. use occupancy prediction techniques to automate the control of a home’s heating system [103], and Mishra et al. use smart home data to automate device usage and prevent spikes in the power grid [104].

7. Challenges for Researchers who Mine the Home

The dream of building smart homes is hampered by very some formidable challenges. A primary concern is the need to consider the implications of maintaining privacy and security while mining smart home data. Many individuals are reluctant to introduce sensing technologies into their home, wary of leaving “digital bread crumbs”. Studies highlight evidence that such wariness is well founded. Residents are willing to allow a home to collect tremendous amount of personal information including behavioral routines, personal preferences, diet, medication, and health status, so that the home can provide context-aware services. However, this same information may be stored in the cloud and either hacked or intentionally sent to individuals and companies that can utilize the information for advertising or for more nefarious purposes [105]. Personal information can be collected in ways that may not be easily anticipated. For example, occupancy detection and behavior inference can be accomplished by monitoring smart meters [106].

While organizations such as the FTC have been discussing ways to ensuring security before it becomes a widespread issue [107], researchers also need to be proactive in investigating related issues. Specifically, researchers need to define and provide guarantees for levels of privacy and for the safety of the technologies. As ambient intelligent systems become more ubiquitous, more information will be collected about individuals and their lives. The impact of such monitoring needs to be better understood [108][109] and kept secure from those who would use the information for malicious purposes. Even when information is shared only with caring friends and family members, studies indicate that the result can impact an individual’s sense of privacy and independence [110].

Many privacy concerns focus on the perception of intrusive monitoring [111]. At the same time, however, many heavily-deployed Internet gadgets and current ambient intelligent systems are nearly devoid of security against adversaries, and many others employ only crude methods for securing the system from internal or external attacks. When smart home technologies were in their infancy, the possibility of hacking such connected environments seemed implausible. As the technology matures, so does the very real ability for hackers to infiltrate the home and collect usable information about its residents [112], [113]. As a result, every aspect of data collection, inference, communication, and storage software design needs to have privacy and security at the forefront. Privacy-preserving sensed environments need to be designed [114], [115] and researchers need to study the relationship between privacy assurance and quality of information gathering [116].

The definition of privacy will continue to evolve as ambient intelligent systems mature [117]. This is highlighted by the fact that even if personal information is not directly obtained by an unwanted party, much of the information can be inferred even from aggregated data. For this reason, a number of approaches are being developed to ensure that important information cannot be gleaned from mined patterns [118][119].

As well as introducing some major challenges for data mining researchers, smart home sensor data and technologies offer some unprecedented opportunities. Because smart homes are becoming more prevalent, sensing and analysis no longer needs to be performed on an individual basis. Data mining researchers can consider modeling and understanding an entire population. This has been considered in initial studies focusing on movement patterns [120] but can be extended to analyze more behaviors such as eating, sleeping, and socializing.

The benefits of smart home technologies and patterns mined from smart homes are typically confined to the interior of the building in which an individual resides. As a result, another critical challenge for data mining researchers is to fuse data and models based on smart home data with those based on external sources such as smart phones. Given the progress that we have witnessed in smart homes and related data mining techniques, researchers and practitioners can now look toward these next steps and anticipate the continued growth of the field.