Recognition of Physical Activities in Overweight Hispanic Youth Using KNOWME Networks

Abstract

Background

KNOWME Networks is a wireless body area network with two tri-axial accelerometers, a heart rate monitor, and mobile phone that acts as the data collection hub. One function of KNOWME Networks is to detect physical activity (PA) in overweight Hispanic youth. The purpose of this study was to evaluate the in-lab recognition accuracy of KNOWME.

Methods

Twenty overweight Hispanic participants (10 males; age 14.6±1.8 years), underwent four data collection sessions consisting of nine activities/session: lying down, sitting, sitting fidgeting, standing, standing fidgeting, standing playing an active video game, slow walking, brisk walking, and running. Data was used to train activity recognition models. The accuracy of personalized and generalized models is reported.

Results

Overall accuracy for personalized models was 84%. The most accurately detected activity was running (96%). The models had difficulty distinguishing between the static and fidgeting categories of sitting and standing. When static and fidgeting activity categories were collapsed, the overall accuracy improved to 94%. Personalized models demonstrated higher accuracy than generalized models.

Conclusions

KNOWME Networks can accurately detect a range of activities. KNOWME has the ability to collect and process data in real-time, building the foundation for tailored, real-time interventions to increase PA or decrease sedentary time.

Background

Childhood obesity has reached epidemic proportions, particularly among minority youth. A study using United States (US) national data from 2007-2008 reported that 18% of adolescents aged 12 - 19 years are obese and 34% are overweight¹. These numbers were substantially higher in Hispanic youth¹, placing them at high and life-long risk of diseases such as cardiovascular disease^2,3 and Type 2 diabetes⁴. Physical activity (PA) is central to reducing and preventing obesity and maintaining physical and mental well-being⁵, yet PA declines precipitously during early adolescence^6,7. In 2007, a nationwide survey of US high school students reported that only 37% of non-Hispanic White, 31% of non-Hispanic Black, and 30% of Hispanic adolescents met the recommended 60 minutes per day of moderate-to-vigorous PA⁸. These numbers show that majority of adolescents are not meeting the PA recommendations, a large number of them are overweight or obese, and that minority youth are disproportionately impacted.

Accurate measurement of PA is essential for determining whether an individual meets PA guidelines and for evaluating the effectiveness of interventions to increase physical activity⁹. Objective measures such as those obtainable using an accelerometer (ACC) or a heart rate (HR) monitor can capture the intensity and duration of an activity and subjective measures such as self-report or diary methods can capture the activity being done as well as the context. However, these methods have limitations. For instance, objective measures are generally unable to identify the type of activity being performed, while subjective measures are vulnerable to recall bias. Increasingly researchers are moving towards measurement of PA using multiple sensors systems and pattern recognition. To date, the majority of research has utilized one or more accelerometers and data processing algorithms to detect physical activity states with an overall accuracy ranging from 81-99%^10-13. Basic activities such as lying^13-15, sitting^13,16, standing^13,17, walking^13,18 and running¹⁴ can be detected with a high degree of accuracy while more complex activities such as walking uphill¹², stretching¹⁹, or ascending and descending stairs²⁰ are more difficult to detect accurately. PA state detection could provide a more informative objective measurement of PA and, when combined with a contextual measure, could offer a rich data set showing where, when and what activity is being performed. The ability to measure PA objectively as well as capture contextual information would allow for a more sophisticated understanding of PA behavior and the development of interventions that target a specific behavior within a specific context.

Interventions to increase PA in children and adolescents have been largely unsuccessful, although interventions to decrease sedentary behavior (SB) are more promising²¹. Traditionally, PA interventions collect data pre- and post-intervention and analyze that data after completion of the study. Such data collection and analysis conventions do not allow researchers to monitor of the effectiveness of or adapt the intervention while it is occurring. Real-time sensing of PA would allow for the development of interventions to increase PA or decrease SB that provide immediate tailored feedback which may be more effective in enacting behavior change²².

Minority youth are at higher risk of being overweight than their Caucasian counterparts, however, few interventions have targeted minorities²¹. Intervention tools that are developed in and appropriate for this population are thus sorely needed. The goal of the KNOWME Networks was to design a mobile, wearable body area network system that can detect PA states in overweight Hispanic youth with a high degree of accuracy using a small number of discrete, wireless sensors. This population presents at least two major challenges. First, ethnicity²³ and body mass^24-26 may influence the energy cost of locomotion, and hence could be key variables in calibrating sensor output and developing accurate algorithms ²⁷. To date, no cut-points or state detection algorithms have been developed specifically for overweight Hispanic youth. Second, age and ethnicity will likely impact which sensor configuration is considered wearable and unobtrusive. Therefore, this study took a participatory approach²⁸ that included in-lab try-outs and focus group interviews to ensure that choice and placement of sensors was acceptable to the targeted group. The current suite of sensors includes accelerometers and a heart rate monitor which interface directly with a mobile phone to process and relay data. This data is sent to a backend server where it is further processed, stored, and relayed to a secure website. In addition, KNOWME Networks enables real-time sensing which will facilitate the monitoring of PA accurately, remotely, and unobtrusively and support the development of real-time interventions. The purpose of this study was to evaluate the accuracy of personalized and generalized models in detecting a range of physical activities performed by overweight Hispanic youth wearing KNOWME Networks in a laboratory setting.

Methods

Subjects

Participants were 20 overweight Hispanic boys and girls aged 12-17 years. Participants were recruited from Los Angeles County, California, through participating physicians and clinics, churches, schools, and community centers. Inclusion criteria were Latino heritage by parental self-report and meeting criteria for overweight [age- and gender-specific body mass index (BMI) ≥ 85th percentile of the Centers for Disease Control and Prevention (CDC) reference data²⁹]. Parents and children were provided with detailed written and verbal information about the study in their primary language and informed written consent from the parent and assent from the child was obtained. The study was approved by the Institutional Review Board of the University of Southern California (USC).

Instrumentation

The current system uses off-the-shelf devices, including the Alive Technologies Heart Monitor and Accelerometer (Arundel, Australia), which contains a single channel electrocardiogram (ECG) monitor and a tri-axial ACC, and the Nokia N95 phone with 8GB built-in flash memory³⁰. The sampling rate for the ECG is 300 Hz and the sampling rate for the ACC is 75Hz. One Alive Technologies monitor is placed on a chest strap and positioned in the center of the chest to record ECG and ACC data, while a second Alive Technologies monitor is placed on the right hip in a custom pouch to record ACC data only. The phone is placed in a holder on the left hip. The device configuration and system setup is shown in Figure 1. Sensor data is continuously streamed to the mobile phone via Bluetooth connection. Data collection on the phone is done using programming that was developed specifically for KNOWME Networks³¹ and written in Python³². For this experiment, data was time-stamped and stored on the phone and then downloaded and processed offline.

Figure 1. Current configuration of KNOWME Networks.

A heart rate monitor/accelerometer is placed on a chest strap and centered on the chest while a second accelerometer is placed on the right hip. The phone can be carried in a case on the left hip as was done in this study, or placed elsewhere on the body.

Study Procedures

Study visits took place in an exercise observation laboratory designed specifically for studies in youth. Two video cameras were installed in the lab to unobtrusively record in-lab activities. Height was measured to the nearest 0.1 cm using a wall-mounted stadiometer and weight was measured to the nearest 0.1 kg with a Tanita TBF 300/A analyzer (Arlington Heights, IL). Measurements were done while wearing light clothing and no shoes. BMI was calculated as weight (kg) divided by height (m) squared. BMI percentiles for age and gender were calculated using publicly available SAS syntax from CDC (available at: http://www.cdc.gov/nccdphp/dnpa/growthcharts/resources/index.htm).

While wearing KNOWME Networks, participants performed a series of 9 activities in a pre-determined order for 7 minutes per activity. The activities were: lying down, sitting motionless, sitting fidgeting, standing motionless, standing fidgeting, standing playing an active video game (Wii), slow walking, brisk walking, and running (both walking and running were at self-selected speeds). These activities were based on a continuous version of System for Observing Fitness Instruction Time (SOFIT) protocol³³ and supplemented to include other activities commonly performed by youth, namely, fidgeting and playing active video games. We developed and used this augmented SOFIT system for continuous behavioral observation successfully in earlier research ³⁴ and it provides a solid basis for expanding behavioral categories for recognition in the future. Nintendo Wii tennis was selected as the active video game because the participants could learn how to play it quickly and enjoyed playing it. Activities were performed in the same order every time unless the participant was physically unable to complete them in that order. Participants were allowed to rest between activities for however long they deemed necessary. Each session lasted between 1 - 1.5 hours. Participants completed 4 sessions with a maximum of 2 sessions per day. Data from multiple days and sessions were collected in order to develop more robust personalized models by introducing variability across days into the models. A researcher in the room provided instruction and supervision while the participant was performing the activities as well as noting the start and end time of each activity. In addition, all sessions were video-recorded using the unobtrusive camera system described above.

Analysis

As described in detail previously ^35,36, ECG and ACC data were processed to extract features from 10 heart beats and 6.72s data windows, respectively. Briefly, the ECG signal was pre-processed to normalize each heartbeat to the same time duration and amplitude range. Using this normalized ECG signal, the cardiac activity mean (CAM) and motion artifact noise (MAN) features were determined. Next, Hermite polynomial expansion (HPE) coefficients were used to model the CAM feature and principal component analysis (PCA) error vectors were used to model the MAN feature. The HPE coefficients and the PCA error vectors were then combined with the standard deviation of multiple normalized heart beats and conventional ECG features such as mean and variance of heart rate. For the ACC signal, a set of temporal features was extracted. The ECG temporal feature sets were used in a Support Vector Machine (SVM) with a linear kernel. Linear discriminant analysis (LDA) based dimension reduction was performed on the ACC temporal features and a radial basis function (RBF) kernel based SVM was adopted for pattern recognition. Furthermore, ECG and ACC signals were also transformed to obtain cepstral features that were modeled by a Gaussian Mixture Model (GMM). Following heteroscedastic linear disciminant analysis (HLDA) to reduce the ACC cepstral feature dimensions, the SVM and GMM systems were fused at the score level to obtain final physical activity state detection results³⁷.

Personalized (subject-dependent) models were created using training data from three data-collection sessions and tested against data from the fourth session for each participant. Generalized (subject-independent) models were created using pooled training data from all but one of the participants and tested against data from the remaining participant (leave-one-out analysis). Analyses were repeated rotating which session or participant was excluded and then averaged to obtain the overall accuracy.

Results

Table 1 shows the characteristics of the 20 participants included in this study and the walking and running speeds that were selected by the subjects. There is overlap between the speeds selected by the subjects for slow walking, brisk walking, and running.

Table 1. Characteristics of the sample population.

Anthropometric/Demographic Measures	Mean ± SD (range)
Age (years)	14.6 ± 1.8 (12 - 17)
BMI (kg/m2)	31.6 ± 6.3 (24.9 - 51.3)
BMI percentile (CDC guidelines)	95.8 ± 3.7 (87 - 99)
% Body fat	36.3 ± 6.7 (22.9 - 50.2)
Treadmill speeds (mph)
Slow walking	1.5 ± 0.5 (0.9 – 2.5)
Fast walking	2.7 ± 0.7 (1.7 – 4.0)
Slow running	4.0 ± 0.7 (2.5 – 5.0)

SD = standard deviation; BMI = body mass index

Table 2 shows the contribution of the main analyses modalities (temporal and cepstral signal features) and the different sensors to the accuracy of the predictions. While the accelerometer temporal features provide fairly good accuracy, the accumulated model using both sensors and types of features is the most accurate. The combination of accelerometer temporal and cepstral features was not more accurate than the combined model using data from both sensors.

Table 2. Final fusion results showing roles of sensors in accuracy of detection.

	Accuracy in 9-class behavioral analyses
1: ECG_Temporal	47%
2: ACC_Temporal	81%
3: ECG_Cepstral	50%
4: ACC_Cepstral	66%
5: 1+3	53%
6: 2+4	83%
7: 1+2	81%
8: 3+4	69%
9: 1+2+3+4	84%

Table 3 shows the accuracy of personalized and generalized models. The models were constructed considering seven or nine classes of physical activity. The nine class activity set reflected the full range of behaviors performed during training, while the seven class system was achieved by collapsing the sitting and sitting fidgeting activities into one category and collapsing the standing and standing fidgeting activities into one category. The seven class system outperformed the nine class system and personalized models outperformed generalized models.

Table 3. Percentage of decisions classified correctly by personalized (subject-dependent) and generalized (subject-independent) models.

	# of Activity Classes Model Type	7 Activity Classes		9 Activity Classes
		Personalized	Generalized	Personalized	Generalized
1	Lying down	93	88	95	91
2	Sit	91	82	63	65
3	Sit & fidget			63	57
4	Stand	96	86	79	73
5	Stand & fidget			79	71
6	Stand & play Wii	94	88	95	93
7	Slow walk	95	84	94	84
8	Brisk walk	93	68	93	72
9	Run	96	82	96	82
	Overall Accuracy	94	83	84	76

Figure 2 shows a confusion matrix of the 9 activities personalized state detection model accuracy (overall accuracy for this model was 84%). The confusion matrix displays the distribution of the decisions made by the model and whether or not that decision was accurate (decisions along the diagonal represent correct decisions). The distribution of decisions is also color-coded such that cells containing 100% of the decisions would be pure black while those cells containing 0% of the decisions would be pure white.

Figure 2. Confusion matrix for personalized modeling of 20 subjects for 9 activities.

Predicted activity is shown on the X-axis and actual activity is shown on the Y-axis. Correct decisions fall on the diagonal. Percentage of decisions falling into each cell is shown. Cells containing 100% of the decisions are pure black while those cells containing 0% of the decisions are pure white.

Discussion

Overall, the highest accuracy obtained using this system was 94% with seven classes of physical activity using personalized models. When classification of fidgeting activities was included, the accuracy dropped to 84%. For personalized models, the most accurately detected activity was running, which is consistent with previous studies of PA state detection^14,38. The least accurately detected activity was sitting. This is most likely due to classification confusion between sitting and sitting fidgeting (Table 3).

Personalized models were more accurate than generalized models. The vast majority of previous studies report findings from generalized models^10-20. In studies that have compared personalized and generalized models, results have been mixed: three studies reported higher accuracy for generalized models^39-41, two reported approximately equivalent accuracy^42,43, and two studies reported higher accuracy for personalized models^44,45. The studies that reported higher accuracy for personalized models^44,45 included HR data in the analysis in addition to ACC data. Heart rate, particularly as it is modeled here³⁷, could be considered a biometric signal that is unique to the individual and is therefore better suited for personalized models⁴⁶. Different fitness levels, as well as different self-selected walking and running speeds, may also have contributed to the lower accuracy in the generalized models. In addition, model accuracy typically improves with increased training data and many studies do not have enough individual training data to generate accurate personalized models. While creating generalized models requires less data collection per individual, such models may be less accurate at the individual level and thus not ideal for real-time interventions that rely on providing accurate feedback in response to detected activities. In addition, individual data collection allows for the development of a model that is tailored to the activities the participant is likely to perform as well as how the individual performs those particular activities.

The detection of fidgeting is unique to this project. Fidgeting states were modeled because they are a component of non-exercise activity thermogenesis (NEAT), which may offset some caloric intake and play a role in combating weight gain^47,48. Relative to sitting motionless, sitting and fidgeting expends 1.46 times the amount of energy, while relative to standing motionless, standing and fidgeting expends 1.69 times the amount of energy⁴⁹. However, the current system was not able to detect fidgeting activities with a high degree of accuracy. This is most likely due to the placement of the ACCs on the hip and chest. Placement of ACCs on the extremities (wrists, ankles) may better identify fidgeting. However, three focus group interviews with six minority adolescents (4 male/2 female, ages 12-17) conducted during the development of the KNOWME Networks sensor suite suggest that sensors worn on ankles or wrists are not desirable in this group. KNOWME Networks is designed with the goal of real-world implementation: while placement of additional sensors could increase the detection accuracy of fidgeting, there is a tradeoff between increased accuracy and the number of sensors participants are willing to wear.

In addition to fidgeting, active gaming was also modeled. Youth spend a large amount of time using the computer or playing video games⁸. Increased participation in “exergames” could be used as a tool to increase PA in kids⁵⁰ and KNOWME Networks could detect and encourage the use of such games. The game selected here – Wii tennis – has been shown to expend 1.61 times the amount of energy relative to a sedentary video game and could result in a small, though potentially meaningful, increase in energy expenditure (EE) across a week of play⁵¹. Wii tennis was modeled with high accuracy and other games could be incorporated into each participant's personalized model on the basis of what games the participant reports playing. Additionally, mobile-phone and internet-based games designed to encourage PA could be incorporated into future versions of KNOWME Networks.

The activities chosen for this study reflect a broad range of EE, ranging from 1.0 metabolic equivalents (METs) for lying down to 8.0 METs for running at 5.0 miles per hour⁵². In addition to distinguishing between basic states such as lying down, sitting, standing, walking and running, KNOWME Networks can differentiate between similar activities performed at different intensities (standing fidgeting vs. standing playing Wii; slow walking vs. fast walking). This information could be used to estimate EE. Other sensor systems, the Bodymedia armband and IDEAA system, have been tested in combination with indirect calorimeter for the estimation of EE^53,54. These studies show that such systems can be used for accurate estimation at the group level, but are less accurate at the individual level⁵⁴ and may yield poor estimates in obese subjects⁵³. EE for a particular activity can differ by age, gender, ethnicity and weight status²³ and overweight Hispanic youth may be less energy efficient in how they perform activities²⁴. Future studies will need to determine the EE of the activities detected by KNOWME Networks in overweight Hispanic youth. Because KNOWME Networks utilizes a personalized training phase, indirect calorimetry performed during the training phase could be used for personalized EE prediction.

This system has several strengths. It achieves accurate state detection using a novel feature set and has the potential for real-time data analysis and feedback. In addition, KNOWME Networks includes a HR monitor, while many other multi-sensor systems consist exclusively of ACCs. While the HR monitor provides only a small improvement in state detection accuracy, consistent with previous studies^44,45, it does provide information about the relative PA intensity of the activity since heart rate correlates linearly with oxygen consumption⁵⁵. Additionally, our earlier work shows that fusion of the multimodal (HR monitor and accelerometer) and multi-domain (time domain SVM and cepstral domain GMM) subsystems at the score level can improve recognition accuracy of specific behaviors ³⁶. Furthermore, our previous work suggests that data provided by certain types of sensors are more informative in distinguishing between certain activities than other types ^35,56. For example, data from the HR monitor is a better discriminator when the subject is lying down, or in other activities that require low levels of energy expenditure, while data from the accelerometer is more pertinent to distinguishing between non-sedentary activities, or activities that demand higher energy expenditure. Detection of additional high intensity activities along with more complex activities such walking uphill/stairs, cycling, playing soccer or riding in a car can be included in future studies to further broaden the range of detected activities. Lastly, PA and SB detection and feedback using KNOWME Networks can be automated and thus is ideally situated to provide long-term interventions. Currently, the KNOWME system feeds data from all sensors to a secure website at 10-minute intervals, where it is analyzed in real-time. The mobile phone interface can prompt youth to be more active when investigator-determined levels of sedentary behavior have been reached. For example, automated as well as personalized text messages, tailored to intervention groups and reacting in real time to incoming data, can be sent to encourage physical activity and discourage long periods of sedentariness. This KNOWME system intervention is currently being pilot tested in a group of overweight Hispanic youth. Obesity prevention and treatment requires long-term care, and once this system has been personalized following the training phase, KNOWME Networks could provide a method to deliver an extended intervention.

KNOWME also has limitations. Some of these limitations are those that are inherent in current available sensors, including the inability to measure water-based activities such as swimming. Also, because the youth that we worked with did not want to wear sensors on their wrists or arms, detection of activities such as weight training might be less accurate. The use of in-lab data to train and test the models in KNOWME may not accurately reflect real-world performance^15,19,57. The ability of the models developed here to accurately identify activities performed in a semi-naturalistic setting, consisting of a video-recorded in-lab free-activity period, is currently being evaluated. Also, the need for individual training may make large scale implementation difficult. Future studies will evaluate the minimal amount of training data necessary to develop accurate personalized models. It is envisioned the KNOWME Networks will be used as an intervention tool by health care researchers and practitioners. In this scenario, participants would undergo a pre-wear interview to establish what activities should be modeled, followed by a brief training protocol to gather minimal data needed to build personal PA models.

The evaluation of multiple sensor systems for PA detection and intervention has been limited to adult populations and normal weight individuals^58-60. This is the first PA detection system to be specifically developed for and tested in overweight Hispanic adolescents. Although this could be seen as a limitation, the prevalence of obesity and obesity-related disorders, such as type 2 diabetes and the constellation of cardiovascular disease (CVD) risk factors known as the metabolic syndrome, has increased dramatically among adolescent Latinos in recent years ^61,62. Latinos have a 50% lifetime risk of developing diabetes ⁶³, raising the specter of significant early morbidity in mid-adulthood, as well as the potential for enormous increases in societal health care expenditures ⁶⁴. Therefore, targeting this group is extremely important. This system and the underlying approach could certainly be used in other populations following the appropriate validation. Systems that measure food intake in adolescents using a mobile phone are being developed by other research groups⁶⁵ and have been shown to be preferred over traditional paper-based records⁶⁶. Such systems could be combined with KNOWME Networks to create a system that monitors both energy intake and energy expenditure in youth. Additional functions such as location tracking or ecological momentary assessment (EMA) could also be utilized to create a rich contextual map of the conditions surrounding specific behaviors for each participant.

Future directions for KNOWME Networks include the evaluation of the system in free-living environments and real-time interventions to increase PA and decrease SB. Prolonged SB has been associated with negative health outcomes⁶⁷ and studies have shown that adolescents spend much of their time engaged in SB⁶⁸. KNOWME Networks can classify an activity such as sitting as SB and calculate the duration of sitting and provide automatic feedback. Immediate feedback may be effective for behavior change⁶⁹ although the exact display and wording of the feedback is important. Some participants do not respond well to negative feedback⁷⁰ and messages should not include negative enforcement or complex activity suggestions⁷¹. Studies to evaluate the effectiveness of different kinds of feedback are also necessary to advance this mode of intervention.

KNOWME targets a unique subject population at high risk for inactivity and obesity, includes a heart-rate monitor which could provide information on relative activity intensity and better detect low level activities, and is designed to provide real-time detection and feedback. Furthermore, while several other projects have advanced the research agenda of accurately detecting behaviors using sensors (^12,19,72-76 and others), most if not all of these studies are primarily focused on technological advances, and the needs of public health efforts are not their primary objective. Much work still needs to be done in addressing the potential end-users' needs and expectations. The needs of youth are different from the general population. KNOWME is moving beyond simple activity recognition to a system that can be used to deliver interventions. Thus, KNOWME Networks represents an innovative, interdisciplinary approach to a complex problem. KNOWME Networks is a tool that can be used by health professionals and researchers for real-time detection, classification and monitoring of PA and other obesity-related behaviors and create the opportunity for highly individualized interventions. These interventions can be tailored to meet the treatment goals of the individual and take into account the individual's abilities, preferences and needs, while performing real-time monitoring and providing real-time feedback.