Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Mar 1.
Published in final edited form as: Ment Health Phys Act. 2016 Mar;10:10–17. doi: 10.1016/j.mhpa.2016.02.001

Wearable Devices and Smartphones for Activity Tracking Among People with Serious Mental Illness

John A Naslund a,b, Kelly A Aschbrenner a,b,c, Stephen J Bartels a,b,c,d
PMCID: PMC4845759  NIHMSID: NIHMS763147  PMID: 27134654

Abstract

Introduction

People with serious mental illness, including schizophrenia spectrum and mood disorders, are more physically inactive than people from the general population. Emerging wearable devices and smartphone applications afford opportunities for promoting physical activity in this group. This exploratory mixed methods study obtained feedback from participants with serious mental illness to assess the acceptability of using wearable devices and smartphones to support a lifestyle intervention targeting weight loss.

Methods

Participants with serious mental illness and obesity enrolled in a 6-month lifestyle intervention were given Fitbit activity tracking devices and smartphones to use for the study. Participants completed quantitative post-intervention usability and satisfaction surveys, and provided qualitative feedback regarding acceptability of using these devices and recommendations for improvement through in-depth interviews.

Results

Eleven participants wore Fitbits for an average of 84.7% (SD=18.1%) of the days enrolled in the study (median=93.8% of the days enrolled, interquartile range=83.6–94.3%). Participants were highly satisfied, stating that the devices encouraged them to be more physically active and were useful for self-monitoring physical activity and reaching daily step goals. Some participants experienced challenges using the companion mobile application on the smartphone, and recommended greater technical support, more detailed training, and group tutorials prior to using the devices.

Discussion

Participants’ perspectives highlight the feasibility and acceptability of using commercially available mHealth technologies to support health promotion efforts targeting people with serious mental illness. This study offers valuable insights for informing future research to assess the effectiveness of these devices for improving health outcomes in this high-risk group.

Keywords: mental illness, mHealth, wearable device, Fitbit, smartphone, physical activity

1. Introduction

Emerging mobile health (mHealth) technologies such as wearable devices and smartphone applications for activity tracking have generated new opportunities for promoting healthy lifestyles across diverse patient groups. Yet, for these devices to reach their full potential in facilitating positive health behavior changes, it is necessary to consider the perspectives of target populations to inform intervention efforts (Chiauzzi, Rodarte, & DasMahapatra, 2015). How can wearable devices and smartphones support existing evidence-based lifestyle interventions targeting fitness and weight loss among high-risk patient groups such as people with serious mental illness? What insights can individuals with serious mental illness provide to inform the use of wearable devices and smartphones as part of these interventions?

People with serious mental illness, such as schizophrenia spectrum or mood disorders, are more likely to be physically inactive than people from the general population (Daumit et al., 2005). Physical inactivity combined with high prevalence of obesity and unhealthy lifestyle behaviors such as smoking contribute to poor cardiovascular health, increased risk of comorbid medical conditions, and reduced life expectancy in this group (Colton & Manderscheid, 2006; Richardson et al., 2005; Walker, McGee, & Druss, 2015). Lifestyle interventions aimed at promoting physical activity have shown substantial promise for improving cardiovascular health and contributing to weight loss among people with serious mental illness. Two recent randomized controlled trials of the In SHAPE program, which includes weekly meetings with a fitness trainer and a gym membership, showed clinically significant reduction in cardiovascular risk through improved fitness or modest weight loss in about half of participants (Bartels et al., 2013; Bartels et al., 2015). Similarly, the STRIDE and ACHIEVE programs, which both promote physical activity as part of weight loss efforts, have contributed to modest weight loss of 5% or greater in up to 40% of participants (Daumit et al., 2013; Green et al., 2015). Despite the success of these interventions, they are resource intensive, costly to deliver, and require individualized support and coaching towards reaching personalized exercise or weight loss goals.

Wearable activity tracking devices and smartphones may afford important opportunities to increase the scalability, reach, and impact of existing interventions aimed at promoting physical activity for weight loss among people with serious mental illness. Wearable devices and the companion smartphone applications connected with these devices are increasing in popularity, availability, and affordability. These devices offer numerous features that promote greater engagement among users and make being physically active more fun (Patel, Asch, & Volpp, 2015). These include options for personal goal setting, rewards or trophies for achieving milestones, the ability to connect, compete, and interact with peers and give or receive social support through mobile application interfaces, as well as the capacity to track activity and individual performance over time through user friendly graphs (Lyons, Lewis, Mayrsohn, & Rowland, 2014).

Research in the general population has shown that personal activity tracking may be an effective technique for supporting health promotion efforts and targeting cardiovascular risk factors (Burke et al., 2015). Basic devices such as pedometers provide immediate feedback about number of steps walked and research suggests that this information can help raise awareness about activity levels and increase self-efficacy to be more active (Rooney, Smalley, Larson, & Havens, 2003). A systematic review of studies demonstrated that self-monitoring daily step count using pedometers and setting personal activity goals contributed to increased physical activity and weight loss over time (Bravata et al., 2007). Another study found that increasing the number of steps walked each day among sedentary adults was associated with improvements in measures of cardiovascular health (Lee et al., 2013). Among people with serious mental illness, pilot studies have shown that pedometers are a feasible tool for tracking physical activity over short durations of 2 weeks or less (Beebe & Harris, 2012; Kane, Lee, Sereika, & Brar, 2012).

Wearable devices and smartphones represent a significant advancement over existing methods for tracking activity because they combine the benefits of self-monitoring with added features through mobile applications such as goal setting, tracking activity over time, reminders and prompts, and social connection (Patel et al., 2015). Preliminary research on these new devices has highlighted important benefits such as accuracy for measuring physical activity and feasibility for promoting activity among individuals with chronic medical conditions (Bai et al., 2015; Case, Burwick, Volpp, & Patel, 2015; Chiauzzi et al., 2015). Despite these promising findings, the approaches necessary for promoting the adoption and use of wearable activity tracking devices and the perceived utility of these devices among high-risk patient groups has received less attention (Chiauzzi et al., 2015).

Prior studies have emphasized the importance of understanding the needs, interests, and perspectives of vulnerable patient groups in order to successfully introduce new mHealth technologies (Zulman et al., 2015). For example, working closely with community partners and carefully considering the perspectives of the target population emerged as essential steps towards facilitating the implementation and uptake of a novel mHealth activity tracking system within a low-income African American neighborhood (Yingling et al., 2016). For people with serious mental illness, a recent pilot study showed that a sample of inpatients and outpatients were satisfied and felt comfortable using smartphone sensors for characterizing their daily activity patterns over a 1 to 2 week period (Ben-Zeev et al., 2015). In our prior work, we also observed that popular mHealth technologies appeared feasible for tracking physical activity among a community sample of people with serious mental illness (Aschbrenner, Naslund, Barre, et al., 2015; Naslund, Aschbrenner, Barre, & Bartels, 2015). However, it is necessary to obtain more detailed feedback and specific recommendations from individuals with serious mental illness about their experiences using wearable devices and the companion smartphone applications for these devices over longer time periods in order to inform efforts to promote physical activity for weight loss in this high-risk group.

In this study we employed an exploratory mixed methods approach (Creswell, 2013) to expand on our prior work and to better understand how wearable devices and smartphones can be used to support a healthy lifestyle intervention promoting exercise for weight loss among people with serious mental illness and obesity. We were interested in whether participants would sustain long-term use of these devices over a 6-month time period. We collected quantitative usability and satisfaction surveys and conducted qualitative interviews with participants following participation in a 6-month group-based lifestyle intervention. Our objective was to explore participants’ experiences using the wearable activity tracking devices, synching these devices with a smartphone, and using the companion mobile application. We were interested in obtaining feedback in order to inform the design of a future larger clinical trial.

1.1 Guiding Theoretical Framework

To guide the current exploratory study, we applied the National Institutes of Health (NIH) Stage Model of Behavioral Therapies Research (Onken, Carroll, Shoham, Cuthbert, & Riddle, 2014). According to this model, our exploratory study is classified as a Stage Ia study because the primary aim was to assess intervention acceptability by eliciting perspectives from the target population through surveys and in-depth interviews (Rounsaville, Carroll, & Onken, 2001). A key part of Stage Ia research is to obtain participant feedback to generate an intervention that maximizes engagement of the target population (Onken & Shoham, 2015). Additionally, a Stage Ia study is intended to inform a subsequent preliminary study aimed at assessing the potential effectiveness of an intervention (Stage Ib). Together, Stage Ia and Stage Ib pilot studies precede an eventual large scale randomized effectiveness trial (Rounsaville et al., 2001).

2. Methods

2.1 Setting and Participants

Participants enrolled in a 6-month lifestyle intervention delivered through an urban community mental health center in Southern New Hampshire were given Fitbit Zip wearable activity tracking devices and smartphones to use for the duration of the program. The lifestyle intervention was adapted from the evidence-based Diabetes Prevention Program curriculum, and wearable activity tracking devices were intended to support the physical activity goals of the program (Aschbrenner, Naslund, Shevenell, Mueser, & Bartels, 2015). Participants had serious mental illness defined by an Axis I diagnosis of schizophrenia, schizoaffective disorder, major depressive disorder, or bipolar disorder; were age 21 or older; spoke English; were on stable pharmacological treatment over the past two months; and had obesity defined as body mass index (BMI) ≥30. Participants were excluded if they had cognitive impairment defined as a Mini Mental Status Exam (Folstein, Folstein, & McHugh, 1975) score <24, could not walk at least one city block, were currently residing in a nursing home or other institution, or had an active substance use disorder. Participants were paid for completing the post-intervention assessments and interviews, but not for participating in the lifestyle intervention or using the Fitbit wearable devices or smartphones. This study and all procedures received ethical approval from committees for the Protection of Human Subjects at Dartmouth College and the New Hampshire Department of Health and Human Services.

2.2 Wearable Devices and Smartphones

At the start of the 6-month lifestyle intervention, participants were provided with Fitbit Zip wearable activity tracking devices. The Fitbit Zip (see https://www.fitbit.com/zip for detailed device specifications) is a small accelerometer about 1 inch in diameter that clips onto participants’ clothing such as a belt, bra, or shirt, or can be kept inside a pant or shirt pocket. The device can be worn discretely, and tracks number of steps, distance, active minutes, and calories burned. We elected to use the Fitbit Zip because it is equipped with a convenient LCD display that can be used to observe progress throughout the day without synching directly to the companion Fitbit mobile application. This feature is convenient for viewing progress on the Fitbit during instances where cellphone reception might be poor, or when engaging in exercise or other activities without the smartphone present. Participants attended a 30-minute individual training session with a research staff member (JN) to learn how to use the Fitbit Zip and how to read the visual display on the device. Participants were instructed to focus primarily on the step-count, because this was considered the most important measure of physical activity and aligned closely with the goals of the lifestyle intervention.

Participants initially used the Fitbit Zip for 1 to 2 weeks to get comfortable wearing it each day and reading and interpreting the display. After this introductory period, participants were provided smartphones to use for the study duration so that they could access the Fitbit mobile application. The Fitbit Zip synchs wirelessly through Bluetooth with the companion mobile application, which allows participants to track progress over time, rewards milestones with colorful trophies, and lets participants set daily step goals. Participants can also compare steps and progress by connecting with each other through the mobile application interface. Participants attended a second training session lasting about 30-minutes to learn to use the smartphone and Fitbit mobile application. Technical support for using the Fitbit Zip, companion mobile application, and smartphone was provided to participants by a member of the research team (JN) as needed.

2.3 Data Collection and Analysis

2.3.1 Wearable Device Use

Use of the Fitbits was calculated by determining the proportion of days enrolled in the study that participants wore the devices. Data on device use was obtained directly from participants’ personal mobile Fitbit accounts.

2.3.2 Quantitative Usability and Satisfaction Questionnaire

After completing the 6-month lifestyle intervention, participants completed a 24-item usability and satisfaction survey. This survey was aimed at understanding different aspects of using the Fitbit, synching the device with the companion smartphone application, and using the different features of the mobile application and smartphone. There were also questions about the perceived usefulness of the device. This survey was based on the USE questionnaire that measures usability, satisfaction, and ease of use (Lund, 2001), and was adapted from the questionnaire used in a prior study of a mobile intervention evaluated in people with psychotic disorders (Ben-Zeev, Kaiser, & Krzos, 2014).

2.3.3 Qualitative Participant Interviews

At the end of the 6-month lifestyle intervention, we conducted in-depth qualitative interviews with participants, lasting 45 to 60 minutes. The interviews were intended to expand upon participants’ quantitative survey responses (Creswell, 2013), and provide more comprehensive details about their experiences and insights using the Fitbits for activity tracking. We were interested in their opinions about the utility, design, and perceived benefits of tracking activity using these wearable devices and the companion smartphone application. We were also interested in participants’ views about using these mHealth devices to support a lifestyle intervention targeting weight loss and any recommendations for improvement.

Participant interviews were audio-recorded and transcribed for analysis. We employed a rapid content analytic approach adapted from directed content analysis techniques (Hsieh & Shannon, 2005). The directed approach is structured in nature with the purpose of answering targeted questions, as opposed to more conventional content analysis techniques where categories and themes are allowed to flow from the data (Hsieh & Shannon, 2005). A directed approach to analyzing the interview transcripts was considered appropriate because our primary objective was to elicit participant feedback and the interviews included targeted questions intended to expand our understanding of participants’ experiences using the wearable activity tracking devices. Two researchers (JA & KA) reviewed the interview transcripts independently. Then, one researcher (JN) extracted all of the participants’ comments about using the Fitbit devices and companion smartphone application, and organized these comments into a list by respondent. The same researcher then grouped similar comments together according to broad categories of feedback (e.g., positive, motivational, useful, difficult to use, challenging) that were representative of the data. The second researcher (KA) carefully reviewed the different categories of comments, and similarity between comments within categories, and overlap between categories. Both researchers (JN & KA) then met together to make revisions to the overarching categories, and resolved disagreements through discussion and consensus. Together, both researchers then identified key recommendations from participants’ comments to inform a future larger trial.

3. Results

3.1 Participant Characteristics and Use of the Wearable Devices

In total, 13 participants were recruited for the 6-month lifestyle intervention, but 2 participants dropped out due to scheduling conflicts and work commitments before starting the program and before receiving the mobile devices. The remaining 11 participants were given Fitbit Zip activity tracking devices and smartphones to use for the study duration. Among these 11 participants, 8 were women, all were non-Hispanic white, the mean age was 48.2±11.2 years (range 21 to 57), and 3 had schizophrenia spectrum disorders, 5 had major depressive disorder, and 3 had bipolar disorder. Demographic and clinical characteristics for these 11 participants are presented in Table 1.

Table 1.

Participant characteristics

Characteristic Total Sample
N 11
Demographic Characteristics
Mean Age (SD) 48.2 (11.2)
Female (%) 8 (73%)
Non-Hispanic white (%) 11 (100%)
Education
 Completed high school 5 (46%)
 Some college 3 (27%)
 College degree 3 (27%)
Living situation
 Living independently 8 (73%)
 Living with family 3 (27%)
Marital status
 Never married 3 (27%)
 Currently married 1 (9%)
 Previously married 7 (64%)
Currently Employed (part or full-time) 3 (27%)
Enrolled in Medicaid (%) 9 (82%)
Dual Eligible (enrolled in Medicaid and Medicare) 8 (73%)
Clinical Characteristics
Mental illness diagnosis (%)
 Schizophrenia spectrum disorders 3 (27%)
 Major depressive disorder 5 (46%)
 Bipolar disorder 3 (27%)
2 BMI (kg/m2) 41.5 (11.5)
Weight (lbs) 243.5 (53.2)
Current smoker (%) 3 (27%)
Open in a new tab

Over the study duration, 3 participants dropped out of the program for various medical or personal reasons. These 3 participants were in the study for between 68 and 159 days (about 10 to 23 weeks) before dropping out. Their Fitbit use data was calculated as the proportion of days enrolled in the study that they used the device before leaving the program. For participants who completed the 6-month lifestyle intervention, they had their devices to use for between 204 and 239 days (about 29 to 34 weeks) because of rolling enrollment at the start of the study, and because the study ran from October through April and included holidays. Figure 1 illustrates the proportion of days enrolled in the study that each participant used the wearable devices.

Figure 1.

Figure 1

Open in a new tab

Proportion of days enrolled in the study that participants’ wore their Fitbit wearable activity tracking device a, b, c

aParticipants wore their Fitbits for an average of 84.7% (SD=18.1%) of days enrolled in the study.

bParticipants wore their Fitbits for a median of 93.8% of the days enrolled in the study (Interquartile Range = 83.6–94.3%).

cNine out of eleven participants (82%) wore their Fitbits for over 80% of the days enrolled in the study.

Overall, participants wore their Fitbits for an average of 84.7% (SD=18.1%) of the days that they were enrolled in the study, with a median of 93.8% of the days enrolled (interquartile range = 83.6% to 94.3%). Nine of the eleven participants (82%) used their Fitbits for over 80% of the days that they were enrolled in the study. Primary reasons for not wearing the devices included device malfunction, dead battery, temporarily misplaced or lost the device, or just forgot to put it on. One participant only wore his device periodically throughout the entire study. When asked why he only wore it occasionally, he explained that he just forgot to put it on but that he really liked it. No participants permanently lost their Fitbits over the study duration, and all smartphones were returned undamaged except for one phone with a cracked screen due to accidentally being dropped.

3.2 Usability and Satisfaction Survey Responses

All 11 participants, including the 3 who dropped out of the 6-month program, completed the usability and satisfaction questionnaires (Table 2). Overall, participants were in agreement that the Fitbit was easy to use, and in general appeared to be highly satisfied with the device. Most participants agreed that the Fitbit helped them to be more active, and that it was fun to use and that they would recommend it to a friend. Importantly, participants also indicated that the Fitbit was something that they would like to continue to use in the future. There were only a few challenges with using the Fitbit, and these were mostly related to learning to use the smartphone and companion mobile application for the Fitbit.

Table 2.

Participant reported usability and satisfaction with the Fitbit, the companion mobile application, and the smartphone

Survey Item Strongly Disagree Disagree Disagree Somewhat Neutral Agree Somewhat Agree Strongly Agree
Usability
 The Fitbit is easy to use 0 0 0 0 0 6 5
 I can easily wear my Fitbit each day 0 0 1 1 2 1 6
 I learned to use the Fitbit quickly 0 0 0 0 0 5 6
 I easily remember how to use my Fitbit 0 0 0 0 0 6 5
 I can easily synch my Fitbit with the smartphone 1 0 0 0 1 5 4
 The Fitbit does everything I would expect it to do 0 0 1 0 0 4 6
 The Fitbit app on the smartphone is easy to use 1 1 0 0 1 3 5
 I learned to use the Fitbit app on the smartphone quickly 1 1 1 0 2 2 4
 I easily remember how to synch my Fitbit with the smartphone 0 1 0 0 0 4 6
 The Fitbit app on the smartphone does everything I would expect it to do 0 1 1 0 1 3 5
 The Fitbit works the way I want it to work 0 0 0 0 0 6 5
 The smartphone is easy to use 2 1 0 0 2 2 4
 I learned to use the smartphone quickly 0 3 0 0 2 2 4
Satisfaction
 The Fitbit is useful 0 0 0 0 0 4 7
 The Fitbit app on the smartphone is useful 0 2 0 0 0 3 6
 The Fitbit helps me keep track of my physical activity 0 0 0 0 2 3 6
 The Fitbit gives me more control over getting physical activity in my daily routine 0 1 0 1 0 3 6
 The Fitbit meets my needs 0 0 0 0 3 2 6
 I am satisfied with the Fitbit 0 0 0 0 0 5 6
 I would recommend the Fitbit to a friend 0 0 0 0 0 5 6
 The Fitbit is fun to use 0 0 0 0 2 2 7
 The Fitbit helps me be more active 0 1 0 0 2 2 6
 I would like to continue using the Fitbit in the future 0 1 0 0 2 1 7
 I feel I need to use a Fitbit 0 1 0 2 1 2 5
Open in a new tab

3.3 Participant Feedback from the Qualitative Interviews

The in-depth qualitative interviews were conducted with all 11 participants, including the 3 who dropped out of the 6-month program. We grouped participants’ feedback from the interviews into the following three categories:

3.3.1 Motivating, encouraging, fun to use, and other benefits

Participants shared positive experiences about using the Fitbits for activity tracking. Several participants found the devices fun and motivating, “It wasn’t just fun, it was like the one on one with myself and the Fitbit so it challenged me” (Participant #1). Most participants (N=8) used the Fitbit for setting daily step goals, while others explained that the Fitbit helped create a sense of accomplishment from being more active and collecting more steps: “Just seeing the steps on the Fitbit was actually good because I was able to read it and see that this is what I’ve done, and I was very happy about that. So I’ve accomplished certain goals that I set for myself.” (Participant #8).

The Fitbit was also helpful for increasing awareness of physical activity and for setting daily step goals. One participant explained how the Fitbit highlighted differences between her active and inactive days: “It makes me see how…when I was walking 10–14,000 steps a day and then one day I didn’t feel like doing anything, I only walked 2,000; it’s like, I gotta get up and start doing more stuff because that’s ridiculous.” (Participant #9). The Fitbit was even described as empowering by one participant, because it compelled her to work towards her daily step goal:

“It helped me, yes, it empowered me, if my step goal’s 5000 steps and I see I’m only doing 2000–3000 a day, I would almost feel guilty like I have to get to 5000, I can’t say I’m perfect that I do it all the time but it keeps it more in my mind.” (Participant #7).

Another participant found that the encouraging messages and prompts through the companion mobile application for the Fitbit helped her to reach her step goals:

“Oh yeah, because it would send a message to my phone that says, “You’re only 650 steps away from your goal!” then I would go oh yeah, I could do that easy. And it’s 500 steps or 600 steps, I can do that. So I just added a few more walks back and forth to the laundry room; so it did encourage me meet and often exceed my step goal.” (Participant #4).

For some participants, the Fitbit offered something tangible in the form of steps as proof of being physically active. One participant explained: “I was able to keep it in my pocket and take it everywhere I go. It just proves that when I’m exercising I gain steps from what I’m doing. It just shows that I can get more active. It shows that I’ve actually reached my goal for the day.” (Participant #8). Another participant described his experience using the Fitbit:

“I think that during the day when you’re walking, you don’t think of how much you’re walking. But you say, I walked that much? Holy Moley! It’s an emotional reward almost like, wow! I wanna do better! You can set the bar higher when you have something tangible to say, this is what I’ve done, or this is what I want to do.” (Participant #11).

Participants were also satisfied with the size of the device, and found it easy to understand, “it was small and it wasn’t overly bulky or anything like that, it was small and it was easy to understand” (Participant #6). And for some participants, they got so used to wearing the Fitbit each day that it became a habit and part of their routine: “in fact if I go out and don’t have it I feel like its missing and I have to have it on… I wear it all the time.” (Participant #7).

3.3.2 Other things the Fitbit can do

Several participants used different features on the Fitbit companion mobile application in addition to tracking steps. Three participants used the dietary tracking feature, 2 used the mobile application to track their weight, and 5 used the application to log how much water they consumed. The research staff did not highlight these features during the Fitbit training sessions. Participants enjoyed these additional features, stating: “I think the Fitbit was great because you could put in your weight and your water intake” (Participant #10). While others appreciated the convenience of having all these different features in one mobile application because it made it easier to use:

“The weight tracking, it was all in one place and immediately accessible. In terms of the step tracking, same thing. The food, same thing. It helped to have it all in one place and immediately accessible. It encouraged me to continue to do it because it was that easy.” (Participant #4).

Also, participants were enthusiastic about being able to see their own data on the mobile application: “I like that it’s all in one central location and I like the data I get from it with the charts to see my progress or if I’m struggling I could see on the graph what it looks like.” (Participant #7). And for some participants, as they became more familiar with using the Fitbit mobile application, then they started to explore more of the different features and viewing progress over time:

“Initially it was steps but then I would look at the other factors such as when I started putting my weight in, sometimes I look at the graphs of the charts that they had for steps to see how I was doing overtime in 3 months, and putting food and to see where my calories are, I used it much more than initially because initially it was just the steps.” (Participant #7).

Not all features of the Fitbit mobile application were considered useful. For example, participants’ considered the tracking of calories burned to be distracting and misleading: “that calorie tracking thing, I would be doing nothing sitting on the couch and it’d be in the morning and it would say I burned 900 calories, whatever the calorie tracking thing is on there for burning is very inaccurate” (Participant #7). For some participants, the calories burned was the only real problem with the Fitbit in general: “the fact that the calories burned really wasn’t relevant or accurate; it is what it is. That’s probably the only thing that I would say negative about it. Otherwise everything else was effective.” (Participant #4).

3.3.3 Technical difficulties, challenges and recommendations for improvement

There were a few challenges that emerged from the interviews, and these were mostly related to using the smartphone and companion mobile application for the Fitbit. For one participant with limited prior experience using mobile technologies, the phone was a source of discouragement, and it made her feel like she could not understand it: “Well, it worked but then it would stop working. I needed to fix it then it would work then it would stop working. It was kind of a pain in the neck so I got discouraged with that. It wasn’t the phone; it was because I didn’t know the phone.” (Participant #2). Other participants mentioned practical concerns with operating the smartphone such as small screen size, short battery life, and the need for basic instructions for making calls or shutting off the phone.

For the few participants with limited prior exposure to mobile technologies, learning to use the smartphone became a source of distraction: “We spent a lot of time on just trying to figure out how to use the phone, which is unfortunate.” (Participant #10). While another participant described her experiences learning to use the smartphone:

“Understanding how it worked. It was a little difficult. I’m not too fast on technology; it takes me a long time to use technology. When it comes to these phones, they’re a little bit more sophisticated than some of the stuff that I use. It was a challenge.” (Participant #6).

In contrast, there were no challenges related to using the actual Fitbit device, except remembering to wear it each day. Several participants described how they were able to problem solve and figure out ways to remember to wear their Fitbits, such as receiving helpful tips from other participants in the program or planning ahead:

“The only way I could remember to keep it on is because I wear my pants. What I ended up doing was keeping the Fitbit on my belt. When I took them off at night it would sit there and then the next day I would put it back on. That’s what helped me to remember, was that the Fitbit was on my belt all the time.” (Participant #6).

Some participants expressed the need for more instruction for using the smartphone and for accessing the companion mobile application for the Fitbit: “Yeah, I think you could have a little class to show us how to use them, do some exercises or stuff… The rest of us never had a smartphone. I think a class on how to use it would be helpful” (Participant #6). Other participants recommended group training and tutorials for introducing the technology instead of individualized training:

“If everyone’s given a phone at once and we all get an introduction; hold your phone, this is how you open your phone, all together instead of one at a time, it’s a time-saving thing.” (Participant #10).

4. Discussion

In this exploratory study we found that people with serious mental illness and obesity enrolled in a lifestyle intervention for weight loss enjoyed using Fitbit activity tracking devices and smartphones, and found that these devices were both motivating and useful for reaching daily step goals. Participants agreed that the devices encouraged them to be more physically active, and they were interested in continuing to use the devices in the future. Specifically, participants’ feedback highlights the acceptability and perceived benefits of wearable devices and smartphones for self-monitoring physical activity, facilitating goal-setting, and providing reminders and prompts to reach daily step goals and to get more exercise.

We have previously demonstrated the initial feasibility of using wearable activity tracking devices among people with serious mental illness (Naslund, Aschbrenner, Barre, et al., 2015). The current findings expand on our prior work by emphasizing the acceptability of using commercially available wearable devices and smartphones to support an evidence-based group lifestyle intervention targeting weight loss. Importantly, we also found that use of these devices was sustained over a 6-month period, which further advances the field of wearable mHealth technology research because to date most published studies of wearable devices have been of relatively short duration (Chiauzzi et al., 2015). Our findings are also consistent with studies highlighting the usability of wearable mHealth devices for activity tracking in general patient populations, including middle age and older adults with chronic medical conditions (Grindrod, 2014) and people with chronic obstructive pulmonary disorder (Vooijs et al., 2014).

Participants were open about challenges they may have faced when learning to use new and unfamiliar technology. Five participants, about half of our sample, reported not having any prior experience using smartphones. For these participants learning to use the smartphone and companion mobile application for the Fitbit was more difficult. However, all participants found the Fitbit wearable device easy to use and easy to understand. Recent studies have highlighted the increasing use and availability of mobile devices among people with serious mental illness (Firth et al., 2015), yet there remain many low-income individuals from this high-risk group who do not have access to contemporary smartphones. When implementing lifestyle interventions using smartphones or wearable mHealth technologies, additional careful consideration of the needs of the target population is essential to ensure that even individuals with limited prior knowledge or experience can similarly benefit.

By closely involving participants in the development, evaluation, and implementation of mHealth interventions, it can help to better understand how these emerging technologies can effectively support health promotion efforts and how to support adoption of these devices in real world settings. For example, development of the FOCUS smartphone application for schizophrenia self-management involved extensive usability testing and feedback from participants to inform specific design elements, including the use of larger buttons, less sensitive touchscreen, and larger font, to accommodate the needs of the at-risk target population (Ben-Zeev, Brenner, et al., 2014; Ben-Zeev et al., 2013). In the current study, we found that participants emphasized the need for more detailed training through group tutorials and more description of practical functions on the smartphones, which may have helped them feel more confident navigating the smartphone interface and accessing the companion mobile application for the Fitbit. As we use these findings to inform a future trial, it is clear that modifications to the way we introduce and deliver mHealth technologies are necessary to avoid excluding anyone who may be unfamiliar with using these devices.

Several participants showed interest in many of the different features on the companion mobile application for the Fitbit. As part of the lifestyle intervention, participants were not specifically instructed to use other features on the mobile application, but could choose to if they were interested. We learned that several participants used the mobile application for weight tracking, logging water, and tracking dietary consumption. This interest in different features highlights the potential for multi-component mHealth interventions that could address multiple health behaviors in this patient group. This is especially important given the high rates of other unhealthy behaviors such as smoking and poor diet among people with serious mental illness. Research shows that smokers with serious mental illness enrolled in lifestyle interventions for weight loss often express high interest in quitting smoking as well (Aschbrenner, Brunette, et al., 2015). This suggests that future mHealth interventions for people with serious mental illness could potentially target physical activity, healthy eating, and smoking cessation simultaneously.

4.1 Limitations

Given that this was an exploratory study, several limitations warrant consideration. First, the generalizability of the findings is limited due to the small sample size and lack of racial or ethnic diversity. Second, more quantitative data on participants’ use of the Fitbits, whether they achieved their step goals, and how they interacted with the companion mobile application would provide a more in depth understanding of the device usability among people with serious mental illness. Third, participant dropout was a concern in this study because 2 participants quit before starting the 6-month lifestyle intervention and before receiving the wearable activity tracking devices, and an additional 3 participants dropped out of the program after using the wearable activity tracking devices for at least 10 weeks. The reasons for quitting the program included personal and medical issues, as well as schedule conflicts. Future research should explore whether popular mHealth technologies and wearable activity tracking devices could be used to promote engagement in lifestyle interventions targeting weight loss among people with serious mental illness. Fourth, while our quantitative usability and satisfaction survey was based on a prior existing questionnaire, we were unable to locate published psychometric properties regarding the validity, reliability, or internal consistency of this measure. Lastly, due to the small sample size we were unable to investigate potential correlates associated with use of the wearable activity tracking devices. Future research should consider whether use of wearable activity tracking devices over time is associated with clinically meaningful outcomes such as weight loss or improved fitness.

Despite these limitations, our findings align closely with the objectives of a Stage Ia feasibility and acceptability study within the NIH Stage Model (Onken et al., 2014). An important strength with our study was that we were able to obtain quantitative and qualitative data surrounding the acceptability of using wearable devices and smartphones for activity tracking from the 8 participants who completed the 6-month program as well as the 3 participants who dropped out. The detailed perspectives and feedback from all participants offers valuable insights necessary to inform a subsequent larger scale study. Our future research will need to include more skill-based instruction and group tutorials to help individuals who may be unfamiliar with contemporary technologies feel confident using the smartphone and companion mobile application for the Fitbit. We will also offer instruction about how to use some of the additional features of the mobile application that may be of interest to participants.

5. Conclusion

This study offers a novel contribution to the rapidly advancing mHealth field because few prior studies have explored the potential for using mHealth technologies for health promotion in people with serious mental illness (Macias et al., 2015; Naslund, Aschbrenner, Marsch, McHugo, & Bartels, 2015). Efforts to promote physical activity in this group are urgently needed given their low cardiorespiratory fitness (Strassnig, Brar, & Ganguli, 2011), high obesity rates (Dickerson et al., 2006), and resulting early mortality gap (Walker et al., 2015). The sequential steps used to refine and evaluate the use of wearable devices and smartphones for activity tracking among people with serious mental illness align closely with recent emphasis at the NIH on adopting an iterative approach to intervention development and evaluation, especially in the context of emerging mHealth technologies for reaching high-risk groups (Marsch, Lord, & Dallery, 2015; Onken et al., 2014; Onken & Shoham, 2015). This Stage Ia study provides strong support for the acceptability and high satisfaction of using popular wearable activity tracking devices such as the Fitbit among people with serious mental illness, and adds to an increasing number of studies demonstrating that mHealth interventions are both feasible and acceptable for targeting this high-risk group (Alvarez-Jimenez et al., 2014; Naslund, Aschbrenner, Marsch, et al., 2015; Naslund, Marsch, McHugo, & Bartels, 2015). Building on this formative work, future research must focus on determining whether use of these devices contributes to improvements in objective health outcome measures, such as increased physical activity, weight loss, or improved cardiorespiratory fitness over time. Establishing the use of wearable activity tracking devices as a practical, feasible, and scalable approach to addressing the needs of the high-risk group of adults with serious mental illness and obesity is a necessary objective of subsequent effectiveness studies.

Highlights.

  • Participants with mental illness and obesity used wearable devices and smartphones.

  • mHealth devices were acceptable for physical activity monitoring and goal setting.

  • Some participants recommended more detailed training for using the devices.

  • Wearable mHealth devices are feasible to include as part of weight loss programs.

Acknowledgments

Grant support was received from the National Institute of Mental Health (R01 MH089811-01) and the United States Centers for Disease Control and Prevention Health Promotion and Disease Prevention Research Center (Cooperative Agreement Number U48DP005018). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

The authors report no competing interests.

References

  1. Alvarez-Jimenez M, Alcazar-Corcoles M, Gonzalez-Blanch C, Bendall S, McGorry P, Gleeson J. Online, social media and mobile technologies for psychosis treatment: a systematic review on novel user-led interventions. Schizophrenia research. 2014;156(1):96–106. doi: 10.1016/j.schres.2014.03.021. [DOI] [PubMed] [Google Scholar]
  2. Aschbrenner KA, Brunette MF, McElvery R, Naslund JA, Scherer EA, Pratt SI, Bartels SJ. Cigarette smoking and interest in quitting among overweight and obese adults with serious mental illness enrolled in a fitness intervention. The Journal of nervous and mental disease. 2015;203(6):473–476. doi: 10.1097/NMD.0000000000000309. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aschbrenner KA, Naslund JA, Barre LK, Mueser KT, Kinney A, Bartels SJ. Peer health coaching for overweight and obese individuals with serious mental illness: intervention development and initial feasibility study. Translational Behavioral Medicine. 2015;5(3):277–284. doi: 10.1007/s13142-015-0313-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aschbrenner KA, Naslund JA, Shevenell M, Mueser KT, Bartels SJ. Feasibility of Behavioral Weight Loss Treatment Enhanced with Peer Support and Mobile Health Technology for Individuals with Serious Mental Illness. Psychiatric Quarterly. 2015:1–15. doi: 10.1007/s11126-015-9395-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bai Y, Welk GJ, Nam YH, Lee JA, Lee JM, Kim Y, Dixon PM. Comparison of consumer and research monitors under semistructured settings. Medicine and Science in Sports and Exercise, [Epub ahead of print] 2015 doi: 10.1249/MSS.0000000000000727. [DOI] [PubMed] [Google Scholar]
  6. Bartels SJ, Pratt SI, Aschbrenner KA, Barre LK, Jue K, Wolfe RS, Mueser KT. Clinically significant improved fitness and weight loss among overweight persons with serious mental illness. Psychiatric Services. 2013;64(8):729–736. doi: 10.1176/appi.ps.003622012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bartels SJ, Pratt SI, Aschbrenner KA, Barre LK, Naslund JA, Wolfe RS, Bird B. Pragmatic replication trial of health promotion coaching for obesity in serious mental illness and maintenance of outcomes. American Journal of Psychiatry. 2015;172(4):344–352. doi: 10.1176/appi.ajp.2014.14030357. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Beebe LH, Harris RF. Using pedometers to document physical activity in patients with schizophrenia spectrum disorders: a feasibility study. Journal of psychosocial nursing and mental health services. 2012;50(2):44–49. doi: 10.3928/02793695-20120112-02. [DOI] [PubMed] [Google Scholar]
  9. Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophrenia Bulletin. 2014;40(6):1244–1253. doi: 10.1093/schbul/sbu033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: A smartphone system for self-management of schizophrenia. Psychiatric rehabilitation journal. 2013;36(4):289. doi: 10.1037/prj0000019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Ben-Zeev D, Kaiser SM, Krzos I. Remote “Hovering” With Individuals With Psychotic Disorders and Substance Use: Feasibility, Engagement, and Therapeutic Alliance With a Text-Messaging Mobile Interventionist. Journal of dual diagnosis. 2014;10(4):197–203. doi: 10.1080/15504263.2014.962336. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Ben-Zeev D, Wang R, Abdullah S, Brian R, Scherer EA, Mistler LA, Choudhury T. Mobile Behavioral Sensing for Outpatients and Inpatients With Schizophrenia. Psychiatric Services. 2015 doi: 10.1176/appi.ps.201500130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Bravata DM, Smith-Spangler C, Sundaram V, Gienger AL, Lin N, Lewis R, Sirard JR. Using pedometers to increase physical activity and improve health: a systematic review. Jama. 2007;298(19):2296–2304. doi: 10.1001/jama.298.19.2296. [DOI] [PubMed] [Google Scholar]
  14. Burke LE, Ma J, Azar KM, Bennett GG, Peterson ED, Zheng Y, Suffoletto B. Current science on consumer use of mobile health for cardiovascular disease prevention: a scientific statement from the American Heart Association. Circulation. 2015;132(12):1157–1213. doi: 10.1161/CIR.0000000000000232. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Case MA, Burwick HA, Volpp KG, Patel MS. Accuracy of smartphone applications and wearable devices for tracking physical activity data. Jama. 2015;313(6):625–626. doi: 10.1001/jama.2014.17841. [DOI] [PubMed] [Google Scholar]
  16. Chiauzzi E, Rodarte C, DasMahapatra P. Patient-centered activity monitoring in the self-management of chronic health conditions. BMC medicine. 2015;13(1):77. doi: 10.1186/s12916-015-0319-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Colton CW, Manderscheid RW. Congruencies in increased mortality rates, years of potential life lost, and causes of death among public mental health clients in eight states. Preventing Chronic Disease. 2006;3(2):A42. [PMC free article] [PubMed] [Google Scholar]
  18. Creswell JW. Research design: Qualitative, quantitative, and mixed methods approaches. 4th. Thousand Oaks, CA: Sage; 2013. [Google Scholar]
  19. Daumit GL, Dickerson FB, Wang NY, Dalcin A, Jerome GJ, Anderson CA, Gennusa JV., III A behavioral weight-loss intervention in persons with serious mental illness. New England Journal of Medicine. 2013;368(17):1594–1602. doi: 10.1056/NEJMoa1214530. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Daumit GL, Goldberg RW, Anthony C, Dickerson F, Brown CH, Kreyenbuhl J, Dixon LB. Physical activity patterns in adults with severe mental illness. The Journal of nervous and mental disease. 2005;193(10):641–646. doi: 10.1097/01.nmd.0000180737.85895.60. [DOI] [PubMed] [Google Scholar]
  21. Dickerson FB, Brown CH, Kreyenbuhl JA, Fang L, Goldberg RW, Wohlheiter K, Dixon LB. Obesity among individuals with serious mental illness. Acta Psychiatr Scand. 2006;113(4):306–313. doi: 10.1111/j.1600-0447.2005.00637.x. [DOI] [PubMed] [Google Scholar]
  22. Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile phone ownership and endorsement of “mHealth” among people with psychosis: a meta-analysis of cross-sectional studies. Schizophrenia bulletin. 2015:1–8. doi: 10.1093/schbul/sbv132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Folstein MF, Folstein SE, McHugh PR. Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research. 1975;12(3):189–198. doi: 10.1016/0022-3956(75)90026-6. [DOI] [PubMed] [Google Scholar]
  24. Green CA, Yarborough BJH, Leo MC, Yarborough MT, Stumbo SP, Janoff SL, Stevens VJ. The STRIDE weight loss and lifestyle intervention for individals taking antipsychotic medications: a randomized trial. American Journal of Psychiatry. 2015;172(1):71–81. doi: 10.1176/appi.ajp.2014.14020173. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Grindrod KA. Assessing the usability and usefulness of wearable activity trackers with adults over age 50: a mixed methods evaluation. Paper presented at the Medicine 2.0 Conference 2014 [Google Scholar]
  26. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qualitative health research. 2005;15(9):1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
  27. Kane I, Lee H, Sereika S, Brar J. Feasibility of pedometers for adults with schizophrenia: pilot study. Journal of psychiatric and mental health nursing. 2012;19(1):8–14. doi: 10.1111/j.1365-2850.2011.01747.x. [DOI] [PubMed] [Google Scholar]
  28. Lee PH, Nan H, Yu YY, McDowell I, Leung GM, Lam T. For non-exercising people, the number of steps walked is more strongly associated with health than time spent walking. Journal of Science and Medicine in Sport. 2013;16(3):227–230. doi: 10.1016/j.jsams.2012.10.005. [DOI] [PubMed] [Google Scholar]
  29. Lund AM. Measuring usability with the USE questionnaire. Usability interface. 2001;8(2):3–6. [Google Scholar]
  30. Lyons EJ, Lewis ZH, Mayrsohn BG, Rowland JL. Behavior change techniques implemented in electronic lifestyle activity monitors: a systematic content analysis. Journal of medical Internet research. 2014;16(8):e192. doi: 10.2196/jmir.3469. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Macias C, Panch T, Hicks YM, Scolnick JS, Weene DL, Öngür D, Cohen BM. Using smartphone apps to promote psychiatric and physical well-being. Psychiatric Quarterly. 2015;86(4):505–519. doi: 10.1007/s11126-015-9337-7. [DOI] [PubMed] [Google Scholar]
  32. Marsch LA, Lord SE, Dallery J. Behavioral health care and technology: using science-based innovations to transform practice. New York: Oxford University Press; 2015. [Google Scholar]
  33. Naslund JA, Aschbrenner KA, Barre LK, Bartels SJ. Feasibility of popular m-Health technologies for activity tracking among individuals with serious mental illness. Telemedicine and e-Health. 2015;21(3):213–216. doi: 10.1089/tmj.2014.0105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Naslund JA, Aschbrenner KA, Marsch LA, McHugo GJ, Bartels SJ. Crowdsourcing for conducting randomized trials of Internet delivered interventions in people with serious mental illness: a systematic review. Contemporary Clinical Trials. 2015;44:77–88. doi: 10.1016/j.cct.2015.07.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Naslund JA, Marsch LA, McHugo GJ, Bartels SJ. Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature. Journal of mental health. 2015;24(5):321–332. doi: 10.3109/09638237.2015.1019054. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science unifying the discipline to improve the public health. Clinical Psychological Science. 2014;2(1):22–34. doi: 10.1177/2167702613497932. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Onken LS, Shoham V. Technology and the stage model of behavioral intervention development. In: Marsch LA, Lord SE, Dallery J, editors. Behavioral health care and technology: using science-based innovations to transform practice. New York: Oxford University Press; 2015. p. 1. [Google Scholar]
  38. Patel MS, Asch DA, Volpp KG. Wearable devices as facilitators, not drivers, of health behavior change. Jama. 2015;313(5):459–460. doi: 10.1001/jama.2014.14781. [DOI] [PubMed] [Google Scholar]
  39. Richardson CR, Faulkner G, McDevitt J, Skrinar GS, Hutchinson DS, Piette JD. Integrating physical activity into mental health services for persons with serious mental illness. Psychiatric Services. 2005;56(3):324–331. doi: 10.1176/appi.ps.56.3.324. [DOI] [PubMed] [Google Scholar]
  40. Rooney B, Smalley K, Larson J, Havens S. Is knowing enough? increasing physical activity by wearing a pedometer. Wisconsin Medical Journal. 2003;102(4):31–36. [PubMed] [Google Scholar]
  41. Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: getting started and moving on from stage I. Clinical Psychology: Science and Practice. 2001;8(2):133–142. [Google Scholar]
  42. Strassnig M, Brar JS, Ganguli R. Low cardiorespiratory fitness and physical functional capacity in obese patients with schizophrenia. Schizophrenia research. 2011;126(1):103–109. doi: 10.1016/j.schres.2010.10.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Vooijs M, Alpay LL, Snoeck-Stroband JB, Beerthuizen T, Siemonsma PC, Abbink JJ, Rövekamp TA. Validity and usability of low-cost accelerometers for internet-based self-monitoring of physical activity in patients with chronic obstructive pulmonary disease. Interactive journal of medical research. 2014;3(4):e14. doi: 10.2196/ijmr.3056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden implications: a systematic review and meta-analysis. JAMA psychiatry. 2015;72(4):334–341. doi: 10.1001/jamapsychiatry.2014.2502. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Yingling LR, Todaro-Brooks A, Wallen GR, Peters-Lawrence M, McClurkin MA, Copper-McCann R, Powell-Wiley T. Community engagement to optimize the use of web-based and wearable technology in a cardiovascular health and needs assessment study: a mixed methods approach. JMIR mHealth and uHealth. 2016 doi: 10.2196/mhealth.4489. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Zulman DM, Jenchura EC, Cohen DM, Lewis ET, Houston TK, Asch SM. How can eHealth technology address challenges related to multimorbidity? Perspectives from patients with multiple chronic conditions. J Gen Intern Med. 2015;30(8):1063–1070. doi: 10.1007/s11606-015-3222-9. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES