Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Apr 14.
Published in final edited form as: J Form Des Learn. 2020 Jan 6;4(2):51–64. doi: 10.1007/s41686-019-00038-x

User experience (re)design and evaluation of a self-guided, mobile health app for adolescents with mild Traumatic Brain Injury

Matthew Schmidt 1, Allison P Fisher 2, Joshua Sensenbaugh 3, Brandt Ling 3, Carmen Rietta 4, Lynn Babcock 2, Brad G Kurowski 2, Shari L Wade 2
PMCID: PMC8046025  NIHMSID: NIHMS1677448  PMID: 33860150

Abstract

Mild Traumatic Brain Injury (mTBI) is a significant cause of morbidity for adolescents. Currently, there is a lack of evidence-based interventions to address common sequelae of mTBI. To address this gap, we designed a program to promote recovery for adolescents following mTBI. Preliminary testing of the Self-Monitoring Activity Regulation and Relaxation Treatment (SMART) program demonstrated good usability but indicated a need for modifications to the program. The SMART application was redesigned with the addition of more interactive and gamified components. Content was also reframed to specifically target and engage adolescents with mTBI. We describe the usability evaluation of the updated SMART application. Children aged 11–18 years diagnosed with mTBI who were 1 to 6 months post mTBI were recruited to participate in a 1–2-hour usability session in which they thought aloud and responded to targeted usability-related questions during their interaction with the SMART program. After completing the session, participants rated their usability experience using the System Usability Scale (SUS) and rated the overall user-friendliness of the program. Participants’ responses during the session were qualitatively coded and analyzed. Six adolescents participated in a usability session (average age = 13.7 years). On the SUS, participants rated the program as highly usable (M = 85.6, SD = 3.24). They also had overwhelmingly positive feedback regarding the content, design and structure of the program. Overall, findings suggest that the redesigned SMART program was usable, acceptable, and relevant to adolescents with mTBI. Based on adolescents’ feedback, additional modifications were made before the program undergoes efficacy testing.

Keywords: concussion, mTBI, adolescent, m-health, recovery, rehabilitation

Over 800,000 individuals ages 5–24 seek acute care in the United States for a Traumatic Brain Injury (TBI) each year (Taylor, Bell, Breiding, & Xu, 2017). It has been estimated that approximately 79% of all TBI’s that occur are mild TBIs (mTBIs) (Cassidy et al., 2004). Sustaining a mTBI results in a constellation of physical, cognitive, and emotional symptoms that affect everyday functioning and quality of life (Kraus, Hsu, Schafer, & Afifi, 2014; Papoutsis, Stargatt, & Catroppa, 2014; Røe, Sveen, Alvsåker, & Bautz-Holter, 2009). Approximately one-third of adolescents who sustain a mTBI will experience persistent symptoms one-month post injury (Zemek et al., 2016). Factors that have been shown to contribute to prolonged recovery include age, sex, injury-related characteristics and pre-injury psychological or learning difficulties (Babcock et al., n.d.; Baker et al., 2016; Emery et al., 2016; Zemek et al., 2016). Additionally, anxiety, depression, and distress are associated with prolonged symptoms after injury (Scheenen et al., 2017). The societal costs of prolonged recovery are considerable in terms of school absences, missed days of work, and impaired social and community functioning (Babcock et al., n.d.; Cancelliere et al., 2014; Kraus et al., 2014; Nowacki et al., 2017).

Traditional treatment recommendations originally emphasized strict cognitive and physical rest until symptoms subside (McCrory et al., 2005). More recent evidence indicates that a paced return to everyday activities, after only a few days of rest, may promote recovery (Lumba-Brown et al., 2018; Silverberg & Iverson, 2013). Importantly, this emphasis on self-regulation represents a substantial change in mTBI management that is consistent with a problem-focused coping/problem-solving approach to actively managing symptoms and taking control of one’s recovery (Nezu & D’Zurilla, 2006). Adolescents may also benefit from active strategies, like problem solving, for managing other common symptoms of mTBI such as headaches, fatigue, and attention and concentration challenges (Woodrome et al., 2011).

As highlighted in the recent guidelines for pediatric mTBI developed by the Centers for Disease Control, support for existing approaches to symptom management in children and adolescents is limited (Lumba-Brown et al., 2018). Three published trials demonstrated that written anticipatory guidance and psychoeducation initiated soon after injury decreased symptom burden compared to controls (Glang, Koester, Beaver, Clay, & McLaughlin, 2010; Glang et al., 2015; Ponsford et al., 2001). Additional studies provide limited evidence that cognitive behavioral therapy and relaxation training may also be effective (Al Sayegh, Sandford, & Carson, 2010; Baker et al., 2016; Gravel et al., 2013). The current approach to treatment, which consists of intermittent evaluations and associated guidance by health care professionals in typical office-based settings, lacks the flexibility to address individual variations in symptom patterns and fails to promote patient self-management.

eHealth and mHealth Interventions for Adolescents

Health services and information delivered via the Internet, also known as electronic health programs or “eHealth” (Eysenbach, 2001), seek to improve the physical and/or mental health of a user (Oh, Rizo, Enkin, & Jadad, 2005). eHealth is being used increasingly to promote disease management and coping with varied conditions, such as arthritis, depression and chronic headaches (Organization, 2010) (World Health Organization, 2011b). With the recent surge in use of smartphones and mobile applications (“apps”), the eHealth landscape has shifted focus to mobile health applications, or “mHealth.” According to the World Health Organization, mHealth is a subset of eHealth and is the utilization of mobile devices, such as mobile phones, personal digital assistants, patient monitoring devices, and other wireless devices, for the purpose of medical and public health practice (World Health Organization, 2011a).

With the rapid advancements in mobile technologies, consideration of how the structure, function, and semiotics of mHealth might influence patient outcomes is needed [28]. One area where mHealth holds particular promise is in aiding the transition from inpatient treatment to outpatient follow-up care. Treatment of serious medical conditions usually does not end when patients leave the hospital. Follow-up care and monitoring is often necessary. mHealth applications provide an avenue to smooth the transition from treatment to follow-up care, such as following long-term chemotherapy for pediatric cancer, or after hospitalization for severe TBI (Schwartz et al., 2019). The use of mHealth can also reduce travel burden to major medical centers, especially for rural families or those with lower socioeconomic statuses (SES) (Zhang et al., 2019).

Mobile technologies are tremendously popular with adolescents, with approximately 95% of having access to cell phones and around 45% reporting that they go online on a near-constant basis (Anderson & Jiang, n.d.; Lenhart et al., 2015). The strong presence of mobile devices in the everyday lives of adolescents necessitates mHealth designs that are both familiar and accommodating to the needs of adolescents. For example, mHealth technologies might be leveraged to change complex adolescent behavior (i.e., sexual health, smoking, and medical adherence) (Bull et al., 2016; Ybarra, Prescott, Mustanski, Parsons, & Bull, 2019). In addition, mHealth potentially could increase access and treatment engagement for adolescents involved in the juvenile-justice system (Bath, Tolou-Shams, & Farabee, 2018) (Bath, Tolou-Shams, & Farabee, 2018).

A number of mHealth tools have been built for the adolescent population. Schwartz and colleagues (Schwartz et al., 2019) describe an adolescent and young adult mobile application called STEPS (Self-Management via Texting, Education, and Plans for Survivorship) for survivors of childhood cancer. STEPS delivers tailored text messages focusing on cancer survivor self-management during follow-up care. STEPS’ goal is to improve and facilitate the enhancement of health and well-being in adolescents. Another program, called Guy2Guy, was developed as a mHealth HIV prevention program targeted at sexual minority boys from 14 to 18-years-old (Ybarra et al., 2019). Guy2Guy was designed to deliver educational and skill-oriented general health information (e.g., self-esteem, bullying, and positive body image) for adolescent males identifying as gay, bisexual, and/or queer. Lastly, O’Brien and colleagues (O’Brien et al., 2019) created a mHealth tool with the purpose of providing a brief alcohol intervention for suicidal adolescents recently discharged from inpatient hospitalization. Because suicidal adolescents who misuse and abuse alcohol are at high risk for suicide following hospitilization, their mHealth program provides an opportunity for reaching a population that possesses a high rate of lethality.

eHealth and mHealth Interventions for Brain Injuries

In the area of brain injuries, a variety of eHealth and mHealth interventions have demonstrated benefits across a range of conditions (e.g., concussion/TBI, acquired brain injury; ABI). Among adolescents with TBI, individual web-based interventions have been shown to decrease externalizing behaviors and internalizing symptoms while improving problem solving, executive functioning, and health management behaviors (S. L. Wade et al., 2011; Shari L. Wade et al., 2012; Shari L. Wade, Wolfe, Brown, & Pestian, 2005; Worthen-Chaudhari et al., 2017). Similarly, web-based family interventions have demonstrated improved behavioral outcomes in children with TBI (S.L. Wade et al., 2018). Further studies are currently in the pilot stages and have yet to explore outcomes from randomized and controlled trials (Schwartz et al., 2019; Ybarra et al., 2019). An example from our own work is the Social Participation and Navigation (SPAN) program, which was found to be feasible and effective in achieving self-identified social participation goals in children with acquired brain injuries; however, these results are preliminary and need replicated in a controlled trial (S.L. Wade et al., 2018). A series of five randomized, controlled trials, a comparative effectiveness study and meta-analysis support the efficacy of an online family problem solving approach in improving externalizing behavior problems and executive dysfunction in adolescents with moderate to severe TBI (Zhang et al., 2019). However, differences in treatment timing and management goals (externalizing behavior problems versus fatigue, headaches, and other symptoms) necessitated a novel, but conceptually related, intervention for acute mTBI, which we detail below.

The high frequency of mTBI in adolescents (Langlois, Rutland-Brown, & Thomas, 2004) and the low cost of self-guided internet-based interventions (Andersson & Titov, 2014) highlight the potential widespread utility of an intervention tailored for this population. However, to date only one mHealth intervention has been trialed in a pilot randomized clinical trial with adolescents with mTBI (Worthen-Chaudhari et al., 2017). This study piloted SuperBetter, a mobile app that uses “gamification,” that is, the application of video game principles in non-game contexts (Deterding, Dixon, Khaled, & Nacke, 2011), to promote resiliency and reduce negative affect in adolescents with persistent symptoms following mild to moderate TBI. Researchers found that symptoms after injury and optimism improved more for users of the app than for the active comparison (Worthen-Chaudhari et al., 2017). However, although SuperBetter was tested with adolescents with mTBI, it was not designed specifically to address mTBI consequences or promote strategies for dealing with them. To address the need for an intervention specifically tailored for adolescents in the acute phase following mTBI, we developed SMART (Self-Monitoring Activity Regulation and Relaxation Training), the design, development, and evaluation of which we describe in the following sections.

Design and Development Iteration One: SMART Version 1.0

The current article builds on our previous work, which investigated the impact of SMART version 1.0 (v.1.0), a software application designed to promote resiliency and recovery in adolescents (ages 12–17) following mTBI. SMART v.1.0 was developed in 2012 using a three-phase process of key-informant interviews and stakeholder input, usability testing [29], and an open single-arm pilot (Kurowski et al., 2016). SMART v.1.0 was a web-based application consisting of two main components: (1) daily symptom and activity monitoring that provided personalized feedback about symptom changes to promote self-management, and (2) learning modules that provided training in problem solving and strategies to manage mTBI-related symptoms such as stress management and relaxation (Figure 1). Availability of modular learning content was based on time since injury and current symptom scores. The first module was released to all users 24 hours after the injury. The other modules were released based on the date of injury and symptom scores from questions asked to participants daily. For example, participants with more severe scores were given an extra day of rest before any additional modules would open. Results from structured formative evaluation suggested that SMART had high usability. At completion of the single-arm pilot trial, all 13 participants endorsed the program as helpful and rated their symptoms as having returned to pre-injury symptom levels at 1-month post injury, with no adverse events or symptom exacerbations.

Figure 1.

Figure 1.

Open in a new tab

SMART v.1.0 learning modules (left) and symptom/activity monitoring input (right).

Usability testing of SMART v.1.0 uncovered the need for: (1) more pictures and graphics, (2) less text per page, (3) the addition of voiceovers, and (4) more videos of children undergoing similar experiences (Dexheimer et al., 2017). Furthermore, findings suggested limited engagement with the intervention in the open-trial. Despite reported high usability and a lack of adverse events, participant feedback suggested a need for substantial refinements to SMART v.1.0 program. Hence, we underwent further design and development to improve SMART along identified dimensions.

Materials and Methods

Design and Development Iteration Two: SMART v.2.0

Based on findings from design and development iteration one, and given high adolescent engagement with mobile devices, a primary goal in the redesign of SMART was to port SMART v.1.0 from a more traditional, web-based eHealth program to a more modern, mHealth design, intended for delivery primarily on smartphones and tablets (Kurowski et al., 2016). Within this frame, we focused on redesigning specific SMART components to improve usability, accessibility, and engagement. As in the prior version of SMART, the new design consisted of both a symptom monitoring/self-management component, as well as eight learning modules. In the following sections we detail the redesign process and subsequent, two-phase formative evaluation.

User Interface Redesign.

Because SMART v.2.0 was to be delivered on smartphones and tablets, the UI was completely re-imagined for delivery on smaller screens and single-handed operation. The prior, web-based user input methods (e.g., radio buttons, drop-downs, etc.) were converted to mobile design patterns (e.g., sliders, buttons, etc.). Graphical elements were added, such as iconography representing mTBI symptoms and “emojis” that replaced numeric Likert scales (Figure 2).

Figure 2.

Figure 2.

Open in a new tab

Redesigned elements of symptom monitoring patient data input screens in SMART v.2.0 based on mobile design patterns, including (1) replacement of numeric Likert scales with emojis, (2) sliders and iconography, and (3) buttons.

Learning Module Segmentation, Content, and Media Redesign.

A content review was performed by subject matter experts (sixth, seventh, and eighth authors on this paper) on the learning modules from SMART v.1.0. Content segmentation was reworked based on lessons learned from SMART v.1.0 and recent advances in the literature. The new set of learning modules consisted of guidelines and strategies for returning to school and physical activities, self-care (including stress management and nutrition), staying positive, staying focused, problem solving, and self-advocacy. A description of the SMART v.2.0 learning modules as contrasted with those from the previous version of SMART can be found in Table 1.

Table 1.

SMART module titles and descriptions.

SMART v.1.0 SMART v.2.0 Description
Introduction + Self Management Introduction A basic overview and introduction to the application and mTBI recovery.
Symptom Maintenance Symptom Monitoring Information about common symptoms, timelines for recovery, and strategies for coping.
Return to Activities Return to Activities Guidelines for returning to school and athletic activities.
Taking Care of You Taking Care of You Strategies for healthy functioning including adequate sleep, proper nutrition, and stress and relaxation techniques.
Staying Positive Staying Positive Training in cognitive reframing strategies to address worries and negative cognitions about symptoms and missed activities.
Staying Focused Staying Focused Tips for minimizing distractions and coping with attention and concentration difficulties.
Stop, Think, Problem Solve Problem Solving Training in 5-step problem solving heuristic (Aim, Brainstorm, Choose, Do It, Evaluate) to address concerns regarding mTBI-related issues.
Managing Stress Self-advocacy Guidelines and strategies for working with the school and other non-athletic activities to make accommodations and ensure a successful re-entry without symptom exacerbation.
Open in a new tab

Module content, which previously had consisted primarily of long, paginated sections of text interspersed with occasional related imagery, was reworked, simplified, and transformed into interactive multimedia formats. Gamification elements were incorporated, including a simple points system in which participants earned stars for completion of modules and activities, as well as an achievement system that awarded digital badges (Figure 3). Also included were mini-games to help practice learned skills. To guide participants through modules, we opted to use pedagogical agents. These pedagogical agents were designed to act as more knowledgeable peers, and were represented as avatars. The avatars were designed to look like anime figures to increase their appeal to adolescents. In SMART v.2.0, pedagogical agents (represented as avatars) guide patients through relevant case-based scenarios, demonstrating and discussing common mTBI-related symptoms and strategies for coping and self-management (Figure 4).

Figure 3.

Figure 3.

Open in a new tab

Elements of gamification in SMART v.2.0 include the ability to earn stars for completing various tasks and activities (top) and to earn achievements for completing entire learning modules (bottom).

Figure 4.

Figure 4.

Open in a new tab

Screenshots of SMART v.2.0 modules showcasing pedagogical agents (represented as avatars) that serve as more knowledgeable peers, coaching and guiding patients through learning modules.

Two Phase Formative Evaluation of SMART Version 2.0

In the following sections, we report findings from a two-phase formative evaluation of SMART v.2.0 and how these phases informed further refinements of program content. Design and development of SMART v.2.0 began in early Fall of 2017. A needs analysis was conducted, followed by delivery of initial UI design mockups and subsequent iterations. The design process consisted of weekly design critiques during which internal stakeholders provided feedback and direction for both interface design and learning content development. Iterative design and development continued until mid-Spring of 2018, when a two-phase user-experience focused formative evaluation was initiated. In Phase 1, a focus group was held with representative participants and their parents (n=7; 3 adolescents, 4 parents). Feedback from the focus group was considered and incorporated into the design so as to improve the UI as well as learning modules visual design and content format. Phase 2 began in July and continued through August of 2018, during which iterative usability testing was conducted with adolescents (n=6). Participants’ perceptions of technical and pedagogical usability were investigated over six sessions. After each usability session, refinements were made to the interface and modules before the next usability session. Examples of usability refinements include increasing font sizes to improve readability and reducing background brightness to reduce eye strain–both important factors for individuals affected by associated sequelae of mTBI.

Focus Group Procedures.

The focus group was conducted in April, 2018 and lasted approximately two hours. A single facilitator led the focus group, with two research coordinators taking field notes. The focus group facilitator provided a live demonstration of SMART v.2.0, including the UI and the learning modules. Slides depicting SMART v.1.0 were also shown to participants. The focus group protocol centered around the following themes: (1) input of symptoms in the newly designed UI, (2) comparison of the old UI to the new UI, (3) learning module look-and-feel, and (4) learning module content.

Focus Group Participants.

The focus group included four mothers and three adolescents (one adolescent was unable to attend). Participants skewed at the high end of the target age range, but with some limited diversity in terms of parental education, sex, and race. Details of focus group participants are provided in Table 2 below.

Table 2.

Focus group participants.

Adolescent Description Parent Description
18 year old Caucasian female. Mother commented that she did not have a college degree.
16 year old Caucasian male. Mother’s profession/educational background unknown.
15 year old Asian female. Mother is a teacher.
Unable to attend Mother is a nurse.
Open in a new tab

Focus Group Data Analysis.

Data were gathered by two research coordinators in the form of detailed field notes. Following the focus group, the two research coordinators met and synthesized their field notes into a single document that summarized the principal findings from the focus group session. The summary document was organized around the key themes described in section 2.2.1 above. Feedback from the focus group was considered by the design team and used to refine the design of SMART v.2.0 before entering the second phase of formative evaluation, usability testing.

Usability Testing Procedures

Usability testing was conducted over two months in July and August of 2018. Usability testing is particularly important with this population given that common mTBI-related symptoms (sensitivity to light, sensitivity to information overflow) may significantly affect intervention usability and potentially exacerbate symptoms. A total of six usability sessions were conducted. Four usability sessions were facilitated entirely by a PhD-level trained usability evaluator. Two sessions were partially facilitated by a graduate student trainee under the guidance of the PhD-level evaluator. At least one additional observer was present at all usability sessions except for Participant 17 (due to a scheduling conflict). During the usability sessions, each participant completed two tasks. The first task focused on the technical usability of the symptom and activity monitoring portion of the SMART program. The second task focused on the usability of one of the learning modules (e.g. “Taking Care of You”, “Staying Focused”, “Self-advocacy”), with participants being randomly assigned to one of the modules. Participants were asked to “think aloud” as they used the SMART program. These primary tasks were interspersed with sub-tasks. For example, the facilitator asked participants where they could go to find additional information, how many pages they had left to complete, and how to return to the previous page. Open Broadcaster was used to record sessions, including screen, webcam, and audio recordings. Four participants completed their usability sessions on a computer and two completed on an iPad mini. Upon completion of each usability session, participants completed the 10-item System Usability Scale (SUS) (Brooke, 1986) and rated overall user-friendliness.

Usability Testing Participants.

A total of six participants, ages 11–18 years (x¯=13.7), were recruited via social media and through the Sports Medicine Clinic in Cincinnati Children’s Hospital Medical Center. Participants were included if they were English-speaking and had sustained a mild traumatic brain injury, as indicated by a Glasgow Coma Scale score of 13–15, one to six months prior to enrollment. Participants were excluded if they had severe pre-existing neurologic or cognitive disorders (e.g., autism, intellectual disability). Parents provided informed consent for children less than 18 years of age, with adolescent participants providing assent if less than 18 or consent if 18 years of age or older. This project was approved by our Institutional Review Board. Participant demographics are provided in Table 3.

Table 3.

Usability testing participant demographics.

Demographics
Participant No. Gender Age Ethnicity Recruitment site
Participant 11 Male 17 Caucasian Rehabilitation clinic
Participant 15 Male 12 African-American Sports-medicine clinic
Participant 17 Female 13 Caucasian Sports-medicine clinic
Participant 18 Female 15 Caucasian Sports-medicine clinic
Participant 19 Female 11 Caucasian Social media
Participant 20 Male 14 Caucasian Social media
Open in a new tab

A short technology competence survey was administered with the participants focusing on self-reported ability and experience using smartphones and online learning. Participants reported their abilities using smartphones (expert=0; experienced=4; capable=2, novice=0), using the Internet (expert=0; experienced=4; capable=2; novice=0), frequency of smartphone usage (more than once per hour=4; once per hour=1; less than once per hour=0; data missing=1), prior experience with online learning (yes=4; no=2), and estimated total time per week spent using the Internet (over 40=1; around 40=3; between 20 and 40=2; less than 20=0).

Post-test Usability Debrief.

After each usability session, all present stakeholders (e.g., facilitator, observers, graduate students) would hold a debriefing meeting. During the meeting, stakeholders would identify any usability problems that were observed during the session, and then prioritize those problems on a five point severity scale (0 = “I don’t agree that this is a usability problem at all”, 4 = “Usability catastrophe: imperative to fix this before product can be released”; (Nielsen, 1995)). These problems were then addressed in order of priority before conducting the next usability study so as to iteratively improve the design.

Quantitative Data Analysis.

Quantitative data were collected using the System Usability Scale (SUS) (Brooke, 1986), an industry-standard scale consisting of 10 items, each rated on a five-point Likert-scale (strongly agree to strongly disagree). The SUS has been used to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites, and computer applications. It has been shown to be easy to administer and to provide reliable and valid results even with small samples (Brooke, 1986). SUS scores were calculated by subtracting one from all odd items and adding one to all even items, then multiplying the sum of all scores by 2.5. These normalized SUS scores were evaluated on a scale of 100, with scores above 68 being considered to be above average usability (Sauro & Lewis, 2012). In addition to the SUS, the single item user-friendliness scale was administered (A. Bangor, Kortum, & Miller, 2009), which states, “Overall, I would rate the user-friendliness of this product as: Worst Imaginable, Awful, Poor, OK, Good, Excellent, or Best Imaginable.” This item was added to help with interpretation of SUS scores and explanation of results.

Additionally, the usability scores were compared between the former SMART version and the newer SMART program to ensure that the usability scores of the new program were at least equivalent to the scores associated with the former program. These data were used to modify the SMART program with additional input from the design team.

Qualitative Data Analysis.

Qualitative usability data (think-aloud videos, screen recordings) were analyzed using video analysis techniques. A coding scheme was developed along the dimensions of (1) usability, (2) efficiency, (3) effectiveness, and (4) satisfaction using guidelines from ISO/IEC 25010 Systems and software Quality Requirements and Evaluation (SQuaRE) System and software quality models (International Organization for Standardization (2011) Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, n.d.). Our focus was specifically related to quality in use, or “the degree to which a product or system can be used by specific users to meet their needs to achieve specific goals.” Our focus was not on the more general product quality aspect of ISO/IEC 25010. Overarching measures and descriptions are outlined in Table 3.

Four graduate students (coders) were trained on how to conduct video analysis using the coding scheme. Training consisted of a calibration phase, during which coders practiced applying the coding scheme to the usability videos. After each calibration session, all coders would identify and discuss any discrepancies and develop coding guidelines to promote inter-rater reliability. Once all coders were able to code with high inter-rater reliability, they then independently coded the videos. Each video was reviewed and coded by two coders. Coder agreements and disagreements were then used to calculate inter-rater reliability using Cohen’s Kappa. Kappa was determined to be high (d = 0.87), suggesting high agreement among coders.

Results

In the following sections, results are presented from the focus group and usability testing Usability results are presented along the dimensions (detailed in Table 3) of efficiency, effectiveness, and satisfaction, as per SQuaRE (International Organization for Standardization (2011) Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, n.d.).

Focus Group Results

Overall, participants expressed a preference for the SMART v.2.0 UI over the previous version. Participants remarked positively regarding imagery used, simplicity in visual design, and mobile-friendly input mechanisms like sliders. Related to symptom input, all adolescent participants indicated they had frequently filled out similar symptom monitoring checklists (but without images). All participants indicated they liked the symptom imagery before the facilitator invited any questions in this regard. One participate stated, “It would be helpful to look at a picture over words.” Participants stated they liked the imagery because reading was hard directly after injury, e,g., “pictures would have helped because reading at all was very hard for awhile.” They also appreciated that there were no colors in the images, as all said they would have found too much color difficult to handle immediately after injury. However, regarding the reporting of activity, participants found it to be overly cognitively intensive. As a result, this section of the UI was reworked and simplified following the focus group.

When presented with the sample learning module “Taking Care of You,” participant feedback was mixed. All participants remarked positively regarding the overall look-and-feel, as well as the color scheme used. Further, there were specific positive comments about some of the content elements; for example, one participant stated, “That was deep. Like dang I didn’t even think that.” However, as the conversation progressed, there were also statements to the effect that the content seemed “young” for them (no participants were under the age of 15). There was also agreement that incorporating videos of real people might be helpful, and participants provided suggestions for modifying the star system showing content completion, although there was no consensus around these ideas. Parents wanted something for themselves in the system, with one parent stating, “I don’t know what I’m going to see or find when I walk through the door [and interact with my daughter each day].” Although the facilitator noted the boundaries of the current design goals, participants’ divergent ideas were nonetheless explored in an attempt to identify the extent to which these requests might be incorporated.

A specific area that was identified as being problematic was the naming conventions used to label subsections of modules. Taking a gamification approach led to titles such as “Chill Out!” (for a section on relaxation) and “Mind over Matter” (for a section on fatigue). However, participants were not sure what kinds of information they would encounter if they explored those areas of the modules. As a result, all participants indicated a desire for help finding resources. This led to less playful and more descriptive titles being used in future iterations, such as “How to De-Stress” and “Coping with Fatigue.”

Usability Study Results

Efficiency.

For Task 1, participants were asked to input their symptoms using the refined SMART v.2.0 UI. All participants successfully completed Task 1. This overarching task was comprised of three sub-tasks. In the first subtask, participants input the severity of their symptoms for the day. In the second subtask, participants input their activity for the day. In the third subtask, participants input their anticipated activity for the next day. On average, participants were able to complete the entirety of task one in 00:21:24, with a minimum time to completion of 00:12:43 and a maximum time to completion of 00:29:18 (SD = 00:04:13). On average, the first subtask required 00:11:48, the second subtask required 00:05:31, and the third subtask required 00:04:05.

For Task 2, participants were asked to complete one section (out of four) of a learning module (assigned at random) within SMART v.2.0. All participants successfully completed Task 2. If the participant completed a section and there was enough time left in the session presumably to complete a second session, he or she was asked to complete another section. Because each learning module differed in terms of amount of content and module structure, some participants were only able to complete a single section, while others were able to complete all four sections. Hence, data are presented here only for the one section of the modules that all participants completed. Across all modules, participants were able to complete section 1 in 00:10:43, on average, with a minimum time to completion of 00:04:00 and a maximum time to completion of 00:16:22 (SD = 00:04:16). Few participants made comments about the length of Task 1 and Task 2; however, one participant said that the “Symptom Monitoring” module was too long to complete without a break.

Effectiveness.

Participants were able to navigate through the application with few misunderstandings, errors, or questions. For Task 1, only four errors were observed across all participants, three of which were made by a single participant (Participant 15). On average, participants made fewer than one error when completing Task 1 (x¯=.67, SD=1.21). For Task 2, a total of five errors were observed across all five participants, again with the majority of errors being made by a single participant (Participant 15). On average, participants made fewer than one error when completing Task 2 (x¯=.83, SD=1.33). All participants were able to exit learning modules, although half experienced some trouble before finding the close button. Some participants experienced difficulty when prompted to identify different menus, instructions, and icons (e.g., an information icon labeled “i”) and required help. Two participants were unable to locate the “menu” button independently.

Data suggested that participants misunderstood some aspects of SMART while engaging in Task 1 and Task 2. Per participant and on average, 2.0 instances of misunderstanding were identified for Task 1 (SD = 1.41) and 1.33 instances for Task 2 (SD = 1.03). Misunderstandings varied in nature. Some participants were not certain of the definitions of specific terminology used for self-monitoring, which were taken from the Post-Concussion Symptom Scale (Kontos et al., 2012), specifically the meanings of the words “insight” and “fatigue.” In addition, some misunderstandings occurred when the consequences of an action did not match what the participants expected to happen. For instance, one participant explained that she did not expect to see the section’s menu page after clicking a “Next” button. Despite minor misunderstandings, participants’ were able to effectively complete both tasks.

Satisfaction.

Satisfaction measures assessed using the SUS for Task 1 indicate high participant usability ratings (x¯ = 84.6, SD = 0.74), with the score for Task 1 of nearly 85 falling somewhere between “good” and “excellent” (A. Bangor et al., 2009) Adjectival user-friendliness scores are very highly correlated with SUS scores (Aaron Bangor, Kortum, & Miller, 2008). User friendliness scores for Task 1 (x¯ = 6.0, SD = 0.0) suggest “excellent” user-friendliness.For Task 2, participants rated the learning modules slightly higher, suggesting “excellent” usability (x¯ = 86.7, SD = 0.72). Correspondingly, user friendliness was rated as “excellent” (x¯ = 6.2, SD = 0.75).

Combined usability ratings on the SUS for SMART v.2.0 (x¯ = 85.6, SD = 3.24) were compared with those of SMART v.1.0 (x¯ = 81, SD = 22.8). Although the average SUS rating for SMART v.2.0 is higher, it is not significantly different from the average SUS ratings for SMART v.1.0 (p = .63). However, combined standard deviations were compared using an F test, with results suggesting a significant difference between standard deviations (p < 0.001).

From a more hedonic perspective, the majority of participants had largely positive qualitative feedback. Participants often remarked that the amount of text on the screen was appropriate (only one participant said overall there was too much text on each page). Participants also noted that the information given in SMART would have been helpful when they were in the acute stage following their mTBI. There were many instances in which participants remarked that the solutions provided by the pedagogical agents in the learning modules were realistic and relatable to their own experiences. Finally, five participants stated that videos included in the modules helped them better understand the information.

Throughout Task 1 and Task 2, participants also remarked on the appearance of the artwork and graphics within the program. Participants noted that the characters were drawn nicely and the background was not too crowded. A majority of participants indicated that navigating through the program was easy and each activity did not take too much time to complete. When completing the assigned module, all participants remarked that they liked earning achievements because it made them feel accomplished and that they were learning.

Discussion

Overall, findings suggest that the SMART program was usable, acceptable, and relevant to adolescents with mTBI. Findings from the focus groups suggested that the initial versions of the UI and modules were largely acceptable, but also uncovered useful directions for making improvements. Participant feedback suggested that the program content addressed symptoms they had experienced, and participants appreciated the graphics, amount of text and gamified learning environment.

Findings from usability testing uncovered a variety of areas for improvement. For instance, we added an introduction module which provides an overview of the program including its goals and objectives. Given participant feedback, we made changes to shorten the program’s length. Overall, the amount of text in the SMART program was reduced. We reduced the number of words on each page, removed redundant content, and simplified recommendations to make them more understandable. To more evenly balance module length, we moved some content from longer modules to shorter modules. Additionally, hyperlinks open up within the application instead of opening within a new web browser. Finally, because some participants were unable to locate the help screens, we redesigned them to be more noticeable by increasing the size of the text and creating text boxes so that the text was separated from the images. These usability issues were fixed on a rolling basis as they were identified, thereby leading to a more robust and usable intervention over time. SUS and user friendliness ratings suggest an intervention with excellent ease-of-use and user friendliness. Participants’ qualitative remarks and usability ratings were supported by their ability to navigate through the program with little guidance. Participants were able to complete all tasks. Very few misunderstandings and errors were noted. Furthermore, participant misunderstandings did not impact the overall usability of the program.

In terms of improvements over the prior version of SMART, no statistically significant differences were observed between mean SUS scores from SMART v.1.0 and v.2.0. However, a comparison of standard deviations did uncover a statistically significant difference. This suggests that although participants rated both versions of SMART highly regarding usability, participants’ responses had significantly less dispersion and spread around the mean regarding usability of SMART v.2.0 than for SMART v.1.0. In other words, participants seemed to be in closer agreement regarding their perceptions of usability for SMART v.2.0.

The redesign of SMART v.2.0 was a significant departure from the previous version. A complete UI redesign was performed, substantially altering the system’s look, feel, and method of use. All learning modules were completely re-imagined and fundamentally reworked to appeal to mobile-savvy adolescents who are regular consumers of digital multimedia. However, despite the fundamental redesign of the symptom monitoring features and the learning modules, the usability of SMART v.2.0 was largely equivalent to that of the prior version, which was high. However, this finding is tempered by small sample sizes. Further, the large variance in usability ratings from SMART v.1.0 renders comparisons somewhat suspect.

Limitations

Findings of the current study should be interpreted in the context of limitations. Given the multi-methods nature of usability testing, the sample size was small (although well within the guidelines for identifying around 80% of usability problems (Bevan et al., 2003, p. 5)). Additionally, a lack of diversity in the sample size may reduce its generalizability. Usability testing occurred in a lab setting with a facilitator, and it is unknown how the intervention will work for adolescents completing the program independently in their homes.

Conclusions

The SMART program was designed to address impacts of mild Traumatic Brain Injury (mTBI) in adolescents. SMART v.1.0 demonstrated good usability, but initial results suggested a need for modifications to the program. After redesigning the application to improve ease of use and engagement, SMART v.2.0 demonstrated high usability and was relevant to adolescents with mTBI. Given the prevalence of mTBI and low cost of mobile interventions, our results highlight the potential widespread utility of the SMART v.2.0 application.

In sum, adolescents found the new version of the SMART v.2.0 program to be highly usable and pertinent to their experiences. Based on participant feedback, we have made a number of minor modifications to the program over multiple iterations, thus leading to a more robust and usable user experience. The next step will be conducting an efficacy trial of SMART v.2.0 to determine its ability to facilitate recovery in adolescents following mTBI. We hypothesize that adolescents randomized to SMART v.2.0 will engage in more problem-focused coping, and report better psychosocial functioning, self-efficacy and quality of life relative to individuals assigned to usual care.

Table 4.

Characteristics from SQuaRE system and software quality models (International Organization for Standardization (2011) Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, n.d.).

Characteristic Definition
Usability Degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.
Efficiency
Resources expended in relation to the accuracy and completeness with which users achieve goals. Relevant resources can include time to complete the task (human resources), materials, or the financial cost of usage.
Effectiveness
The accuracy and completeness with which users achieve specified goals.
Satisfaction
The degree to which user needs are satisfied when a product or system is used in a specified context of use.
Open in a new tab

Acknowledgments:

We would like to thank Stephanie Boyd, Noura Barzai, Aimee Miley and Jessica Aguilar for their contributions to the study and manuscript.

Funding: This work was funded by the National Institute of Child Health and Human Development [R21HD087844].

Footnotes

Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Al Sayegh A, Sandford D, & Carson AJ (2010). Psychological Approaches to Treatment of Postconcussion Syndrome: A Systematic Review. Journal of Neurology, Neurosurgery & Psychiatry, 81(10), 1128–1134. [DOI] [PubMed] [Google Scholar]
  2. Anderson M, & Jiang J (n.d.). Teens, Social Media & Technology 2018. Retrieved September 27, 2019, from Pew Research Center website: https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/
  3. Andersson G, & Titov N (2014). Advantages and Limitations of Internet-based Interventions for Common Mental Disorders. World Psychiatry, 13(1), 4–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Babcock L, Byczkowski T, Wade SL, Ho M, Mookerjee S, & Bazarian JJ (n.d.). Predicting Postconcussion Syndrome After Mild Traumatic Brain Injury in Children and Adolescents Who Present to the Emergency Department. JAMA Pediatr, 2(156–161). 10.1001/jamapediatrics.2013.434 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Baker JG, Leddy JJ, Darling SR, Shucard J, Makdissi M, & Willer BS (2016). Gender Differences in Recovery From Sports-Related Concussion in Adolescents. Clinical Pediatrics, 55(8), 771–775. [DOI] [PubMed] [Google Scholar]
  6. Bangor A, Kortum P, & Miller J (2009). Determining what individual SUS scores mean: Adding an adjective rating scale (Vol. 4). [Google Scholar]
  7. Bangor Aaron, Kortum PT, & Miller JT (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10/b344sc [Google Scholar]
  8. Bath E, Tolou-Shams M, & Farabee D (2018). Mobile Health (Mhealth): Building the Case For Adapting Emerging Technologies For Justice-Involved Youth. Journal of the American Academy of Child & Adolescent Psychiatry, 57(12), 903–905. https://doi.org/10/gf8kcq [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Bevan N, Barnum C, Cockton G, Nielsen J, Spool J, & Wixon D (2003). The Magic Number 5: Is It Enough for Web Testing? CHI’03 Extended Abstracts on Human Factors in Computing Systems, 698–699. https://doi.org/10/cvtt2w [Google Scholar]
  10. Brooke J (1986). SUS - a Quick And Dirty Usability Scale. Digital Equipment Corporation. [Google Scholar]
  11. Bull S, Devine S, Schmiege SJ, Pickard L, Campbell J, & Shlay JC (2016). Text Messaging, Teen Outreach Program, And Sexual Health Behavior: A Cluster Randomized Trial. American Journal of Public Health, 106(Suppl 1), S117–S124. https://doi.org/10/f9bmn4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Cancelliere C, Kristman VL, Cassidy JD, Hincapié CA, Côté P, Boyle E, … Borg J (2014). Systematic Review of Return to Work After Mild Traumatic Brain Injury: Results of the International Collaboration on Mild Traumatic Brain Injury Prognosis. Archives of Physical Medicine and Rehabilitation, 95(3), S201–S209. https://doi.org/10/f2z69t [DOI] [PubMed] [Google Scholar]
  13. Cassidy JD, Carroll L, Peloso P, Borg J, Von Holst H, Holm L, & Coronado V (2004). Incidence, Risk Factors And Prevention of Mild Traumatic Brain Injury: Results of the WHO Collaborating Centre Task. Force on Mild Traumatic Brain Injury. Journal of Rehabilitation Medicine, 36(0), 28–60. [DOI] [PubMed] [Google Scholar]
  14. Deterding S, Dixon D, Khaled R, & Nacke L (2011). From Game Design Elements to Gamefulness: Defining Gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, 9–15. https://doi.org/10/ctbr [Google Scholar]
  15. Dexheimer JW, Kurowski BG, Anders SH, McClanahan N, Wade SL, & Babcock L (2017). Usability Evaluation of the SMART Application For Youth With Mtbi. International Journal of Medical Informatics, 97, 163–170. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Emery CA, Barlow KM, Brooks BL, Max JE, Villavicencio-Requis A, Gnanakumar V, … Yeates KO (2016). A Systematic Review of Psychiatric, Psychological, and Behavioural Outcomes following Mild Traumatic Brain Injury in Children and Adolescents. The Canadian Journal of Psychiatry, 61(5), 259–269. https://doi.org/10/gf873z [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Eysenbach G (2001). What is e-Health? Journal of Medical Internet Research, 3(2). https://doi.org/10/c6xwj3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Glang AE, Koester MC, Beaver S, Clay J, & McLaughlin K (2010). Online Training In Sports Concussion For Youth Sports Coaches. International Journal of Sports Science & Coaching, 5(1), 1–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Glang AE, Koester MC, Chesnutt JC, Gioia GA, McAvoy K, Marshall S, & Gau JM (2015). The Effectiveness of a Web-Based Resource In Improving Postconcussion Management In High Schools. Journal of Adolescent Health, 56(1), 91–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gravel J, D’Angelo A, Carrière B, Crevier L, Beauchamp MH, Chauny J-M, & Chaillet N (2013). Interventions Provided In the Acute Phase For Mild Traumatic Brain Injury: A Systematic Review. Systematic Reviews, 2(1), 10–1186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. International Organization for Standardization (2011) Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. (n.d.). Retrieved from https://www.iso.org/standard/35733.html
  22. Kontos AP, Elbin RJ, Schatz P, Covassin T, Henry L, Pardini J, & Collins MW (2012). A Revised Factor Structure for the Post-concussion Symptom Scale: Baseline and Postconcussion Factors. The American Journal of Sports Medicine, 40(10), 2375–2384. https://doi.org/10/f398kf [DOI] [PubMed] [Google Scholar]
  23. Kraus JF, Hsu P, Schafer K, & Afifi AA (2014). Sustained Outcomes Following Mild Traumatic Brain Injury: Results of a Five-Emergency Department Longitudinal Study. Brain Injury, 28(10), 1248–1256. 10.3109/02699052.2014.916420 [DOI] [PubMed] [Google Scholar]
  24. Kurowski BG, Wade SL, Dexheimer JW, Dyas J, Zhang N, & Babcock L (2016). Feasibility and Potential Benefits of a Web-based Intervention Delivered Acutely After Mild Traumatic Brain Onjury in Adolescents: A Pilot Study. Journal of Head Trauma Rehabilitation, 31(6), 369–378. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Langlois JA, Rutland-Brown W, & Thomas KE (2004). Traumatic Brain Injury In the United States. Retrieved from http://www.ncdsv.org/images/CDC_TBIintheUSEDVisitsHospitalizationsAndDeaths_2006.pdf
  26. Lenhart A, Duggan M, Perrin A, Stepler R, Rainie H, & Parker K (2015). Teens, Social Media & Technology Overview 2015. Pew Research Center [Internet & American Life Project]. [Google Scholar]
  27. Lumba-Brown A, Yeates KO, Sarmiento K, Breiding MJ, Haegerich TM, Gioia GA, … Giza CC (2018). Centers For Disease Control And Prevention Guideline On the Diagnosis And Management of Mild Traumatic Brain Injury Among Children. JAMA Pediatrics, 172(11), e182853–e182853. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. McCrory P, Johnston K, Meeuwisse W, Aubry M, Cantu R, Dvorak J, … Schamasch P (2005). Summary And Agreement Statement. Of the 2nd International Conference on Concussion in Sport, Prague, 39(4), 196–204. 10.1136/bjsm.2005.018614 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Nezu AM, & D’Zurilla TJ (2006). Problem-Solving Therapy: A Positive Approach to Clinical Intervention. Springer Publishing Company. [Google Scholar]
  30. Nielsen J (1995). Severity Ratings For Usability Problems. Papers and Essays, 54, 1–2. [Google Scholar]
  31. Nowacki R, van Eldik N, Eikens M, Roijen R, Haga N, Schott D, & Wennekes M (2017). Evaluation of a Follow-up Program for Mild Traumatic Brain Injury in Schoolchildren. European Journal of Paediatric Neurology, 21(2), 382–387. [DOI] [PubMed] [Google Scholar]
  32. O’Brien KHM, Wyman Battalen A, Sellers CM, Spirito A, Yen S, Maneta E, … Braciszewski JM (2019). An mHealth Approach to Extend a Brief Intervention for Adolescent Alcohol Use and Suicidal Behavior: Qualitative Analyses of Adolescent and Parent Feedback. Journal of Technology in Human Services. https://doi.org/10/gf8j47 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Oh H, Rizo C, Enkin M, & Jadad A (2005). What is eHealth: A Systematic Review of Published Definitions. Journal of Medical Internet Research, 7(1), e1. https://doi.org/10/cnv7cd [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Organization, W. H. (2010). Telemedicine: Opportunities and developments in member states. Report on the second global survey on eHealth. World Health Organization. [Google Scholar]
  35. Papoutsis J, Stargatt R, & Catroppa C (2014). Long-Term Executive Functioning Outcomes For Complicated And Uncomplicated Mild Traumatic Brain Injury Sustained In Early Childhood. Developmental Neuropsychology, 39(8), 638–645. 10.1080/87565641.2014.979926 [DOI] [PubMed] [Google Scholar]
  36. Ponsford J, Willmott C, Rothwell A, Cameron P, Ayton G, Nelms R, & Ng K (2001). Impact of Early Intervention On Outcome After Mild Traumatic Brain Injury In Children. Pediatrics, 108(6), 1297–1303. [DOI] [PubMed] [Google Scholar]
  37. Røe C, Sveen U, Alvsåker K, & Bautz-Holter E (2009). Post-Concussion Symptoms After Mild Traumatic Brain Injury: Influence of Demographic Factors And Injury Severity In a 1-Year Cohort Study. Disability and Rehabilitation: An International, Multidisciplinary Journal, 31(15), 1235–1243. 10.1080/09638280802532720 [DOI] [PubMed] [Google Scholar]
  38. Sauro J, & Lewis JR (2012). Standardized Usability Questionnaires. In Sauro J & Lewis JR (Eds.), Quantifying the User Experience (Vol. 8, p. ). Boston: Morgan Kaufmann. [Google Scholar]
  39. Scheenen ME, Spikman JM, de Koning ME, van der Horn HJ, Roks G, Hageman G, & van der Naalt J (2017). Patients “At Risk” of Suffering from Persistent Complaints After Mild Traumatic Brain Injury: The Role of Coping, Mood Disorders, And Post-Traumatic Stress. Journal of Neurotrauma, 34(1), 31–37. 10.1089/neu.2015.4381 [DOI] [PubMed] [Google Scholar]
  40. Schwartz LA, Psihogios AM, Henry-Moss D, Daniel LC, Ver Hoeve ES, Velazquez-Martin B, … Szalda D (2019). Iterative Development of a Tailored Mhealth Intervention For Adolescent And Young Adult Survivors of Childhood Cancer. Clinical Practice in Pediatric Psychology, 7(1), 31–43. https://doi.org/10/gf8j42 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Silverberg ND, & Iverson GL (2013). Is Rest After Concussion “The Best Medicine?”: Recommendations For Activity Resumption Following Concussion In Athletes, Civilians, and Military Service Members. The Journal of Head Trauma Rehabilitation, 28(4), 250–259. 10.1097/HTR.0b013e31825ad658 [DOI] [PubMed] [Google Scholar]
  42. Taylor CA, Bell JM, Breiding MJ, & Xu L (2017). Traumatic Brain Injury–Related Emergency Department Visits, Hospitalizations, And Deaths—United States, 2007 And 2013. MMWR. Surveillance Summaries, 66(9), 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Wade SL, Walz NC, Carey J, McMullen KM, Cass J, Mark E, & Yeates KO (2011). Effect on Behavior Problems of Teen Online Problem-Solving for Adolescent Traumatic Brain Injury. PEDIATRICS, 128(4), e947–e953. https://doi.org/10/fmkvn6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Wade Shari L., Walz NC, Carey J, McMullen KM, Cass J, Mark E, & Yeates KO (2012). A Randomized Trial of Teen Online Problem Solving: Efficacy in Improving Caregiver Outcomes After Brain Injury. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 31(6), 767–776. https://doi.org/10/f4fd6b [DOI] [PubMed] [Google Scholar]
  45. Wade Shari L., Wolfe CR, Brown TM, & Pestian JP (2005). Can a Web-based Family Problem-solving Intervention Work for Children with Traumatic Brain Injury? Rehabilitation Psychology, 50(4), 337–345. https://doi.org/10/bm29fp [Google Scholar]
  46. Wade SL, Taylor HG, Yeates KO, Kirkwood M, Zang H, McNally K, & Zhang N (2018). Online Problem Solving For Adolescent Brain Injury: A Randomized Trial of 2 Approaches. Journal of Developmental and Behavioral Pediatrics, 39(2), 154–162. 10.1097/DBP.0000000000000519 [DOI] [PubMed] [Google Scholar]
  47. Woodrome SE, Yeates KO, Taylor HG, Rusin J, Bangert B, Dietrich A, & Wright M (2011). Coping Strategies as a Predictor of Post-concussive Symptoms in Children With Mild Traumatic Brain Injury Versus Mild Orthopedic Injury. Journal of the International Neuropsychological Society, 17(2), 317–326. 10.1017/S1355617710001700 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. World Health Organization. (2011a). mHealth: New Horizons For Health Through Mobile Technologies. Retrieved from http://www.who.int/goe/publications/goe_mhealth_web.pdf
  49. World Health Organization. (2011b). Second Global Survey on eHealth. Retrieved from http://www.who.int/goe/publications/goe_mhealth_web.pdf.
  50. Worthen-Chaudhari L, McGonigal J, Logan K, Bockbrader MA, Yeates KO, & Mysiw WJ (2017). Reducing Concussion Symptoms Among Teenage Youth: Evaluation of a Mobile Health App. Brain Injury, 31(10), 1279–1286. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Ybarra ML, Prescott T, Mustanski B, Parsons J, & Bull SS (2019). Feasibility, Acceptability, and Process Indicators for Guy2Guy, an mHealth HIV Prevention Program for Sexual Minority Adolescent Boys. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 65(3), 417–422. https://doi.org/10/gf8j44 [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Zemek R, Barrowman N, Freedman SB, Gravel J, Gagnon I, McGahern C, … for the Pediatric Emergency Research Canada (PERC) Concussion Team. (2016). Clinical Risk Score for Persistent Postconcussion Symptoms Among Children With Acute Concussion in the ED. JAMA, 315(10), 1014. https://doi.org/10/gctk64 [DOI] [PubMed] [Google Scholar]
  53. Zhang N, Kaizar EE, Narad ME, Kurowski BG, Yeates KO, Taylor HG, & Wade SL (2019). Examination of Injury, Host, and Social-environmental Moderators of Online Family Problem Solving Treatment Efficacy for Pediatric Traumatic Brain Injury Using an Individual Participant Data Meta-analytic Approach. Journal of Neurotrauma, 36(7), 1147–1155. https://doi.org/10/gf873t [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES