Abstract
This case study discusses the design, development, and formative evaluation of Epilepsy Journey, an individually-tailored, web-based intervention designed to address the unique executive functioning (EF) needs of adolescents with epilepsy. This intervention was designed through a three-phase iterative, patient-centered participatory action research process. First, a front-end analysis was completed to identify the unique needs of adolescents with epilepsy and initial design ideas via focus groups. Second, a preliminary design of the intervention was developed from focus group results. Finally, we iteratively incorporated revisions and refinements based on patient-centered feedback collected during usability sessions. Findings based on usage analytics, user reports, and coded qualitative themes from usability sessions suggest that our iterative, patient-centered approach to design, development, and evaluation resulted in a visually appealing interactive and brief intervention that is both engaging and individually tailored to the needs of adolescents with epilepsy.
Keywords: formative evaluation, formative design, cognitive behavioral intervention, executive functioning deficits, epilepsy, adolescents with epilepsy
The remarkable growth of the Internet and its constant availability on smart devices has led to a growing interest in utilizing web-based platforms as learning environments (Godwin-Jones, 2011). As digital technologies have become more prevalent in society, it has become increasingly important that instructional designers implement principles and methods that increase the usability and adoption rates of such learning environments and educational programs. With a strong focus on user needs and an emphasis on the usability and understandability of products, user-centered design has been shown to be an effective approach to increasing the acceptance of technology-driven learning environments (Norman, 1988). A hallmark of user-centered design is the involvement of users throughout the design process (Dumas & Redish, 1993; Eason, 1988; Gould & Lewis, 1985; Shackel, 1991). In this paper, we describe how an instructional design team at a large midwestern university and a midwestern children’s hospital utilized a user-centered design framework to design, develop, and evaluate a web-based intervention for youth with epilepsy, called Epilepsy Journey, with promising results.
Epilepsy Journey is an individually-tailored, web-based intervention for youth with epilepsy who exhibit executive functioning (EF) deficits. It was developed to help these individuals identify, understand, and develop strategies for improving EF. Executive functions are cognitive processes used to form and complete goals, and includes attention control, inhibition, working memory, self-regulation, planning, organizing, and problem solving (Lezak, Howieson, & Loring, 2004). Adolescents with epilepsy may exhibit weaknesses in EF that can impact their academic pursuits, social interactions, and daily routines (Piccinelli et al., 2010). Youth with newly diagnosed epilepsy have been shown to have significant EF deficits compared to healthy controls that worsen over time (Jackson et al., 2013). Despite the acute need for EF interventions, very few executive functioning interventions exist, and none are specifically designed for adolescents with epilepsy (Macallister, Vasserman, Rosenthal, & Sherman, 2014). Epilepsy Journey was developed in response to this need.
Project Description
Epilepsy Journey is an online learning environment that incorporates an overarching journey metaphor as a central design element. When patients log in to Epilepsy Journey, they are presented with a map of ten different “lands,” each representing a distinct instructional unit. For example, the problem-solving unit is represented as “Problem-solving Peak” and the unit on sleep and stress is called the “Land of Nod.” Patients are guided on their journey through the different lands by a cartoon character named Brainy who acts as a virtual coach and guide, providing assistance and encouragement (see Figure 1). As patients make their way through each land, they encounter a variety of multimedia and game-like features, including animations, videos, and mini-games. Patients are able to track their status using a progress bar, and earn digital badges upon completion of each unit.
Figure 1.
Annotated screenshot of the Epilepsy Journey interface showing a game in which patients click on distractions to remove them.
Epilepsy Journey was built using the Wordpress content management system. WordPress is a highly popular PHP- and MySQL-based CMS that is used by more than 28% of the top ten million websites (W3Techs, 2017). Given its popularity, this open-source platform has a strong online development and support community, and is highly extensible due to its plugin architecture. Wordpress plugins allow administrators to enhance and extend the basic functionality of the software. This software provided key affordances that allowed our small design and development team (n=2) to rapidly develop an intervention that was compatible with both desktop and mobile devices and allowed us to include a diverse set of plugins and extensions that greatly expanded the functionality of the website.
The content included in Epilepsy Journey borrowed heavily from Wade and colleagues’ teen-only problem solving (TOPS) intervention (Wade, Carey, & Wolfe, 2006; Wade, Walz, Carey, Williams, Cass, Herren, Mark, & Yeates, 2010). TOPS is a family-oriented, web-based intervention to improve EF and behavior problems for youth following pediatric traumatic brain injury (TBI). Over the past decade, Wade and colleagues have investigated the efficacy of TOPS for addressing EF issues, with findings from clinical trials suggesting that a problem-solving approach is particularly useful for improving adolescent self regulation skills and a web-based delivery format can capitalize on the increasing prevalence of technology use by adolescents. While these findings are promising from the perspective of TBI, research is limited for youth with epilepsy. Research regarding the use and impact of educational interventions for improving cognitive impairments and commonly associated executive functioning deficits for youth with epilepsy is also sparse (Kerr & Blackwell, 2015). The needs of youth with epilepsy differ substantially from those of teenagers with TBI (for an overview, see Wade et al., 2013). Because TOPS was originally developed for teenagers with TBI, substantial modifications were needed. In the following sections, we describe our three-phase process of incorporating information and principles from TOPS into Epilepsy Journey, as well as subsequent formative evaluation and iterative refinements.
Methodology
The purpose of this project was to design, develop, and evaluate the Epilepsy Journey intervention for adolescents with epilepsy and executive functioning deficits at a midwestern epilepsy center. Two overarching questions guided our research. (RQ1) How do participants rate the ease-of-use of Epilepsy Journey? (RQ2) To what extent does Epilepsy Journey meet the design goals of being acceptable, feasible, easy to use, and relevant to the unique needs of adolescents with epilepsy? These questions were approached both quantitatively and qualitatively using data collected during Phases 1 and 3 of a three-phase participatory action research project.
Design Phases
Epilepsy Journey was designed and developed over three phases using a participatory action research approach (PAR; Kindon, Pain, & Kesby, 2007; McIntyre, 2007). Using PAR allowed us to intentionally and meaningfully involve members of the target population and their families to collaboratively create an intervention that met the design goals of being: (1) feasible for the target population to readily and easily navigate the site and to make meaning of the content, (2) specifically targeted at ameliorating EF deficits, (3) individualized, and (4) user-friendly through its alignment with focus group criteria of desired features, and the positioning of content and or delivery that takes into account the participant’s executive functioning deficits. The three-phases of the project included front-end analysis (Phase 1), design and development (Phase 2), and formative, user-centered evaluation (Phase 3). We present this three phase instructional design trajectory as a concurrent and recursive adaptation of the five core elements of ADDIE (Analyze, Design, Develop, Implement, and Evaluate: Branch & Kopcha, 2014), relying heavily on user-centered design and rapid prototyping (Figure 3). We describe each of the phases in the following sections.
Figure 3.
Screenshot of the journey-themed level selection map from Epilepsy Journey.
Phase 1: Front-end analysis
Our task during Phase 1 was to evaluate the existing teen-only problem solving (TOPS) intervention with the goal of developing a new intervention that would better meet the unique needs of adolescents with epilepsy. Patients and their primary caregivers completed background forms to report demographic information as well as medical information including seizure type and symptoms. Licensed psychologists and trainees then conducted a series of focus groups with the goals of identifying EF challenges experienced by adolescents with epilepsy and completing a small group evaluation on the TOPS intervention. The latter generated input regarding revisions and modifications that would inform the design and development of Epilepsy Journey.
Phase 2: Design and development
Phase 2 was informed by the results of the Phase 1 focus groups and centered on developing rapid prototypes of Epilepsy Journey. Information gleaned from focus groups, subject matter expert (SME) interviews, and team design meetings suggested three areas of design considerations. In line with our PAR approach to the design of the intervention, the first tier of design considerations focused on the patient, and built upon these considerations from the perspectives of pedagogy and technology. These design considerations are outlined in Table 1. Some interconnections exist between the three areas of design considerations; however, for clarity they are represented in Table 1 as distinct and non-overlapping.
Table 1.
Patient, pedagogical, and system considerations for design of the Epilepsy Journey.
| Patient Considerations | |||
|---|---|---|---|
| Epilepsy | Executive functioning deficits | Adolescent | Content |
|
|
|
|
| Pedagogical Considerations | |||
|
| |||
| Gamification | Motivation | Scaffolding and support | Universal Design for Learning |
|
|
|
Multiple means of:
|
| System Considerations | |||
|
| |||
| Open-Source | Mobile-ready | FERPA | User Logging |
|
|
|
|
Patient considerations
Necessary changes were identified from the TOPS intervention. Based on patient design suggestions, the following guiding principles were identified, (1) the intervention must be designed for adolescents, (2) content must be related to the specific EF deficits that were identified during focus group evaluations, and those same deficits must be taken into account when creating content for the intervention, (3) the content needs to be epilepsy-specific, (4) the content needs to be delivered in a brief and skimmable format to fit the attention span and EF abilities of participants.
These considerations were realized by utilizing an online and mobile-friendly format that wrapped content within a multimedia-rich journey narrative. Knowing that adolescents with epilepsy exhibit EF deficits, including memory and attention issues, the content was chunked into pages that would take between 30 seconds and two minutes to complete on average, and these pages were incorporated into brief modules that required less than 30 minutes to complete. Progress monitoring features were included, such as a progress bar, to allow users to readily navigate and recall their position within the course.
Pedagogical considerations
Pedagogical decisions were driven by unique needs revealed in Phase 1 focus groups. Since the intervention was designed to provide an individually-tailored path through the modules, it was imperative to include instruction that would promote self-efficacy, motivation, and support features. Gamified course content and characteristics were also included as these features not only magnified the previously-mentioned requirements, but also satisfied the need for youthful and engaging content that was not overly didactic in nature. Realizing the unique issues that EF deficits can have on participants, we also implemented concepts of universal design for learning. Not only was information presented in multiple formats and mediums (e.g. video, interactive content, imagery, and games), but participants were also able to use various outlets of expression and or action throughout the intervention.
System considerations
Using Wordpress, we created Epilepsy Journey as a means to approach the aforementioned pedagogical and patient considerations. Wordpress is a free and open-source content management system that can be installed onto either a private or local server, making it an attractive choice for meeting the regulations of the Family Educational Rights and Privacy Act (FERPA), which governs how sensitive user data must be stored and protected at the acting research institution. Wordpress also features a diverse plugin and template system with a breadth of customizable options allowing us to rapidly test, develop, modify, and implement program features. Included among the set of tools that Wordpress afforded us were digital badging options and a plethora of responsive and mobile templates allowing the encompassing project needs to be realized, such as the ability to be accessed on a wide range of devices and the ability to securely store participant logs and trace data. Perhaps most importantly, Wordpress is open-source, meaning that the design team was free to view and modify the underlying source code to meet program needs.
Phase 3: Formative, user-centered evaluation
Phase 3 consisted of user-centered formative evaluation of Epilepsy Journey. We collected usage information, survey data, usability testing feedback, and other data from participants during the development of the intervention, which we subsequently used to further extend and refine the intervention (Dick & Carey, 1990; Worthen & Sanders, 1987). Formative evaluation took the form of multi-modal usability testing (Hix & Hartson, 1994; Krug, 2010; Nielsen, 1994) which sought to elicit feedback on applicability, content, and the individualized approach of the intervention, as well as ease-of-use, acceptance, error rates, and time to completion of modules. As defined by Dumas and Redish (1993), usability testing is “a systematic way of observing actual users trying out a product and collecting information about the specific ways in which the product is easy or difficult for them” (p. 12). Formative evaluation in general and usability testing specifically have similar goals of collecting data about the effectiveness of a product from the target population during the design and development process. They both employ an iterative process of collecting data throughout numerous sessions, and they often times include data collection strategies that include observation, think-aloud protocols, questionnaires, interviews, surveys, and data logs.
Data Collection and Analysis
Data were obtained through a variety of measures throughout the different phases of the project. At the outset of the project, a background information form provided us with participant demographics including age, gender, history of seizures, and comorbid illnesses. Focus group methods were utilized in Phase 1. Usability testing and heuristic testing were utilized in Phase 3.
Demographics
Following best practice guidelines for formative evaluation, participants included people of differing expertise and perspectives (Saroyan, 1992). Specifically, participants included adolescents with epilepsy (n=11), their primary caregivers (n=5), and one usability expert. Participants with epilepsy were between 13 and 17 years of age with a medical diagnosis of epilepsy. Recruitment occurred during routine medical visits to an epilepsy center at a large midwestern children’s hospital. Participants were provided an overview of study procedures, benefits, and risks. Informed consent and assent forms were obtained, and all research procedures were approved by the hospital’s Institutional Review Board. Participants were recruited for all three different project phases. In Phase 1 (focus groups), five adolescents with epilepsy and their caregivers participated. In Phase 3 (usability testing), six adolescents with epilepsy and one usability expert participated. Adolescent participant demographics are provided in Table 2 below.
Table 2.
Adolescent participant demographic information by study phase.
| Phase 1: Focus Group (n=5) | Phase 3: Usability Testing (n=6) | |
|---|---|---|
| Child Age (years; M(SD)) | 14.3 (1.29) | 15.9 (1.29) |
| Time since diagnosis (years; M(SD)) | 3.6 (2.5) | 2.6 (2.5) |
| Gender (Male) | 40% | 17% |
| Gender (Female) | 60% | 83% |
| Ethnicity: White | 100% | 66.6% |
| Ethnicity: Hispanic | 0% | 16.7% |
| Ethnicity: African American | 0% | 16.7% |
| Caregiver (Mother) | 83% | 83% |
| Caregiver (Father) | 17% | 17% |
Focus groups
In Phase 1, licensed psychologists and psychology trainees conducted focus groups with the intent of identifying relevant challenges with EF and reviewing the existing TOPS intervention for potential adaptations/modifications. Adolescents and their parents were queried about strategies to manage EF deficits. Input regarding logistics, execution and layout of the TOPS intervention were elicited from adolescents and caregivers. Parents and adolescents participated in separate simultaneous focus groups or individual interviews if they were unable to attend the focus group. To analyze these data, three researchers examined thematic content, and emergent themes were defined by consensus.
Usability testing and heuristic evaluation
In Phase 3, an iterative series of one-to-one usability sessions was conducted. Each adolescent participant worked through two randomly-assigned modules. A semi-structured think-aloud usability protocol was used, which enabled later evaluation of participants’ thought processes and decision making. With participants who have epilepsy, however, think-aloud can be problematic due to increased cognitive load, seizures, and EF deficits. Hence, we coupled think-aloud with All-Views Empirical Analysis (Goggins, Schmidt, Guajardo, & Moore, 2011), an approach for conducting usability evaluations for individuals with cognitive impairments. This approach is multimodal, allowing researchers to infer usability and accessibility issues in the absence of think-aloud narrative. The Morae usability software suite was selected as it offers a feature rich package for usability testing. For example, Morae allows for recording a participant’s interactions while several people remotely observe and take notes in real-time. Morae also captures user data such as audio and video recordings, screen recordings, eye-tracking, mouse movements, and keystrokes. In addition to Morae, trace data were collected using Google Analytics, such as time spent per page, number of pages viewed, and participants’ navigational path through the site.
As suggested in Tessmer’s seminal work on formative evaluation in instructional design (1993), an expert reviewer also performed a heuristic evaluation during Phase 3. A heuristic evaluation involves having a product judged by a usability specialist to determine whether design elements follow best practices of usability principles (Nielsen, 1994). The same protocol and data collection measures were used for the heuristic evaluation as for the one-on-one usability sessions. Participant sessions were coded and feedback was addressed between usability sessions. Two researchers examined thematic content and usability issues, with emergent themes and severity of usability issues being arrived at by consensus.
Finally, upon completion of each usability session, adolescents completed a 19-item online satisfaction survey to help gauge the helpfulness of the intervention and participant expectations, as well as to identify any potential modifications to the intervention. Eight open-ended questions were also completed by usability participants to assess their feelings regarding their experiences, expectations, ease of use, comfort, and opinions of the intervention.
Results
Results of Phase 1: Front-end Analysis
Results from front-end analysis provided guidelines for the design and development of the Epilepsy Journey. These guidelines fell into four categories: (1) methods for addressing EF deficits, (2) presentation of content, (3) intervention functionality and features, and (4) thematic areas of concern for individuals with epilepsy.
Methods identified as potentially beneficial for addressing EF deficits across domains included visual cues, the organization of school materials, pill boxes, and the use of mnemonics.
Feedback regarding presentation of content focused on length, functionality, instructional style, and overarching themes of the original TOPS website.
Related to intervention functionality and features, Focus group results indicated a need for more interactivity, a higher relevance to the targeted age group and population, a less professional aesthetic, and a desire for gamification or game-like elements. More specifically, adolescent participants in focus groups requested more color, mobile compatibility, more graphics, and the inclusion of videos. Primary caregivers also noted a desire for embedded rewards and customization of the site to address the unique needs of their children. Both adolescents and their caregivers stressed the need for brevity (i.e., less than 30 minutes per module).
Ten thematic areas of concern were identified for youth with epilepsy, including memory, attention, organization, etc. A complete list of these themes is presented in Table 3.
Table 3.
Focus group themes and specific examples.
| Theme | Adolescent | Parent |
|---|---|---|
| Memory | Hard to remember things, forgetting assignments, “brain fog” | Hard to remember homework, conversations, medicine, family activities |
| Attention | Lack of attention and focus, mind jumping from one thing to another | Distracted often, difficulty focusing and getting back on-task |
| Organization | Losing important things, time management, prioritizing tasks | Disorganized, difficulty turning in assignments and remembering homework |
| Monitoring | Lack of personal monitoring, unrealistic expectations of how long it takes to complete tasks, finishing more complex tasks is difficult | Review of work assignments is lacking, unknown whether medicine is taken |
| Initiation | Start projects at the last minute, procrastination, lack of motivation | Getting started on tasks is difficult, procrastination, difficulty starting day |
| Impulsivity | Rushing through assignments, regretting impulsive actions, speaking without thinking | Interrupting, lacking awareness of social cues for engagement with others |
| Emotion | “Emotional overload,” fighting with siblings, low frustration tolerance | Overly sensitive to criticism, mood dysregulation, worries |
| Sleep | Lack of sleep is a seizure trigger, need for consistent sleep | Lack of sleep is a seizure trigger |
| Lack of awareness in school | Teachers don’t understand how epilepsy affects memory-”blame you” | Teachers and schools unsure if problems are epilepsy-related or laziness/ADHD |
| Stress | Not specifically addressed | Easily upset and stressed by school demands |
Results of Phase 2: Design and Development
Data obtained from focus group evaluations were used to guide the design of Epilepsy Journey. In alignment with the themes identified in Phase 1, ten instructional modules were created covering various issues that could impact adolescents with epilepsy/EF deficits. Retaining the journey theme, the modules were conceived of as different lands (see Figure 3), with each land aligning with an instructional module: (1) Peninsula of Positive Thoughts (positive framing), (2) Problem-Solving Peak (problems-solving/shifting skills), (3) Mists of Memory (working memory), (4) Island of Initiation (initiation), (5) Mesa of Monitoring (monitoring), (6) Orchards of Organization (planning, organization, organization of materials), (7) Falls of Inhibition (inhibition), (8) State of Emotional Control (emotion regulation), (9) Land of Nod (sleep hygiene and stress management), and (10) Wrap-Up (review). An introductory module was also developed to provide an orientation to the website.
Modules were designed to be completed independently and asynchronously by participants within 30 minutes with follow-up, synchronous interactions with therapists via Skype. A cartoon character named Brainy was also designed and integrated throughout the website to provide tips, coaching, and feedback, and to act as a scaffolding tool (Figure 1). Progress monitoring features were developed to allow participants to navigate easily between modules and to gauge time to completion (Figure 1). Digital badges and achievements were created using the BadgeOS Wordpress plugin as a way not only to assist with self direction through the site, but also to align with the need for more interactive elements.
Results of Phase 3: Formative, User-centered Evaluation
Results from usability sessions fell into two overarching categories: (1) usability issues and (2) features of the intervention that promoted usability. Emergent usability issues included problems with navigation, periodic confusion with website content, and desire for more interactivity. Site features that promoted usability included use of progress monitoring features for easy navigation, the brevity of modules, the journey theme, the coach (Brainy), and readily skimmable content. Subsequent modifications were made to make these traits more prominent. Examples of usability issues included navigation problems (“just make it easy to get into”), lengthy modules (“time consuming”), and dense content (“too much reading”). Participants enjoyed the journey theme (“like it is going to take you on a journey!”) and the coach/mentor (“really liked Brainy”), and felt the website was relevant to them (“I will use information from here in the future”). Some participants expressed the need for more engagement (“I kind of think that making things seem more like social media that kids use everyday, you know like, I don’t know if you can do that… You know how instagram is? You know that kind of layout or background.”), although others found the content to be sufficiently engaging (“I felt engaged the whole time… I liked it”). Participants also had suggestions, such as making the website less “childish” and adding information on what to avoid for people with epilepsy.
Results from the satisfaction survey were in agreement with usability findings. Participants, on the whole, rated the intervention as easy to use and thought the site was helpful. Content length was found to be appropriate, a characteristic that was deemed to be important during group evaluations. According to participants, the most helpful traits of the site included its organization, the addition of “Brainy,” the interactive content, the journey theme, and the relevance of the content to the needs of the adolescents with epilepsy.
Google Analytics data indicated that participants averaged 56.17 page views per session, or an average of 28 pages per module. The average time spent on each module was 24.87 minutes, approximately five minutes below the suggested 30 minute threshold from the focus group evaluations, suggesting that the content was sufficiently tailored for individuals with EF deficits.
Discussion
The formative design of Epilepsy Journey was guided by patient, pedagogical, and system considerations that were unveiled through an intentional, user-centric design process. Alignment of evaluation and design methodology allowed us to identify the unique needs of adolescents with epilepsy/EF deficits and to subsequently devise an individually tailored approach to learning that specifically targeted these needs. Borrowing from PAR, our formative design was inexorably tied to the needs of participants, serving to address a gap in theory related to executive functioning for youth with epilepsy while at the same approaching the practical problem of increasing access to such interventions for these youth (Kindin, Pain, & Kesby, 2007).
The design of Epilepsy Journey serves as an example of how instructional designers can create interventions that are highly usable and show evidence of potential promise for populations with similar challenges. This paper illustrates how we were able to uncover initial design considerations and make principled decisions about how to approach them through a user-centered formative design process. We were able to adjust our approach so as to better meet the needs of our target population based on strengths and weaknesses unveiled during design and evaluation phases. For example, focus groups confirmed the presence of EF deficits for adolescents with epilepsy such as memory and inhibition. Caregivers and their children were able to identify several useful strategies to manage EF deficits which were subsequently built into Epilepsy Journey, such as the use of problem-solving for improving adolescent self regulation skills. Focus groups suggested that a web-based delivery format would capitalize on the increasing prevalence of technology use by adolescents.
Formative evaluation results suggest that Epilepsy Journey is an intervention that is both acceptable and feasible for the target population. The majority of participants stated that they enjoyed the intervention and found the program easy and satisfactory to use. Findings also suggest the site was engaging, it included content that was both helpful and relevant, and that the modules were appropriate in length. We assert that these promising findings were facilitated through inclusion of target users in our user-centered design and evaluation processes and prioritization of their needs.
Limitations
Due to the nature of our formative design approach, findings from this study cannot be considered to be generalizable to the entire population of adolescents with epilepsy/EF deficits. However, the purpose of this study was not to create a generalizable knowledge, but rather to refine an intervention that could be found acceptable and feasible for the target population. In this light, the current paper offers new insights, identifies key design decisions that address the needs of adolescents with epilepsy/EF deficits, and provides preliminary support for an individually-tailored intervention for these youths.
Future Research
While a pre-post single armed trial is currently underway to investigate empirically the efficacy of Epilepsy Journey (with encouraging initial results), the focus of the current research was on ease of use and acceptance an important first step. However, given current trends indicating higher usage of mobile devices among youth than traditional computing platforms (e.g., laptop, desktop), future design and evaluation should focus on ease-of-use and acceptance of a mobile-only version of Epilepsy Journey. Indeed, participants expressed a desire for more responsive and mobile-friendly site designs. In addition, future design research might explore integrating social features into the platforms, as some participants indicated a desire for social media-like designs and affordances.
Conclusion
Individually-tailored web-based interventions are a promising approach for meeting the needs of adolescents with epilepsy exhibiting executive functioning deficits. The goal of the Epilepsy Journey project was to create an environment and curriculum that would address identified needs of participants and be easy to use. This paper provides instructional designers insight into how a user-centric formative design approach can be used for the development of individually-tailored web-based interventions for sensitive populations.
Figure 2.
Three-phase instructional design process for Epilepsy Journey.
Table 4.
Usability testing feedback on Epilepsy Journey modules/lands.
| Domain | Examples from usability testing | Examples of changes made to the intervention website |
|---|---|---|
| Usability Problems | ||
| Navigation problems |
|
|
| Confusion |
|
|
| Need for more engagement/interaction |
|
|
| Features that Promoted Usability | ||
| Progress bar |
|
|
| Length |
|
|
| Journey theme |
|
|
| Positive feelings towards “Brainy” character |
|
|
| Skimming content |
|
|
| Relevant to teens with epilepsy/self |
|
|
Footnotes
Conflict of Interests
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Contributor Information
Noah J. Glaser, University of Cincinnati
Matthew Schmidt, University of Cincinnati.
Shari L. Wade, Division of Rehabilitation Medicine, Cincinnati Children’s Hospital Medical Center
Aimee Smith, Division of Behavioral Medicine and Clinical Psychology, Cincinnati Children’s Hospital Medical Center.
Luke Turnier, Division of Behavioral Medicine and Clinical Psychology, Cincinnati Children’s Hospital Medical Center.
Avani C. Modi, Division of Behavioral Medicine and Clinical Psychology, Cincinnati Children’s Hospital Medical Center
References
- Baek E-O, Cagiltay K, Boling E, Frick T. User-centered design and development. Handbook of Research on Educational Communications and Technology. 2008;(1):660–668. [Google Scholar]
- Branch RM, Kopcha TJ. Instructional Design Models. In: Spector JM, Merrill MD, Elen J, Bishop MJ, editors. Handbook of Research on Educational Communications and Technology. New York, NY: Springer; 2014. pp. 77–87. [Google Scholar]
- Dick W, Carey L. The systematic design of instruction. 3. New York, NY: Harper-Collins; 1990. [Google Scholar]
- Dumas JS, Redish JC. A practical guide to usability testing. Norwood, NJ: Ablex Publishing Corporation; 1993. [Google Scholar]
- Eason K. Information technology and organisational change. London: Taylor & Francis; 1988. [Google Scholar]
- Godwin-Jones R. Emerging technologies: Mobile apps for language learning. Language Learning & Technology. 2011;15(2):2–11. [Google Scholar]
- Goggins S, Schmidt M, Guajardo J, Moore J. 3D virtual worlds: Assessing the experience and informing design. International Journal of Social and Organizational Dynamics in Information Technology. 2011:1. [Google Scholar]
- Gould JD, Lewis C. Designing for usability: Key principles and what designers think. In: Baecker R, Grudin J, Buxton W, Greenberg S, editors. Readings in human-computer interaction, toward the year 2000. New York, NY: Morgan-Kaufman; 1985. pp. 528–547. [Google Scholar]
- Hix D, Hartson HR. Developing user interfaces: Ensuring usability through product and process. New York, NY: John Wiley & Sons, Inc; 1994. [Google Scholar]
- Jackson DC, Dabbs K, Walker NM, Jones JE, Hsu DA, Stafstrom CE, … Hermann BP. The neuropsychological and academic substrate of new/recent-onset epilepsies. The Journal of Pediatrics. 2013;162(5):1047–1053e1. doi: 10.1016/j.jpeds.2012.10.046. https://doi.org/10.1016/j.jpeds.2012.10.04. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kerr EN, Blackwell MC. Near-transfer effects following working memory intervention (Cogmed) in children with symptomatic epilepsy: An open randomized clinical trial. Epilepsia. 2015;56(11):1784–1792. doi: 10.1111/epi.13195. https://doi.org/10.1111/epi.13195. [DOI] [PubMed] [Google Scholar]
- Kindon S, Pain R, Kesby M. Participatory action research: Origins, approaches and methods. In: Kindon S, Pain R, Kesby M, editors. Participatory action research approaches and methods: Connecting people, participation and place. London, UK: Routledge; 2007. pp. 9–18. [Google Scholar]
- Krug S. Rocket surgery made easy: The do-it yourself guide to finding and fixing usability problems. Canada: New Riders; 2010. [Google Scholar]
- Lezak MD. Neuropsychological assessment. Oxford, UK: Oxford University Press; 2004. [Google Scholar]
- McIntyre A. Participatory action research. New York, NY: Sage; 2007. [Google Scholar]
- MacAllister WS, Vasserman M, Rosenthal J, Sherman E. Attention and executive functions in children with epilepsy: What, why, and what to do. Applied neuropsychology: Child. 2014;3(3):215–225. doi: 10.1080/21622965.2013.839605. [DOI] [PubMed] [Google Scholar]
- Nielsen J. Usability engineering. Boston, MA: AP Professional; 1994. [Google Scholar]
- Nielsen J. Conference companion on Human factors in computing systems. ACM; 1994. Usability inspection methods; pp. 413–414. Retrieved from http://dl.acm.org/citation.cfm?id=260531. [Google Scholar]
- Northrup PT. Concurrent formative evaluation: Guidelines and implications for multimedia designers. Educational Technology. 1995;35(6):24–31. [Google Scholar]
- Piccinelli P, Beghi E, Borgatti R, Ferri M, Giordano L, Romeo A, … Balottin U. Neuropsychological and behavioural aspects in children and adolescents with idiopathic epilepsy at diagnosis and after 12 months of treatment. Seizure. 2010;19(9):540–546. doi: 10.1016/j.seizure.2010.07.014. https://doi.org/10.1016/j.seizure.2010.07.014. [DOI] [PubMed] [Google Scholar]
- Saroyan A. Differences in expert practice: A case from formative evaluation. Instructional Science. 1992;21(6):451–472. [Google Scholar]
- Shackel B. Usability—context, framework, definition, design and evaluation. In: Shackel B, Richardson S, editors. Human factors for informatics usability. Cambridge: Cambridge University Press; 1991. pp. 21–37. [Google Scholar]
- Tessmer M. Formative evaluation alternatives. Performance Improvement Quarterly. 1994;7(1):3–18. [Google Scholar]
- Usage of content management systems for websites. 2017 Retrieved from https://w3techs.com/technologies/overview/content_management/all/
- Wade SL, Carey J, Wolfe CR. An online family intervention to reduce parental distress following pediatric brain injury. Journal of consulting and clinical psychology. 2006;74(3):445. doi: 10.1037/0022-006X.74.3.445. [DOI] [PubMed] [Google Scholar]
- Wade SL, Stancin T, Kirkwood M, Brown TM, McMullen KM, Taylor HG. Counselor-Assisted Problem Solving (CAPS) Improves Behavioral Outcomes in Older Adolescents With Complicated Mild to Severe TBI. Journal of Head Trauma Rehabilitation. 2014;29(3):198–207. doi: 10.1097/HTR.0b013e31828f9fe8. https://doi.org/10.1097/HTR.0b013e31828f9fe8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wade SL, Walz NC, Carey J, Williams KM, Cass J, Herren L, Yeates KO. A randomized trial of teen online problem solving for improving executive function deficits following pediatric traumatic brain injury. The Journal of head trauma rehabilitation. 2010;25(6):409–415. doi: 10.1097/HTR.0b013e3181fb900d. [DOI] [PubMed] [Google Scholar]
- Wade SL, Walz NC, Carey JC, Williams KM. Preliminary efficacy of a web-based family problem-solving treatment program for adolescents with traumatic brain injury. The Journal of head trauma rehabilitation. 2008;23(6):369–377. doi: 10.1097/01.HTR.0000341432.67251.48. [DOI] [PubMed] [Google Scholar]
- Worthen BR, Sanders JR. Educational evaluation: Alternative approaches and practical guidelines. London, England: Longman Pub Group; 1987. [Google Scholar]