Abstract
Objectives This study evaluates the usability and feasibility of a Web-based intervention (Web-MAP) to deliver cognitive behavioral therapy (CBT) to adolescents with chronic pain and their parents. Methods The Web site was evaluated in two stages. In stage one, recovered adolescents and parents (n = 5 dyads), who had completed office-based CBT through a pediatric pain management clinic, completed ratings of Web site content, usability, appearance, and theme. In stage two, treatment-seeking adolescents and their parents (n = 6 dyads) completed the full-length Web program. Program usage data were obtained to assess interaction with the Web site. Results Participants rated moderate to strong acceptability of the program. Usage data indicated that participants interacted with the site and used communication features. Conclusions Feedback from usability testing provided important information in the process of designing a feasible Web-based treatment for adolescents with chronic pain for use in a randomized controlled trial.
Keywords: adolescents, chronic and recurrent pain, computer applications/ehealth, family therapy
There is strong empirical support for the use of cognitive behavioral therapy (CBT) in the treatment of pediatric chronic pain (Eccleston, Morley, Williams, Yorke, & Mastroyannopoulou, 2002). Despite support for CBT interventions, many youth do not have access to outpatient psychological services for chronic pain due to a number of barriers, including lack of trained clinicians, expense, competing school and work schedules, and travel distances. To date, the largest treatment advances in making CBT more accessible have been minimal therapist contact and self-administered treatment delivery strategies. These treatments use written materials as a supplement to or substitute for outpatient appointments with a therapist. Self-administered treatments for headache management have produced results that are equal or better than the equivalent therapist-led treatments (Larsson, Daleflod, Hakansson, & Melin, 1987; Richardson & McGrath, 1989).
Recently, two randomized controlled trials have tested computer-based technologies to deliver minimal therapist contact CBT to children with headache and abdominal pain. Headstrong, a CD-ROM intervention for 7–12-year-old children with recurrent headache successfully reduced headache frequency, severity, and duration (Connelly, Rapoff, Thompson, & Connelly, 2006). In the second study, an online treatment program was developed for 9–16 year olds with headache and/or recurrent abdominal pain, demonstrating clinically significant pain reduction (Hicks, von Baeyer, & McGrath, 2006). This 7-week program included online content, written materials for children and parents, and therapist telephone and email contact. These studies provide support for the feasibility of delivering CBT for chronic pain via computer-based technology.
While the existing computer-based interventions for pediatric recurrent pain are quite promising, they do not utilize the interactivity, multimedia formats, personalization, and communication possible with the Internet. These features may increase engagement and positively impact treatment outcomes (Wade, 2004). Initial research on Internet interventions for delivering behavioral treatments has demonstrated feasibility for a variety of pediatric conditions (Ritterband et al., 2003; Wade, 2004). Web-based interventions also provide researchers with unobtrusive measurement of Web site usage and access patterns, which can be used to assess compliance and engagement (Danaher, Boles, Akers, Gordon, & Severson, 2006). Evaluating the feasibility and usability of Web-based interventions early in Web site development is important, as these factors may impact frequency of use, understanding, and the likelihood that a user will implement recommendations (Ritterband et al., 2003). Evaluation of usability can also help determine the appropriateness of Web site interface and content for a target population (Gustafson & Wyatt, 2004). Lack of compliance with accessing Web-based information is a significant barrier to this type of intervention, and improvement of Web content and usability has been suggested as one means of improving compliance (Ritterband et al., 2005).
The purpose of this report is to describe usability testing of an online family CBT program for adolescents with chronic pain (Web-based Management of Adolescent Pain; Web-MAP). This formative evaluation model may be applied in the development of other online treatments. We hypothesized that acceptance of content and usability of Web-MAP would be high, and that participants would appropriately interact and engage with the Web site.
Method
This study was approved by the Institutional Review Board at the academic medical center where the study was conducted. Written informed consent was obtained from parents and written assent was obtained from adolescents for participation in this study.
Program Development and Evaluation Model
We utilized a three-phase model based on existing program evaluation models (Rand, 2004) and previous research on Web-based interventions. In Phase 1, program development, program content is developed based on existing evidence-based treatments, and is adapted to fit within the interface and functional features of a Web site. The adaptation must be appropriate to the intended audience. We utilized independent expert feedback to evaluate the program adaptation in Phase 1. Phase 2, formative evaluation, may include monitoring of program usage, as well as evaluation of the acceptability of program content, usability, and other features by users. This phase is designed to test the adaptation completed in Phase 1 and identify problems that may need to be addressed. Phase 3, outcome evaluation, measures the impact that the fully developed Web-based treatment program has on predetermined outcome measures. This phase may also include evaluation of user's satisfaction with the program, as well as usage patterns, which may be investigated as predictors of outcomes (Gustafson & Wyatt, 2004). We focus this article on Phase 2, formative evaluation of the Web-MAP program.
Web-MAP Program
Program content was developed from existing evidence-based, outpatient CBT treatments for chronic pain (Kashikar-Zuck, 2006; Kashikar-Zuck, Swain, Jones, & Graham, 2005; Robins, Smith, Glutting, & Bishop, 2005), including child pain management skills and parent interventions (e.g., reinforce positive coping and activity participation). The adolescent program includes topics considered core components of CBT for chronic pain, including education, relaxation training, and other cognitive and behavioral strategies. The parent program includes training on reinforcement of positive coping, reward systems for activity participation, modeling positive coping skills, and communicating with teenagers (see additional description in Table I). The Web-based program aims to reduce pain and associated functional limitations. Content was reviewed by an independent expert in CBT for adolescents with chronic pain, and by pediatric pain experts from the fields of nursing, psychology, pediatrics, and anesthesiology.
Table I.
Recovered Adolescent and Parent (n = 5 dyads) Ratings of Web-MAP Modules
| How useful or relevant? 0–5 (not at all–very) M (SD) | How easy to understand? 0–5 (very diff.–very easy) M (SD) | |
|---|---|---|
| Teen modules | ||
| 1. Introduction: education about pain signals, how cognitive and behavioral strategies can impact pain signals, goal setting | 4.20 (0.45) | 4.60 (0.55) |
| 2. Stress and negative emotions: education about the role of stress and tension in pain, identification of sources of stress | 4.40 (0.55) | 4.00 (0.71) |
| 3. Relaxation skills: training in deep abdominal breathing and progressive muscle relaxation, audio exercises | 5.00 (0.00) | 4.00 (0.71) |
| 4. Distraction skills: attention focusing, imagery, mental games, and pleasant activity participation | 4.50 (0.50) | 4.00 (0.71) |
| 5. Cognitive skills: identify negative and catastrophizing thoughts, mindful observation, thought stopping, thought replacement | 3.80 (0.84) | 4.60 (0.55) |
| 6. Sleep and lifestyle: healthy eating and drinking, importance of uptime and physical activity, role of sleep, sleep calculator | 4.60 (0.55) | 4.60 (0.89) |
| 7. Activity participation and pacing: tips for gradual return to desired levels of social, school, and physical activity | 3.80 (1.09) | 4.20 (0.84) |
| 8. Relapse prevention: tips for maintaining gains and facing problems in the future, review of strategies that have been helpful | 4.30 (0.67) | 4.60 (0.55) |
| Overall teen Web site | 4.40 (0.89) | 4.40 (0.55) |
| Parent modules | ||
| 1. Introduction: education about pain signals, how cognitive and behavioral strategies can impact pain signals, goal setting | 4.20 (0.45) | 5.00 (0.00) |
| 2. Stress and negative ermotions: education about the role of stress and tension in pain, identification of sources of stress | 4.00 (0.71) | 4.80 (0.45) |
| 3. Behavior I: reinforcement principles, identification of coping behaviors to praise, guidelines to minimize secondary gain | 4.80 (0.45) | 4.80 (0.45) |
| 4. Behavior II: creation of individually tailored reward or privilege program to increase participation in targeted activities | 3.20 (1.64) | 3.40 (1.34) |
| 5. Modeling: explanation of learning by example, ways to show and explain adaptive coping strategies for pain and stress | 4.40 (0.89) | 4.60 (0.55) |
| 6. Sleep and lifestyle: healthy eating and drinking, importance of uptime and physical activity, role of sleep and sleep calculator | 4.40 (0.89) | 4.80 (0.45) |
| 7. Communicating with teens: tips for gradually increasing teen independence, communicating expectations, listening | 3.40 (1.82) | 4.40 (0.55) |
| 8. Relapse prevention: tips for maintaining gains and facing problems in the future, review of strategies that have been helpful | 4.40 (0.89) | 4.60 (0.55) |
| Overall parent Web site | 4.60 (0.55) | 4.60 (0.55) |
The online program is designed to be used by teens and their parents and includes multimedia elements to enhance information. Peer modeling is included through video interviews of teens who have experienced chronic pain and their parents. Audio files of relaxation instructions are available for practice. Web-MAP is designed to be interactive and personalized. Content includes over 200 page views with graphics. Each user creates a login, chooses treatment goals, and interacts with fields in the Web site. The program is travel-themed, and users visit eight destinations (e.g., China, Costa Rica; one per week) each designed to take about 30 min to complete. Users are instructed to spend additional time practicing skills, and are required to complete assignments at the end of six modules. At some destinations, users receive postcards from previous places they have visited reminding them to practice skills. A message center allows users to ask questions as they progress through the program. A study therapist reviews assignments and messages from participants and provides individualized feedback through the Web-MAP message center. Prospective diary measures of pain and activity limitations are incorporated into the Web site to assess pain and activity limitations before and after program completion.
Participants
Web site usability ratings were obtained during two stages of formative evaluation from two groups of patients. In stage one, recovered patients rated program content in each module and basic Web site function using an HTML version of the Web system. Module content was improved based on participant feedback from stage one. In stage two, treatment-seeking patients were given access to the fully functional Web system to rate overall program content and to obtain information on their interaction with the Web site.
Recovered Adolescents
A purposeful sample of adolescent patients with a history of chronic pain and their parents (n = 5 dyads) was recruited from a multidisciplinary pediatric chronic pain clinic. Adolescents had previously received 4–14 (M = 7.40, SD = 4.16) sessions of office-based CBT over the course of 3–8 months (M = 5.4, SD = 1.95), and were recovered such that they were no longer experiencing significant pain or disability. Adolescents were 13–17 years of age (M = 15.80, SD = 1.23), 60% (n = 3) were female, and had a history of headache (n = 2), abdominal (n = 1), or musculoskeletal (n = 2) pain. Parents were four mothers and one father.
Treatment-Seeking Adolescents
Patients who were currently experiencing chronic pain and associated functional disability and were seeking treatment were recruited from a multidisciplinary pediatric chronic pain clinic. Six consecutive adolescents and their parents (n = 6 dyads) completed the full 8-week Web-MAP program. Adolescents were 12–16 years of age (M = 14.58, SD = 1.59), four were female, and had chronic headache (n = 3), abdominal (n = 2), or musculoskeletal (n = 1) pain. Parents were four mothers and two fathers.
Procedures
User Ratings of Web-MAP Content and Usability
In stage one, six recovered adolescent–parent dyads were contacted by telephone and five dyads (83%) agreed to participate in the study. Participants were given access to a preliminary HTML version of Web-MAP on their home computers for 2 weeks. This version of the Web-MAP program included all text, basic navigation features, and some images, but no other functional features such as interactive fields, assignment submission, or the message system. A survey designed to measure perceptions of the content and usability of Web-MAP was mailed to participants for completion while viewing the eight treatment modules. Recovered participants rated the overall program, as well as the content of each individual module. Module content was improved based on recovered participant feedback prior to stage two.
In stage two, consecutive adolescents seeking treatment for chronic pain and their parents were recruited during pediatric pain management clinic visits. Participants were instructed to complete the fully functional Web-MAP program over 8 weeks. After program completion, surveys designed to measure perceptions of overall Web site content and usability were mailed to participants. Treatment-seeking participants rated the Web site content and features overall, but did not rate the content of each module of the program.
Measures
Web-MAP Content and Usability Questionnaire
This self-report measure was designed to assess a user's perception of the Web-MAP content and Web site usability. Users rated content (“How useful or relevant was the Web site/module content?”) and ease of Web site use (“How easy or difficult was it to use and navigate the Web site/module?”) on Likert scale items (Table I). Likeability of the Web site's appearance and theme were assessed using Likert scale items ranging from 0 (Did not like at all) to 5 (Liked very much). Two open-ended items asked users to “List 2–3 things that you liked about this module/the Web site” and “List 2–3 things that you did not like about this module/the Web site,” and open-ended items asked users “What would you suggest we remove from/add to the Web site to make it more fun or useful?” and solicited other comments or suggestions for improvement. Recovered participants rated the overall Web site and each module, while treatment-seeking participants rated only the overall Web site.
Web-MAP Program Usage Patterns
In stage two, information about program usage was obtained for treatment-seeking participants who accessed the program. Program completion time, number of electronic messages exchanged with study staff via the study Web site, number of fill-in-the-blank fields completed while progressing through modules, number of completed assignments, and word count of completed assignments was collected from Web-MAP administrative databases.
Results
Stage One: Individual Module and Overall Web site Ratings by Recovered Patients (n = 5 dyads)
Ratings of perceived usefulness and ease of use were moderate to high, with some variation across modules (Range = 3.20–4.80; Table I). Ratings for the Web-MAP appearance were high, with a mean of 4.60 (SD = 0.52) on a 0–5 scale, as were ratings for the travel theme (M = 4.65; SD = 0.67). In responses to open-ended questions, parents reported that site content was helpful, particularly content about pain and stress, instructions for reinforcing positive coping, recommendations regarding modeling, and information about lifestyle factors. Adolescents reported that content was helpful, including information about the pain pathway, content about stress and worry, relaxation instructions, and instructions for using distraction strategies. Users liked a number of images, including a diagram of the pain signal pathway. Users also provided suggestions for improvement, including removal of particular images, and areas of content that were difficult to understand, lengthy, or repetitive. These comments were addressed and full Web site functionality was implemented before the site was used by treatment-seeking patients in stage two.
Stage Two: Overall Web site Ratings and Usage Patterns of Treatment-Seeking Patients (n = 6 dyads)
Ratings of perceived usefulness and ease of use of the overall program were moderate to high among treatment-seeking patients and their parents. Mean ratings for overall Web-MAP appearance were moderate (M = 3.67, SD = 1.30). Ratings for ease of use were high (M = 4.50, SD = 0.64). Usefulness of Web-MAP content was rated as moderate (M = 3.58, SD = 1.62), as was the travel theme (M = 3.42, SD = 1.56). While treatment-seeking patients’ ratings were slightly lower than those of recovered patients, no differences were statistically significant. Treatment-seeking patients liked that the program was easy to navigate, video and audio features, personalization, and content, particularly parent behavioral recommendations. Users disliked some images, long Web site load times, and portions of the content that were repetitive.
Program Usage: Interactivity and Engagement
Treatment-seeking participants took 7–15 weeks to complete the program, with a mean completion time of 10.26 weeks (SD = 2.33). Teen users (n = 6) completed on average 30.17 (SD = 4.40) of 35 interactive fields while progressing through the modules. All teen users entered at least three personal treatment goals, such as “To not allow pain to bring my mood down,” “To swim for longer periods,” and “To be able to see friends more.” All teens completed the six assignments. Total word count of assignments ranged from 121–280 (M = 187.33, SD = 53.14). Teens used the message system to communicate with staff (range = 1–7 messages; M = 3.00; SD = 2.10).
Parent users (n = 6) completed on average 26.17 (SD = 5.88) of 44 interactive fields while progressing through the modules. Five of the six parent users entered three or more personal treatment goals. Examples of parent treatment goals include “To be more active,” “To participate more in activities she loves,” and “To have a low level of pain that he can manage.” All parents completed the seven assignments. Total word count of assignments ranged from 204–879 (M = 497.50, SD = 241.93). All parents utilized the Web-MAP message system to communicate with study staff (range = 1–11 messages; M = 4.00; SD = 3.90).
Discussion
This study was designed to assess the usefulness and ease of comprehension of program content, as well as the usability, appearance, and theme of the Web-MAP program before testing the intervention in a randomized trial. It also provides a model for formative evaluation of other online interventions. User responses were positive, with users providing moderate to high ratings for the content, usability, appearance, and theme. Program usage patterns showed that users completed the program as instructed, utilized communication features, and were engaged through goal setting and completing personalized information.
Recovered patients rated the Web-MAP program more positively than treatment-seeking patients, although these differences did not reach significance likely due to the small sample size. Because families had previously participated in CBT, ratings of content may have been confounded by previous knowledge, and high acceptability ratings may have been influenced by prior success with CBT. The two stage model of evaluation used in this study allowed feedback from recovered patients about content while also incorporating usage data from patients who were currently seeking treatment. This model may prove useful in future studies evaluating content and usability of such interventions.
There are a number of limitations to the current study. The sample was small and limited to treatment-seeking and recovered adolescents. Thus, it is unknown whether the treatment would be feasible or preferred by adolescents with chronic pain in the community. While this study allowed for comparison of recovered and treatment-seeking patient ratings, interpretation of the precise meaning of ratings is difficult. For instance, we cannot infer that a 4/5 rating is significantly less favorable than a 5/5 rating. Without comparison ratings of a similar treatment program we are limited in our ability to draw conclusions in this area. However, we were able to identify Web site modules that users rated less highly than others and will use this information to target future program improvement efforts. Word count of assignments was used as a proxy for engagement, which presents similar problems as no comparison is available to assist with interpretation. We examined user acceptability ratings, while other usability studies have examined real time user performance data such as navigational errors and number of pages viewed. Future research will investigate the clinical efficacy of Web-MAP in a randomized controlled trial. In the context of this trial, additional usability information will be available such as number of visits to the site. The ability to track patterns of user activity and Web site usage is a research advantage provided by such programs (Marks, Cavanaugh & Gega, 2007). Future studies might investigate predictors of compliance and Web site access (Ritterband et al., 2005), and associations between program usage and treatment outcomes.
This study provides strong initial support for the usability and feasibility of Web-MAP, a Web-based family CBT intervention for adolescents with chronic pain. Results also support the importance of incorporating feedback from potential treatment-seeking consumers into the design of Internet interventions. The Internet may be critical to reaching the large population of children with chronic pain who are not able to receive behavioral treatment in their communities. Research during this formative phase of Web-based treatment design may be an important step in developing Internet interventions that are effective and acceptable to specific populations.
Acknowledgments
This work was supported by the National Institutes of Health (HD050674 to T.P.). The authors would also like to thank Anne Manees, Psy.D. for her assistance with this research.
Conflicts of interest: None declared.
References
- Connelly M, Rapoff MA, Thompson N, Connelly W. Headstrong: A pilot study of a CD-ROM intervention for recurrent pediatric headache. Journal of Pediatric Psychology. 2006;31:737–747. doi: 10.1093/jpepsy/jsj003. [DOI] [PubMed] [Google Scholar]
- Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. Journal of Medical Internet Research. 2006;8:e15. doi: 10.2196/jmir.8.3.e15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eccleston C, Morley S, Williams A, Yorke L, Mastroyannopoulou K. Systematic review of randomised controlled trials of psychological therapy for chronic pain in children and adolescents, with a subset meta-analysis of pain relief. Pain. 2002;99:157–165. doi: 10.1016/s0304-3959(02)00072-6. [DOI] [PubMed] [Google Scholar]
- Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services. British Medical Journal. 2004;328:1150. doi: 10.1136/bmj.328.7449.1150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hicks CL, von Baeyer CL, McGrath PJ. Online psychological treatment for pediatric recurrent pain: A randomized evaluation. Journal of Pediatric Psychology. 2006;31:724–736. doi: 10.1093/jpepsy/jsj065. [DOI] [PubMed] [Google Scholar]
- Kashikar-Zuck S. Treatment of children with unexplained chronic pain. Lancet. 2006;367:380–382. doi: 10.1016/S0140-6736(06)68118-X. [DOI] [PubMed] [Google Scholar]
- Kashikar-Zuck S, Swain NF, Jones BA, Graham TB. Efficacy of cognitive-behavioral intervention for juvenile primary fibromyalgia syndrome. Journal of Rheumatology. 2005;32:1594–1602. [PubMed] [Google Scholar]
- Larsson B, Daleflod B, Hakansson L, Melin L. Therapist-assisted versus self-help relaxation treatment of chronic headaches in adolescents: A school-based intervention. Journal of Child Psychology and Psychiatry. 1987;28:127–136. doi: 10.1111/j.1469-7610.1987.tb00657.x. [DOI] [PubMed] [Google Scholar]
- Marks IM, Cavanaugh K, Gega L. Hands-on help: Computer-aided psychotherapy. New York: Psychology Press; 2007. [Google Scholar]
- RAND. Getting to outcomes 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation. Rand corporation technical report series sponsored by the Centers for Disease Control and Prevention. 2004 [Google Scholar]
- Richardson GM, McGrath PJ. Cognitive-behavioral therapy for migraine headaches: A minimal-therapist-contact approach versus a clinic-based approach. Headache: The Journal of Head and Face Pain. 1989;29:352–357. doi: 10.1111/j.1526-4610.1989.hed2906352.x. [DOI] [PubMed] [Google Scholar]
- Ritterband LM, Borowitz S, Cox DJ, Kovatchev B, Walker LS, Lucas V, et al. Using the internet to provide information prescriptions. Pediatrics. 2005;116:643–647. doi: 10.1542/peds.2005-0404. [DOI] [PubMed] [Google Scholar]
- Ritterband LM, Gonder-Frederick LA, Cox DJ, Clifton AD, West RW, Borowitz S. Internet interventions: In review, in use, and into the future. Professional Psychology: Research and Practice. 2003;34:527–534. [Google Scholar]
- Robins PM, Smith SM, Glutting JJ, Bishop CT. A randomized controlled trial of a cognitive-behavioral family intervention for pediatric recurrent abdominal pain. Journal of Pediatric Psychology. 2005;30:397–408. doi: 10.1093/jpepsy/jsi063. [DOI] [PubMed] [Google Scholar]
- Wade SL. Commentary: Computer-based interventions in pediatric psychology. Journal of Pediatric Psychology. 2004;29:269–272. doi: 10.1093/jpepsy/jsh035. [DOI] [PubMed] [Google Scholar]