Content Validity and Satisfaction with a Caregiver Integrated Web-based Rehabilitation Intervention for Persons with Stroke

Abstract

Background

Family members provide valuable contributions during rehabilitation after stroke, but frequently report higher incidences of burden, depression and social isolation during caregiving. Thus, effective interventions to reduce stroke impact on the family are needed.

Objective

To evaluate the content validity and satisfaction of a caregiver focused web-based intervention designed to improve stroke survivor physical function while reducing caregiver negative outcomes.

Method

Caregivers of individuals with stroke (N=6) and expert rehabilitation researchers (N=4) were presented with a novel, web-based intervention (CARE-CITE) designed to foster problem solving and skill-building while facilitating caregiver involvement during constraint-induced movement therapy. Caregivers rated CARE-CITE for usefulness, ease of use, acceptability, and time to complete. Rehabilitation experts evaluated content for accuracy, feasibility, acceptability, problem relevance and ease of use. Ratings were assessed using a 5-point Likert-type response scales (1=strongly disagree to 5=strongly agree).

Results

On average, all caregivers agreed or strongly agreed that the modules were useful (4.42), easy to use (4.60) and acceptable (4.41). Mean total satisfaction score was 4.45, and average review time was 15 minutes per module. Expert reviewers agreed or strongly agreed that each module was accurate (4.95), feasible (4.8), easy to use (4.86) acceptable (4.96) and had appropriate problem relevance (4.65).

Conclusions

The CARE-CITE intervention may be a viable program for caregivers of patients with stroke. Currently a pilot study is underway to evaluate the impact of the intervention on caregiver mental health, family conflict around stroke recovery and stroke survivor upper extremity function.

INTRODUCTION

Stroke affects individuals (on average) every 40 seconds in the United States¹ and is a leading cause of serious, long-term disability. With improving acute treatment of stroke, more survivors are returning home and requiring assistance often from untrained and ill prepared family members.^{2, 3} Caregiver management can improve stroke survivor recovery; but can also increase the caregiver’s fatigue, depression, frustration, and resentment.^{4, 5} Unmet caregiver needs negatively affect the rehabilitation of the stroke survivor and increase caregiver risk of mortality.^6–8 Thus, effective interventions aimed at reducing the impact of stroke on the family are needed and aligned with NIH 2016 Rehabilitation Research priority areas.⁹ However, frequently stroke dyad research focuses on the stroke survivor and does not specifically target the health or well-being of the family.¹⁰ Our area of study explores a novel, web-based intervention (CARE-CITE)¹¹ designed to facilitate caregiver involvement in the home based application of upper extremity constraint-induced movement therapy (CIMT) while promoting caregiver self-management strategies. Based on the Self-Determination Theory¹² of motivation that identifies an autonomy supportive environment as a way to foster an individual’s autonomy and competence, CARE-CITE collaboratively engages the stroke survivor and caregiver in a mutually supportive environment (characterized by empathy, problem-solving, choice and reduction of controlling language). There is evidence that autonomy supportive environments result in better outcomes for persons with other health conditions¹³ such as diabetes¹⁴ and heart failure.¹⁵ The central hypothesis underpinning this research is that a theory-based, caregiver-centered intervention focused on skill building and problem-solving will improve stroke survivor physical function while reducing caregiver negative outcomes and increasing accessibility of participation. Recognizing that caregiver interventions frequently exclude the patients, CARE-CITE addresses the caregiver-stroke survivor interactions, similar to work in traumatic brain injury by Kreutzer and Arango-Lasprilla^16–18 and in stroke by Lutz,^{19, 20} and Oswald²¹ and colleagues. Caregiver training intervention trials initiated during inpatient rehabilitation have had mixed results. Findings from the Family-led Rehabilitation after Stroke in India (ATTEND)²² that shifted rehabilitation activities (including activities of daily living, communication, walking and upper extremity task practice) from healthcare professionals to family members did not improve stroke survivor outcomes. The London Stroke Carers Training Course was a comprehensive competency training for caregivers that reduced total health and social care costs, improved quality of life and reduced caregiver burden in a single-center study,²³ but did not produce similar results in a multi-site clinical trial.²⁴ What was not evaluated in these studies was a progressive and collaborative approach for caregiver knowledge and skill instruction that targets a defined rehabilitation area while addressing the needs and preferences of the family as well as the stroke survivor.

Preliminary work²⁵ from a clinic based CARE-CITE intervention in the format of a printed workbook showed improvements in stroke survivors’ task performance in addition to reductions in depressive symptoms and family conflict for caregivers. However, exit interview results indicated caregivers struggled to meet the time and travel demands of attending the sessions. Consequently, a digital format of CARE-CITE accessed via electronic portable tablet devices within the home environment was developed to provide an ecologically valid delivery option that reduces time and transportation barriers to participation. This web-based version of the CARE-CITE intervention addresses a knowledge gap identified in the American Heart Association (AHA)/American Stroke Association (ASA) stroke caregiver recommendations that noted a lack of web-based interventions that “meet the changing needs of technologically diverse caregivers” (pg3, Bakas, et al., 2017).¹⁰ Working with stroke dyads in tandem, CARE-CITE uses emerging technology by providing a more dynamic learning platform and increasing potential impact by reaching a greater number of individuals. This revised format is currently being evaluated in an ongoing pilot study (NICHD K23 1k23HD080837).¹¹ Data collected from this ongoing study will provide preliminary estimates of the efficacy of CARE-CITE and the potential impact on caregiver mental health, family conflict surrounding stroke recovery, and stroke survivor upper extremity function.

Prior to initiating study enrollment, the web-based version of the CARE-CITE intervention was assessed by content experts and caregiver users. Interventions should be theory-driven and grounded in sound content and conceptual models. Evaluation by content experts familiar with interventions and the intended patient populations establishes content validity but also offers opportunity for refinements that ultimately strengthen the intervention.²⁶ To facilitate translation of effective rehabilitation interventions into clinical practice, assessing user perceptions of usefulness, ease of use, acceptability, as well as overall satisfaction, is a critical preliminary step in research.²⁶ Evaluating caregiver satisfaction and acceptability offers opportunities to connect research results to family healthcare needs by engaging the end user in the preliminary refinements of interventions. Consequently, the purpose of this paper is to evaluate the content validity (by experts) and satisfaction (by caregivers) of the web-based CARE-CITE intervention designed to improve stroke survivor physical function while reducing caregiver negative outcomes.

METHODS

This study conforms to the STROBE guidelines. The framework for obtaining content validity was based on a similar study by Bakas and colleagues²⁶ evaluating the Telephone Assessment and Skill-Building Kit (TASK) for stroke caregivers. Two phases occurred during the assessment and refinement of the CARE-CITE intervention prior to initiation of the ongoing pilot study. After the initial transformation of the content to a digital platform, instructional videos were developed for each section to replace some of the text-heavy content of the hard copy workbook. Once finalized, the web-based intervention was first reviewed by four rehabilitation experts, then by stroke survivor caregivers (N = 6). A structured instrument with open-ended questions for qualitative data was used by experts and caregivers.

CARE-CITE intervention

CARE-CITE was designed to enhance the home-based intervention of CIMT, by helping the caregiver create a therapeutic home environment and encouraging practice of the weaker arm in functional tasks. CARE-CITE informational and skill development content was delivered via six online modules for the caregiver to review in parallel to the 10 sessions of CIMT given to the stroke survivor. Module topics and brief content descriptions are listed in Table 1. At the end of each module, the caregivers respond to self-reflection questions to foster application of information to their own lives. The CARE-CITE content begins with an introduction to the principles of CIMT, including how CIMT is thought to work, methods to adapt tasks at home, the importance of progressing challenging tasks and safety when wearing the mitt. Strategies to address potential stroke survivor frustration and improving adherence to practice using the affected arm are reviewed. Underpinning the content is discussion of the concept of autonomy support, with examples of fostering empathy (caregiver wearing mitt on dominant hand during activities to simulate stroke survivor experience), problem solving (guidance for adaptation of functional activities at home consistent with CIMT principals), instruction in the use of non-controlling language with role playing situations and the importance of creating choice and supporting autonomy in activities. The use of autonomy support was adapted from an autonomy support intervention used in caregiver of persons with heart failure.¹⁵ Each module includes one or more short videos illustrating a caregiver and stroke survivor engaging in task practice in the home environment.

Table 1.

Content of the CARE-CITE Intervention for the Carepartner

Modules	Content
Overview of Modules Structure	Each module has the purpose, information in text format and video clips to provide additional detail and/or illustrate examples of behavior discussed. At the end of each module are 4–5 reflection questions to allow for application of content and 7 questions to obtain feedback on ease of use, acceptability and usefulness of modules.
Module I: Welcome to CARE-CITE	Description of the CARE-CITE project, overall summary of the modules and intent. Welcome survey for CP to fill out with the research interventionist to practice using website and completing questionnaires. One video clip with introductory information.
Module II: Introduction to Concepts and Applications of CIMT	Overview of CIMT, including behavior contract (CP and individual with stroke determine activities when mitt worn and taken off) and home diary (to record activities/time wearing mitt). Four video clips with information about wearing the mitt, behavior contract, home diary, and committing to practice.
Module III: Practice and Goal Setting	Review of role of practice in driving neuroplasticity and recovery after stroke. Discussion of challenge threshold, making mistakes while learning new skills. Guidance provided for problem solving to adapt functional activities at home, both reducing complexity when a task was too difficult and increasing challenge when the task was easily mastered. Two to three brief video clips for each of six themes of practice.
Module IV: Autonomy Support – Creating Partnerships	Creating an Autonomy Supportive Environment – How to be empathic, problem solve in performing tasks in the home setting, use non-controlling language and offer choice. Examples include suggesting CP wear mitt on non-dominant hand while trying tasks and offering alternative activities when individual with stroke becomes frustrated during a challenging task. Recognizing challenges and exploring ways to improve communication (avoid controlling language such as “you should exercise”, “you have to do this”). Seven video clips illustrating understanding another’s viewpoint, problem-solving strategies, providing rationale and providing choice.
Module V: Taking Care of Yourself as a Carepartner	CP self-care – recognizing demands of caregiving role, strategies for stress reduction, opportunities for self-care activities and community resources. (no videos)
Module VI: Reflections	Three videos (limited text) of stroke survivors reflecting on rehabilitation and recovery. Encouraging CP reflection on his/her role in recovery of the individual with stroke.

Participants

A group of four expert researchers (a physical therapist and 3 doctoral nurse researchers) with experience in stroke and caregiving research reviewed the modules using a structured instrument. The physical therapist was an American Physical Therapy board certified neurologic clinical specialist and had coordinated multiple stroke clinical trials for the upper extremity, one nursing researcher with expertise in measurement and who served as PI or co-PI on several caregiving clinical trials, and two nursing researchers specialized in caregiver interventions. Study methods were approved by the university institutional review board and caregiver participants provided informed consent.

Six caregivers of stroke survivors were recruited from local clinics. A caregiver was defined as a family member dwelling in the same household and self-identified as the primary caregiver of a stroke survivor with minimal to moderate upper extremity (UE) deficits (ability to initiate wrist and finger extension). Caregivers were required to be greater than 21 years of age, able to read and write English and have no significant cognitive deficits. All caregivers were female and spouses and Caucasian (4/6), African-American 1/6) or Asian (1/6).

Measures

To assess the content validity of the CARE-CITE intervention, investigator-designed forms were developed based on work by Bakas and colleagues.²⁶ Each expert was asked to rate the degree to which the content in each module addressed the problem relevance within the content domain. This method uses intervention content validity descriptions²⁷ that define elements of an intervention as the cause construct which are then translated into an effect construct (observations or implementation of the intervention).²⁶

The expert review form included seven Likert-type items to assess each of the CARE-CITE modules using a 1 = strongly disagree to 5 = strongly agree rating scale. The items rated: problem relevance (3items), accuracy of content (1 item), feasibility (1 item), acceptability (1 item), and ease of use (1 item). Nine open-ended questions addressed general areas (e.g. most/least helpful modules, areas of concern for improvement, whether clinicians could include the intervention in clinical practice). To obtain a score, each domain (e.g. problem relevance, accuracy) responses were averaged across the four experts.

To assess satisfaction with the CARE-CITE intervention, the adapted forms based on work by Bakas and colleagues,²⁶ were embedded at the end of each of the six modules and caregivers were instructed to complete these assessments immediately after reviewing a module. Satisfaction was defined as: 1) usefulness of overall content, 2) usefulness of written text, 3) usefulness of videos, 4) ease of use, and 5) acceptability. Each area was rated using a 5-point Likert type response scale ranging from 1 = strongly disagree to 5 = strongly agree. Average scores were calculated for each subscale as well as total score. Time in minutes to complete was averaged for each module and these averages were summed to obtain total time to review all modules. Finally, to supplement this data one open-ended question invited general comments or suggestions for improvements to gain a deeper understanding of the experience of the intervention.

Procedures

Content validity was established for the CARE-CITE intervention in the same method typically used for outcome assessments²⁸ through expert review. The content validity form and instructions explaining how to use the form were provided to the four content experts. They also received a password and link to access the CARE-CITE web-based modules. Content experts completed the forms and then returned them electronically to the investigator.

After revisions were integrated based on the expert review, the identified caregivers received the password to access the CARE-CITE modules and were asked to review the modules in the home environment through an electronic tablet or personal computer. At the end of each module, the caregivers completed questions and the responses were downloaded from the electronic database by the investigator.

Data Analysis

Data were analyzed using descriptive statistics. Following the approach of Bakas, et al.,²⁶ the qualitative caregiver comments were reviewed and identified using the pre-determined categories of the satisfaction domains of usefulness, ease of use and acceptability.

RESULTS

Content Validity

Mean expert ratings provided evidence of content validity for the CARE-CITE intervention in all areas, as all subscales were > 4.5 indicating they strongly agreed all components were acceptable (See Table 2). The average overall expert rating was 4.84. Lowest scores occurred for problem relevance category of Module I (Introduction to CARE-CITE) and Module V (Caregiver Self-care). These modules were revised to better address the intent of the study and expand on caregiver self-care strategies. All modules were revised based on qualitative comments and editorial comments from experts. Text in each module was reviewed for literacy level and edited as necessary. Overall text was reduced and video content increased to better illustrate content. Expert comments noted that for content related to task practice, most videos illustrated how to make functional tasks easier for the stroke survivor, but did not have descriptions of how tasks could be progressed, a key concept for CIMT therapy. Consequently, videos were added to illustrate how to make tasks more challenging through collaborative problem solving with the caregiver. General layout of each module was modified to improve ease of navigation, interaction with content and entering free text for reflection questions. Summary of module content was added to the beginning of each module to prompt the viewer of content to be covered in that section. Finally, recommendations were made to include in-person instruction for all caregivers for the first module (reviewing navigation and allowing for practice in answering feedback/reflection questions) to assure understanding of the digital platform.

Table 2.

Mean Content Validity Expert Ratings for CARE-CITE Modules (n=4)

CARE-CITE MODULES	Problem Relevance	Ease of Use	Accuracy	Feasibilitya	Acceptability
MOD I : Intro to Study	4.50	4.75	5.00	4.75	5.00
MOD II: CIMT	4.67	4.75	4.67	5.00	5.00
MOD III: Practice/Goals	4.75	5.00	5.00	4.75	5.00
MOD VI: Autonomy Support	4.75	4.67	5.00	4.75	5.00
MOD V: CP Self-care	4.50	5.00	5.00	4.75	5.00
MOD VI: SS Reflections	4.75	5.00	5.00	----	4.75
Column Average (SD)	4.65 (0.11)	4.86 (0.14)	4.95 (0.12)	4.80 (0.10)	4.96 (0.09)

Response scale: 1=Strongly disagree; 2= disagree; 3=neither agree/disagree; 4=agree;, 5=strongly agree

^aFeasibility was not assessed for Module 6 due to reflective nature of content

Satisfaction with Caregiver Use of Web-based CARE-CITE

Aspects of caregiver satisfaction with CARE-CITE and time to review modules are provided in Table 3. Overall satisfaction with CARE-CITE modules was high. The total average time for viewing the all modules was 1.6 hours.

Table 3.

Carepartner Satisfaction (Usefulness, Ease of Use, and Acceptability) with CARE-CITE and Time to Review (n=6)

CARE-CITE MODULES	TIME (minutes) Mean (SD)	Usefulness (overall; text; video) Mean (SD)	Ease of Use Mean (SD)	Acceptability Mean (SD)	Satisfaction Total Mean (SD)
MOD I: Intro to CARE-CITE	5.80 (2.77)	3.80 (0.77)	3.80 (1.64)	4.00 (0.71)	3.84 (0.74)
MOD II: CIMT	11.00 (7.48)	4.53 (0.52)	4.80 (0.45)	4.60 (0.55)	4.60 (0.47)
MOD III: Practice/Goals	34.67 (21.43)	4.61 (0.50)	4.67 (0.52)	4.67 (0.52)	4.63 (0.34)
MOD IV: Autonomy Support	20.83 (11.21)	4.50 (0.51)	4.83 (0.41)	4.33 (0.52)	4.53 (0.39)
MOD V: CP Self-care	11.50 (4.23)	4.33(0.78)	4.83 (0.41)	4.33 (0.82)	4.46 (0.64)
MOD VI: SS Reflections	11.80 (4.32)	4.75 (0.45)	4.67 (0.52)	4.50 (0.55)	4.67 (0.30)
Average (SD)	15.93 (10.38)	4.42 (0.33)	4.60 (0.40)	4.41 (0.24)	4.45 (0.31)

Response scale: 1=Strongly disagree; 2= disagree; 3=neither agree/disagree; 4=agree;, 5=strongly agree

©Tamilyn Bakas, 2009. Used and modified with permission. Requests for the Satisfaction Scale and scoring instructions can be made to Tamilyn Bakas, University of Cincinnati College of Nursing, PO Box 210038 Cincinnati, Ohio 45221-0038 tamilyn.bakas@uc.edu

Module I (Introduction to CARE-CITE) had lowest scores in usefulness, ease of use and acceptability and the primary qualitative feedback was related to length of introductory video. This video was examined for redundancy, edited and subsequently shortened for overall view time. Ease of use was low, although acceptable in this module, most likely due to the first attempts at using the system.

Qualitative comments from Caregivers

All of the caregivers felt the CARE-CITE intervention was a beneficial concept and expressed appreciation to be included as a focus of a rehabilitation intervention. For each domain of satisfaction, representative caregiver quotes are presented here.

Usefulness

• Video that contrasted positive action with the negative action was helpful. Use more often if possible to better illustrate the approach.

• [Module VI – Stroke Survivor reflections] was an excellent way to complete the program, tied the segments together.

Ease of use

• Completed module but trouble saving, had to re-do.

• Found all modules easy to navigate.

• [Module III – Practice/goals] videos awesome but too long and should be split in two.

Acceptability

• Real people – real strokes - easy to identify with.

• [Module III – Practice/goals] I like seeing other patients work through their challenges, especially with different types of strokes and after-effects.

• Very good concepts. If these are given to the patients in the early stages when they have not become comfortable with their strong hands [non-hemiplegic side], will make a big difference in rehab therapy. I strongly recommend these modules to help the patients and the caregiver.

• [Module IV – Autonomy support] I liked this module and the reminders of the motivating words and collaborative approach are important.

General comments

One of the caregivers express reflections that demonstrated new insights and understanding of her husband’s memory and behavior changes and cultivation of empathy through observation of other stroke survivor’s perspectives.

Going through CARE-CITE helped me to realize that so many of [my husband’s] behaviors were related to the stroke… it was hard to tell if this was him or the stroke. I thought - wow, maybe he isn’t the ass I think he is sometimes, sometimes he is, but I realized that there are times he can’t help it. Then I think – what is he thinking/going through/struggling with on the inside? It changed me going through some of the questions [questions asking about autonomy support] – caused me to react to him differently, when he is angry, depressed. I think he doesn’t know some of his behaviors are related to the stroke. We think of the brain attack as the physical, but miss how it affects thoughts, emotions, feelings, non-feelings.

A caregiver who was a young mother shared different challenges encountered by caregivers depending upon age and family demands.

Caregivers need support. I believe speaking with someone who is ahead of you on the curve is extremely helpful. Not just anyone though, someone who shares your circumstances.

DISCUSSION

The primary goal of this study was to evaluate the content validity and satisfaction of a caregiver focused web-based intervention designed to improve stroke survivor physical function while reducing caregiver negative outcomes and increasing accessibility of participation. While evidence is building supporting interventions for caregivers, little is known about family focused approaches that deliver interventions that target both the stroke survivor and caregiver and evaluate outcomes for the dyad. Home based stroke caregiver interventions delivered via telephone²⁹ and through multi-disciplinary in-home educational sessions²¹ have been developed, but these address a spectrum of post-stroke sequelae, versus being specifically designed to engage the caregiver in rehabilitation therapies. This study provides a model to evaluate a dyadic intervention in upper extremity task oriented rehabilitation.

Content Validity

Expert reviewers supported the content validity of CARE-CITE and agreed or strongly agreed that each module was accurate, feasible, easy to use, acceptable and had appropriate problem relevance. The video content was described as “excellent” and consistently noted by reviewers as “helpful to understand the information” in each module. The primary reviewer modifications suggested were related to length and literacy level of written text. Most of the research surrounding caregiving in stroke stems from nursing, so incorporating rehabilitation nurses with extensive caregiver research was a critical component to establishing relevancy of the intervention. Thus, the transition of the previous version of the intervention was successful to the web-based format.

Satisfaction

Overall, all caregivers were satisfied with CARE-CITE and agreed or strongly agreed that the modules were useful, easy to use and acceptable. These results are similar to caregiver satisfaction reported by Ostwald and colleagues³⁰ of overwhelming satisfaction (3.80; range= 1-not satisfied to 4-very satisfied) for the CAReS (Committed to Assisting with Recovery after Stroke) study evaluating an in-home educational guideline training program and total satisfaction scores by Bakas and colleagues²⁶ (4.41; range 1=strongly disagree to 5=strongly agree) for their Telephone Assessment and Skill Building Kit (TASK) intervention. Although interventions described by Ostwald³⁰ and Bakas²⁶ offer a broader spectrum of content than the specific rehabilitation therapy approach targeted in CARE-CITE, caregiver feedback and satisfaction provided valuable information for intervention refinements for each of these approaches. Results from our study suggest that caregivers value a focused, skills-based, problem-solving intervention to improve upper extremity function in the stroke survivor as well as caregiver interventions that may include broader training in areas such as medical management, mobility training and swallowing. Gaining perspectives from stakeholders is a key aspect of patient centered care and patient centered outcomes research. Supporting caregiver engagement to shape the development of the CARE-CITE intervention prior to evaluation in a large scale and costly efficacy study provides opportunities to refine the approach to better address caregiver needs and concerns. Some feedback indicated that the modules may not fully address younger patients with stroke, although further information will be needed to identify specific areas upon which to expand. Comments regarding length of the videos resulted in revision of the introduction video.

By assessing both expert rated content validity and participant acceptance (satisfaction), results from this assessment guided the revisions of the CARE-CITE intervention. The ongoing study of CARE-CITE will provide an initial evaluation of the impact of the CARE-CITE intervention on caregiver mental health, family conflict around stroke recovery and stroke survivor upper extremity function.

Study Limitations

All caregivers were female and spouses, limiting applicability to male or other dyad relationships. Further research is recommended that includes a more diverse set of caregivers based on ethnicity, race, gender and relationship to stroke survivor. While the content experts spanned a range of backgrounds addressing expertise in rehabilitation therapies, stroke, caregiving and intervention development, the number of experts and caregivers was small, as this evaluation was an initial assessment prior to initiation of the pilot study. The same caregiver assessments for each module as well as an overall study exit interview to further assess feasibility and satisfaction with CARE-CITE are included in the ongoing pilot study and will be used to shape future applications of the intervention.

CONCLUSIONS

The CARE-CITE intervention is a novel web-based family centered intervention designed to improve outcomes for both the stroke survivor and caregiver. This study demonstrates evidence supporting CARE-CITE content validity and user satisfaction, a critical first step before testing the efficacy of an intervention in a large clinical trial. The overall significance of this research is to increase understanding and further development of interventions to foster family involvement in the rehabilitation process.

Abbreviations

CIMT

Constraint-induced movement therapy

CARE-CITE

Carepartner and Constraint-Induced Therapy