Fidelity and Acceptability of an Adaptive Intervention for Caregivers: An Exploratory Study

Abstract

Objective

There has been growing interest in providing tailored or adaptive interventions to family caregivers as a way of addressing their heterogeneity of risk factors and other needs. A particular challenge in an adaptive study is to implement the individualized intervention protocol as planned (program fidelity). This study explores the fidelity of implementation of an adaptive intervention for family caregivers of persons with dementia and its acceptability to caregivers.

Method

Using a sample of 35 caregivers of person with dementia who participated in a program development study, we gathered information on acceptability and fidelity of the program from multiple sources, including caregiver and counselor reports and ratings of recordings of sessions.

Results

Findings show that caregivers have high levels of acceptance of the intervention plan and high ratings of satisfaction with the program. Ratings of satisfaction and counselor competence were not associated with the amount of treatment provided. Ratings by counselors and independent raters found good fidelity for two of the three program domains.

Discussion

The results demonstrate that trained counselors can follow a tailored intervention plan and that caregivers’ experience of the program did not differ depending on how much intervention was provided. A next step is to determine how an adaptive protocol would affect caregiver outcomes.

Adaptive interventions, in which a program or intervention is tailored to the specific needs of an individual, have been used increasingly for treatment of many different types of social and behavioral challenges (Borhani et al, 1991; Breslin, Sobell, Sobell, Cunningham, Sdao-Jarvie, & Borsoi, 1998; Burgio et al., 2009; Dishion & Kavanagh, 2000; Lavori, Dawson & Rush, 2000; Murphy, Lynch, McKay, Oslin & TenHave, 2007). For example, the highly successful Fast Track Program for prevention of conduct disorders in children assigned two program components, family counseling and tutoring in reading, based on initial assessment of need in each of those areas (Conduct Problems Prevention Research Group, 2002). In contrast to the traditional fixed intervention where the goal is to give each person similar exposure to elements of the intervention, parents and children in Fast Track received varying amounts of the two program components, thereby achieving a better match of intervention and need. This type of tailored approach may be particularly appropriate for interventions with family caregivers. Caregivers have considerable heterogeneity in the type and amount of stressors they experience, and in their subjective reports of the impact that caregiving has on their social, emotional and physical health (Zarit, Femia, Kim, & Whitlatch, 2010; Knight, Lutzky & Macofsky-Urban, 1993). Knight and colleagues (1993) noted that when considering caregiver interventions the important question is not simply whether the intervention works for the “average” caregiver, but whether the intervention works for a single caregiver with his or her own specific needs. A typical one-size-fits-all intervention can miss the mark, because it may target a problem the caregiver does not have, while failing to provide a sufficient amount of the intervention for those challenges that most trouble the caregiver. The methodological danger in providing an intervention to address an issue that a caregiver does not have is the reduction of the statistical power to detect change in the dependent variables of interest. The psychological danger is that it could actually lead to a worsening of the targeted problem (e.g., depressive symptoms) (Collins, Murphy and Bierman, 2004). Failing to address the problems or concerns that are most troubling will also lead to poor outcomes. By contrast, an adaptive intervention that tailors the type and amount of program delivery to a caregiver’s specific needs can address the unique stressors or challenges that caregivers face, but would not intervene on challenges that they do not have.

Clinical interventions have long been tailored to the specific needs of clients. In the caregiving literature, the New York University Caregiver Counseling and Support Intervention tailored its specific focus based on the clinician’s evaluation of the caregiver’s most pressing needs. While this intervention has reported impressive short-term and long-term outcomes (e.g., Mittelman, Roth, Coon & Haley, 2004), the lack of objective criteria for tailoring decisions makes replication difficult.

An alternative approach is to assign intervention components based on objective and hence replicable criteria of need (Collins et al., 2004; Collins, Murphy, Nair, & Strecher, 2005). The REACH II study represents the best example to date of an adaptive intervention applied to caregivers (Belle et al., 2006). In REACH II, program components were assigned based on caregiver scores on a risk appraisal measure that assessed six domains (depressive symptomatology, caregiver burden, self care and healthy behaviors, social support, safety, and patient behavior problems). The final decisions about which components to implement were determined through a negotiated process with caregivers that identified up to three challenges related either to management of the care receiver or to the caregiver’s own stress and emotional health. The results generally showed improved quality of life for caregivers, using a composite measure of caregiver outcomes (e.g., burden, depression). Subsequent work has refined the risk assessment measure used in REACH II to improve intervention planning (Czaja et al., 2009).

Although REACH II and other adaptive interventions represent a promising direction, evaluations of their efficacy face complexities over and above traditional, manual driven intervention programs. Two issues are highlighted: fidelity of program implementation and acceptability of the program. Fidelity is defined as the degree to which the interventionist utilizes specified procedures while foregoing other intervention techniques (Kazdin, 2003; Waltz, Addis, Koerner, & Jacobson, 1993). Fidelity of implementation is especially challenging in adaptive interventions, because protocols differ from one person to the next. Interventionists may not have enough guidance or be tempted to use all their therapeutic tools, rather than stay within a more restricted protocol. With regard to fidelity of implementation, there have, to date, been no published reports pertaining to adaptive intervention plans for caregivers.

For adaptive interventions that vary in the amount of a program that is provided, one useful marker of fidelity is the number of program sessions. The number of sessions should be consistent with the initial plan for a particular individual. More central to the issue of fidelity is whether the specific program components that were assigned for each individual were actually administered during the sessions. Implementation has been measured in a number of ways, including checklists completed by interventionists at the end of sessions (Judge, Yarry & Orsulic-Jeras, 2009) and ratings made by participants either after sessions or following the end of the program (Hogue, Liddle, Rowe, Turner, & Dakof, 1998; Hogue, Daube, Faw, Cecero, & Liddle, 2006). The gold standard for assessing fidelity is ratings of audio or video recordings of the sessions. Waltz et al. (1993) observe that ratings of sessions may focus on simple occurrence versus nonoccurrence of an intervention strategy or may assess the frequency or amount of coverage of an intervention component during a session. The rating of amount of coverage of a program component would be particularly important in adaptive interventions for determining whether the implementation of that component varied according to the pre-determined plan. These ratings would indicate that for any given individual the intended program procedures are actually implemented, while other procedures are used less frequently or not at all. In contrast to traditional fixed research protocols, where fidelity is confirmed when the program is delivered to each person in a similar frequency and intensity, an adaptive intervention should deliver different components and/or different amounts of each component to each person depending on his/her need. Although clinicians would be expected to vary the amount and type of program components according to the adaptive plan, they also would be expected to provide similar levels of general therapeutic qualities, such as empathy and support. Likewise, their competence in delivering the intervention should be constant across varying levels of the program (Waltz et al., 1993).

The second missing piece in the literature is the acceptability of adaptive interventions to the people who receive them. Participants in research trials of tailored interventions have not typically been involved in discussions of the assignment of specific program components, other than passively providing information about the stressors they encounter or their well-being. We do not know whether participants agree with the goals of the intervention, or if at the end of their participation they are satisfied with the types and amount of intervention components they received. REACH II (Belle et al., 2006) again is an exception. In the REACH II intervention, caregivers and counselors worked together to formulate up to three goals of the program. Expanding on this approach, caregivers could be actively engaged in evaluating the acceptability of an intervention’s goals. This could be accomplished both prior to the implementation of the program when goal setting takes place, and again following completion of the program. Acceptability of adaptive interventions could also be assessed using participants’ overall satisfaction with the program or with specific program components. As with non-specific interventionist qualities, overall satisfaction with the intervention should be consistent regardless of the combination or total number of implemented components. Satisfaction with specific program components, however, should be greater for those components that were actually implemented or when “dosage” of these components was greater.

Study Aims

This study used data from the ACES project (Advanced Caregiver Education and Support Program), whose goal was to design components and procedures for implementing an adaptive intervention with family caregivers who have primary responsibilities of care to individuals with dementia (IWD) (Zarit et al., 2010). Specific aims of the present study were: (1) Determine the acceptability of the adaptive approach to caregivers; (2) Determine the fidelity of the intervention, that is, how well the delivery and intensity of the program components matched the caregivers’ initial intervention assignment; (3) Assess the competence of counselors as an indicator of adherence to the intervention plans; and (4) Examine caregivers’ satisfaction with the program and their counselors, and determine if satisfaction varied by the amount or type of program components received.

Method

Participants

The sample was comprised of family caregivers of community-dwelling individuals with dementia (IWD) living in the greater Cleveland, Ohio (USA) metropolitan area. Participants were recruited from service programs and through direct outreach. People who expressed interest in participating were screened for eligibility with a telephone interview. Eligibility criteria were: self-identified primary caregiver for an IWD, having regular contact and responsibility for the IWD, and reporting some level of care-related stress. Of 39 eligible caregivers who were contacted, 35 (90%) of them agreed to participate in the study. Table 1 presents demographic characteristics, including caregivers’ and care receivers’ education, marital status, work status, and income.

Table 1.

Social Demographic Characteristics of, Caregivers and Care Receivers

	Caregivers (n = 35)	Care Receivers (n = 35)
Means and Standard Deviations
Age	61.34 (12.32)	76.63 (10.20)
Educationa	3.20 (0.94)	2.49 (1.33)
Incomeb	5.58 (2.90)	2.47 (2.48)
Percentages
Kin Relationship of Caregivers
To Care Receivers
Husband	20.00
Wife	34.30
Daughterc	31.40
Daughter-in-lawc	2.90
Sonc	8.60
Other	2.90
Marital Status
Married	68.60	54.30
Gender
Women	65.70	68.60
Ethnicity
White	65.70	65.70
Employed (part- and full-time)	48.60	--
Living in Same Household	82.90	--

^a0 = less than high school degree, 1 = some high school, 2 = high school degree, 3 = some college, 4 = college degree, 5= post college graduate.

^b0 = less than $10,000, 1= $10,000 – 19,999, 3 = 20,000 – 29,999, 11 = 100,000 or more.

Note. Standard deviation in parentheses.

^cAll parents were mothers.

Procedures

Participants first completed an initial interview conducted by one of the study counselors, who assessed the caregivers on three broad domains of caregiver stress: (1) Care demands; (2) Caregiver’s Roles and Relationships; and (3) Resources (formal help). Each domain was assessed by a set of measures (see Zarit, et al., 2010) from which risk scores were calculated. The research team then analyzed the scores, applying an algorithm that yielded an initial rating of high, medium or low stress for each of the three domains. The algorithm had previously been developed in pilot work with a sample of 30 caregivers meeting the same eligibility criteria as the current participants (Zarit et al., 2010). A consensus conference consisting of the counselor, project manager and senior investigators reviewed the initial stress ratings and used input from the counselor and caregiver (using the open-ended responses in the interview) to assign the caregiver into a recommended level of program intensity, that is, a high, medium, or low level for each of the three broad domains. Following the consensus conference, the counselor and caregiver met to discuss the findings of the assessment and the resulting program recommendations.

The varied levels of intensity for each of the program’s three components (high, medium, and low levels) reflect a tailored intervention plan that was constructed for each participant. Depending on the caregiver’s responses to the risk measures in the assessment, the plan included coverage of specific program components to be implemented over a recommended number of sessions. A component delivered at a low intensity involved relatively brief coverage of the material, providing mostly a review of the domain and answering any questions or concerns that the caregiver might have. At a medium level, the component involved greater coverage of the material, emphasizing a few to several of the specific strategies to address the caregiver’s challenges over one to three sessions. At the high level, the component was delivered at the greatest level of coverage, emphasizing all of the specific intervention strategies over three or more program sessions. The number of overall sessions thus ranged from a minimum of three sessions for a caregiver assigned at low levels for all three domains, to a maximum of seven sessions for a caregiver assigned at high levels for all three domains. Note that these sessions were in addition to the initial assessment session and the follow-up planning session. For caregivers who were assigned a high stress level in the domain, Roles and Relationships, one of the sessions was a family meeting (see below).

The three program components were broadly based on a cognitive-behavioral framework, drawing upon widely used strategies for addressing management of behavioral stressors and other care demands, how caregivers view their role and their relationships with potential and actual informal helpers, and lack of formal help (Belle et al., 2006; Gallagher-Thompson et al., 2000; Gitlin et al., 2003; Mittelman, Roth, Haley, & Zarit, 2004; Whitlatch, Zarit, & von Eye, 1991; Zarit, Orr, & Zarit, 1985). Each component was carried out systematically within each session. For example, for a caregiver at a high level for Care Demands, medium level for Roles and Relationships, and low level for Resources, a total of four sessions would be recommended whereby each session would cover material for each of the domains. In the example, session 1 would cover an introduction of the ABC (antecedent-behavior-consequences) model of behavior management, a discussion of care values to begin work on feelings of loss and strain surrounding the relationship, and an exploration of how the caregiver’s resources can help him or her achieve specific goals (i.e., take care of oneself or enjoy respite). In each session, homework assignments were given to caregivers to reinforce the skills learned from the session. In addition, for caregivers who were prescribed a family meeting, one session was devoted to preparing for the meeting (i.e., discussing who would be invited and reviewing what will be covered) and one session was the family meeting itself.

Measures

Acceptability of the Adaptive Intervention Plan

Acceptability of the adaptive intervention plan was assessed by determining the degree to which caregivers agreed with the initial assessment of their stress levels and with the proposed intervention plans. Within two weeks of the initial interview, caregivers met with their counselor to review the findings of the assessment and to engage in session planning, including which components would be emphasized and the planned number of sessions. The findings of the assessment were presented using a visual display that showed the caregiver’s stress levels for the components of each of the three domains. A 3-point numerical rating scale was used to assess the caregiver’s agreement on their stress levels ranging from (0) stress rating is accurate, (1) rating is too low, and (2) rating is too high. Following discussion of stress levels for a domain, counselors then presented an overview of the tailored intervention plan for the domain, which caregivers read and discussed with the counselor. To measure acceptability of the intervention plans, caregivers were asked to rate the plan for each of the three domains on a 3-point numerical rating scale ranging from (0) the plan is reasonable, (1) the plan is more than I need, and (2) the plan is less than I need.

Acceptability was also assessed in two other ways after the completion of the final program session. First, caregivers were asked about the assigned intervention strategies they received and if they found each of those strategies to be helpful. Strategies were drawn from components of each of the program domains. These included receiving information on Alzheimer’s disease, learning tips for improving communication, learning methods for controlling behavior problems, learning the warning signs of stress, and learning about resources available to caregivers. Caregivers rated each strategy on a 4-point numerical rating scale ranging from (0) not at all helpful to (3) very helpful.

Second, caregivers rated their overall satisfaction with the intervention and the number of sessions they received. Overall satisfaction was rated on a 5 point numerical rating scale, ranging from (0) very dissatisfied to (4) very satisfied. Satisfaction with the number of sessions received was rated on a 3-point numerical rating scale ranging from (1) just the right number, (2) too few, and (3) too many.

Program Fidelity and Adherence

Treatment fidelity and adherence were assessed with three measures.

(1) Counselor Ratings of Program Fidelity

The Treatment Summary Checklist (TSC) is a 12-item measure of program adherence, that is, whether specific therapeutic strategies were used in a session. The TSC was developed for this study, incorporating items from the widely used Therapeutic Procedures Inventory (TPI; McNeilly & Howard, 1991) that describes cognitive and behavioral treatment strategies commonly used in caregiver interventions. Items were selected from the TPI that were relevant for the proposed components or modified where necessary so that they corresponded more appropriately to elements of the intervention (e.g., Explored practical solutions to managing behavior problems; Reframed barriers to accepting more help). The resulting scale was comprised of 12 strategies that mapped onto the three treatment domains: (1) Care demands; (2) Roles and Relationships; and (3) Resources. At the end of each session, counselors rated the amount of coverage in the session of each strategy using a four-point scale ranging from (0) not at all to (3) extensively.

(2) Ratings of Program Fidelity from Audio Recordings of Sessions

With caregivers’ consent, counselors recorded the program sessions. Audio recordings were made for 50 percent of the total number of sessions across all participants. The remaining sessions were not recorded because not all caregivers gave consent and due to equipment failures. We did not record family sessions, because of confidentiality issues.

Codes were developed to indicate the frequency with which counselors used each of the 12 strategies included in the TSC, with possible ratings for each strategy ranging from 0 (not at all) to 5 (a great deal). The resulting measure was called the Treatment Summary Checklist Audio (TSC-A). Higher scores indicated that greater focus was given to the specific domain/area during the session.

Raters were undergraduate students who were trained by one of the authors (JEL). Once raters reached a threshold reliability of intraclass correlation (ICC) of 0.80, they began coding the session recordings. During the coding period, the raters were supervised weekly by two of the authors (JEL, SHZ) to prevent rater drift and to clarify any questions pertaining to rating the sessions. Two raters were assigned to code each recording, and pairs of ratings were averaged to create a final score for each scale item. Raters were blind with respect to the intervention plan for any particular participant. The mean intra-class correlation across all coder pairs was .82.

An exploratory factor analysis using the principal components procedure and a varimax rotation was conducted on the coded ratings of session recordings. Results of the factor analysis are shown in Table 2. Four factors were obtained with eigenvalues greater than 1. Two factors included items that mapped directly onto domain 1 (Care Demands) and domain 3 (Resources). The other two factors included components of domain 2 (Roles and Relationships); these were labeled “role and relationship issues” and “self-care”. As shown in Table 2, the reliability of subscales ranged from minimally acceptable to very good with alphas from .67 to .86.

Table 2.

Factor Loadings from the Principal Components Analysis of Ratings of Program Adherence from Session Recordings (Treatment Summary Checklist—Audio)

	Factor 1 Behavior issues	Factor 2 Interpersonal issues	Factor 3 Self-care	Factor 4 Resources (paid help)
Internal reliability (Cronbach’s α)	0.84	0.67	0.79	0.86
Managing CR’s behavior	0.88
Responding to CR’s behavior	0.85
Presenting information about dementia	0.85
Reframing beliefs about CR’s behavior	0.80
Developing ways to respond to family members		0.84
Identifying informal support from family/friends		0.82
Reframing beliefs about informal support		0.79
Identifying conflicts with family regarding CR		0.66
Increasing time for taking care of CG			0.85
Identifying self-care activities (e.g., exercise)			0.79
Presenting information about resources				0.87
Exploring practical solutions to getting more paid help				0.86

NOTE: CR= care receiver; CG = caregiver

We correlated the four subscales of the TSC-A with comparable subscales created from the TSC. Given that not all sessions were recorded, correlations were made only for those sessions where there was both an audio recording (50% of all sessions) and counselor ratings. Results revealed that there were moderate correlations between the TSC and TSC-A (ranging from .44 to .61, p<.05; see Table 3).

Table 3.

Correlations between Coders’ and Counselors’ Ratings of Coverage of Program Components within Sessions

Treatment Modules	1	2	3	4	5	6	7	8
1. Manage behavior issues (Rater)	--
2. Manage behavior issues (Counselor)	0.61*	--
3. Interpersonal issues (Rater)	−0.28	−0.18	--
4. Interpersonal issues (Counselor)	−0.14	−0.01	0.52*	--
5. Resources (Rater)	−0.07	−0.06	0.10	−0.00	--
6. Resources (Counselor)	−0.07	−0.05	0.08	0.010	0.44*	--
7. Self-care (Rater)	−0.35	−0.45*	0.12	−0.10	0.12	0.19	--
8. Self-care (Counselor)	−0.23	−0.23	0.24	0.24	0.26	0.38	0.47*	--

^*p < .05.

(3) Ratings of Counselors’ Competence

Counselors’ competence was rated both from audio recordings of individual sessions and by caregivers at the end of their participation with the program. Coders used a single global coding item of Counselors’ Competence, which was derived from the Cognitive Therapy Scale (Beck & Young, 1985). Counselor competence was defined as counselors’ skillfulness in providing a therapeutic milieu, in conceptualizing the caregivers’ distress and problems within a specific theoretical framework (DeRubeis, Hollon, Evans, & Bemis, 1982), and in applying recognized intervention techniques or methods of the intervention. High ratings of counselor competence required a good therapeutic alliance and adherence to the intervention protocol.

To rate the counselors’ competence, raters considered the entire duration of the session and then assigned a competence rating for that session along a 7-point scale ranging from 0 (poor) to 6 (excellent). For the purpose of analysis, we took an average competence score across all sessions for each counselor. We calculated ICCs to assess interrater reliability on the basis of independently produced ratings. The average ICC between raters across all ratings was .81.

Counselors’ competence was also rated by the caregiver. A measure consisting of two items was developed for this project and administered during the post-session assessment. The first item assessed the counselor’s skill and knowledge, and the second assessed the counselor’s helpfulness. Caregivers rated each item on a scale ranging from 0 (very bad) to 4 (excellent). Cronbach’s alpha was 0.67. The correlation between caregivers’ and coders’ ratings on counselors’ competence was in the moderate range (r=.46, p<.05).

Other measures

Number of sessions delivered was recorded in order to determine if caregivers’ ratings of satisfaction with the intervention and of their counselor’s competence were affected by the amount of intervention.

Analysis plan

For the first aim of the study, the acceptability of the adaptive approach, we used descriptive statistics to explore the degree to which caregivers agreed or disagreed with the stress ratings generated by the initial assessment and the recommendations for the intervention plan derived from that assessment. For the second aim of the study, to examine whether the level of the program was delivered as intended, we conducted univariate analyses of variance (ANOVA) statistics to examine the relation between the proposed level of the program for each of the three domains (high, medium, and low levels of intensity) and the amount of actual program coverage given to that area. The amount of program coverage was determined using coded ratings from session recordings on the TSC-A and from counselors’ end of session ratings on the TSC. For the third aim, assessing counselor competence, competence assessed by caregivers and rated from audio-recordings were correlated with both the number and level of program sessions assigned for each domain. Finally, caregivers’ appraisals of the intervention (overall satisfaction, ratings of helpfulness of specific components) following their completion of the program were considered. The effect of program “dosage” (number of sessions) on these scores was also evaluated.

Results

Acceptability of the Adaptive Approach: Caregivers’ agreement with the intervention plan

Caregivers indicated high levels of agreement with both the stress ratings determined in the initial assessment and with their proposed intervention plans. For the first domain, Care Demands, 97% of caregivers agreed with their stress rating on the sub-component, Activity of Daily Living Tasks and Challenges, and 91% agreed with their rating on the other subcomponent, Behavior Challenges. Similarly, 91% agreed that the proposed amount of the intervention for the Care Demands domain was “just right.” For the other 9% of caregivers, 3% responded that the proposed intervention plan was “more than what they needed,” and 6% responded that the plan was “less than what they needed.” For the second domain, Roles and Relationships, the caregivers’ level of agreement on the stress ratings of three sub-components, Relationship with the Care Receiver, Relationship with Family and Friends, and Self Care were similarly high, at 97% agreement, 94% agreement, and 91% agreement, respectively. In terms of their proposed intervention plan for Roles and Relationships, 89% reported agreement with the plan. As with Care Demands, the 11% who did not agree with the proposed plan were split evenly between those who felt the plan was too much and too little. For the third domain, Resources, 97% of caregivers agreed with their ratings on two sub-component stress scores: Informal Support and Formal Support. Ninety seven percent of caregivers indicated that their informal support stress score was correct, and 91% of caregivers agreed with their formal support stress score. Eighty nine percent of caregivers agreed with the proposed intervention plan for Resources with the remaining 11% equally split as before. As would be expected with these high agreement ratings, caregivers’ level of agreement did not vary based on initial levels of stress scores.

Program Adherence: Program delivery as intended

Table 4 shows the relation between the amount of coverage within sessions for each domain and the participants’ assigned level of intensity (low, medium, or high level) for that domain. Each cell of the table shows the degree to which specific strategies and material were covered during the sessions based on the counselors’ ratings and the ratings made from recordings. A higher mean value indicated greater coverage of that material during sessions. A series of univariate ANOVAs were used to assess if the amount of coverage of material was related to the assigned level of intensity. Turning first to counselors’ ratings, level of intensity had significant effects on coverage for Roles and Relationships [F (2, 31) =5.71, p<.01] and for Resources [F (2, 31) = 11.17, p<.01). For both domains, there was greater coverage of the material when the level of assigned intensity of the intervention was higher. A Tukey post-hoc test for Roles and Relationships showed that the significance lay in the difference between the medium and high level groups. A Tukey post-hoc for Resources revealed that the significant differences lay between caregivers in the low and high level groups and in the medium and high level groups. For the Care Demands domain, the differences in the amount of coverage of the material among the levels of intensity trended in the predicted direction, but were not statistically significant. The effect sizes of each ANOVA were evaluated by partial eta-square and, as shown in Table 4, ranged from .13 to .42, indicating that intended level of intensity explained between 13 and 42 percent of the overall variance in the outcome.

Table 4.

The Relation of Ratings of Amount of Coverage in Sessions of Each Treatment Domain (Counselors’ and Raters’ Ratings) by Level of Assigned Intervention

	Domain 1	Domain 2	Domain 3
	Counselor Raters	Counselor Raters	Counselor Raters
Assigned Intervention Level	Mean (SD)	Mean (SD)	Mean (SD)
Low	2.00a /4.00a	2.50 (3.6) / 1.25 (1.0)	5.10 (1.7)b/ 2.00 (0.9)b
Medium	10.43 (4.8) / 7.10 (4.7)	6.50 (1.5)b / 7.80 (4.0)	4.50 (1.4)bc/ 2.01 (1.5)
High	12.47 (5.8) / 9.40 (2.5)	11.70 (1.1)b /8.30 (2.7)	14.40 (1.7) bc/3.60 (1.1)b
Effect sizes	0.13/ 0.15	0.27/ 0.23	0.42/ 0.35

Note.

^aThere was only one subject for this cell.

^bTukey’s posthoc test showed a significant difference (p < .05) between the low and high level groups.

^cTukey’s posthoc test showed a significant difference (p < .05) between the medium and high Level groups.

Results of the relation of ratings of session recordings to prescribed level of program intensity were similar to those of the counselors’ ratings (see Table 4). There were significant associations of assigned level of intensity with the amount of coverage in two domains, Roles and Relationship [F (2, 22) = 3.28, p <.05] and Resources [F (2, 22) = 5.85, p<.01]. Tukey post-hoc tests revealed that significance was due to differences between the low and high levels for Resources. The post-hoc test was not significant for Roles and Relationships. For Care Demands, the difference was again in the predicted direction, but was not significant. As shown in Table 4, effect sizes ranged from .15 to .35.

Caregiver and Raters’ Rating of Counselors’ Competence

Caregivers’ scores for counselor’s competence had a mean of 3.43 (SD=.54) out of a possible 4 points. Scores of competence from the audio recordings were also high, with a mean of 5.7 (SD=.32) out of a possible 7 points. Caregivers’ and raters’ competence scores were significantly correlated (r = .41, p < .05). Correlations of the two competence scores with amount of planned treatment and number of treatment sessions are shown in Table 5. None of the scores was statistically significant, though the relation of ratings of competence from the recordings and number of sessions approached significance (r = −.39, p < .05), with more sessions associated with lower competence scores.

Table 5.

Correlations of Ratings of Counselors’ Competence (Caregivers’ and Coders’ Ratings) with Assigned Intervention Level and Number of Program Sessions.

Competence	Caregivers’ Ratings	Coders’ Ratings
Assigned Intervention Level
Domain 1	.18	−.12
Domain 2	.12	.01
Domain 3	.10	−.07
Number of Sessions	.05	−.39 t

^tp = .052

Caregiver Satisfaction with the Adaptive Intervention

Table 6 shows the mean levels of caregivers’ ratings of the helpfulness of various intervention components that comprised the three program domains. Caregivers rated the helpfulness of the majority of the components around 2 out of a possible 3 points, indicating that the components were at least somewhat helpful. Helpfulness ratings, however, were not associated with the level of the component received. Overall satisfaction with the program was high with 66 percent of caregivers indicating they were very satisfied, and 28 percent reporting they were somewhat satisfied. Eighty five percent of caregivers indicated that the number of sessions they received was just the right number. Overall satisfaction was not significantly correlated with the number of sessions that caregivers received (r = .31, p > .05).

Table 6.

Mean Ratings of the Helpfulness of Program Components for Each Level of Assigned Intervention

Domains and Components	Assigned Intervention Level
Care Demands	Level 1	Level 2	Level 3
Information about Alzheimer’s	1.00	1.92	1.78
Tips/Techniques for improving communication	--	2.08	2.00
Controlling behavioral challenges	2.00	2.38	1.88
Engaging relative to reduce behavior challenges	--	1.50	1.59
Caregiver’s Roles and Relationships
Warning signs of caregiving stress	2.50	2.15	1.82
Examining beliefs about the care role	0.50	1.15	1.47
Strategies for time management and relaxation	2.00	1.85	1.76
Balance of caregiving and self care	1.50	1.85	1.94
Developing a family plan			1.62
Resources
Community Resources/Accepting help from others	2.00	1.75	2.00
Plan for future needs	1.67	1.62	1.56
Rationale for allowing others to help	1.90	1.69	1.78
Identify areas of need where help would be beneficial	--	1.85	2.00
Linking to services	--	1.38	1.78
Exploring strategies for allowing help	--	--	2.00

Note: Ratings of helpfulness did not differ significantly by level of assigned program intensity. Some program components were not administered to Levels 1 and 2 and caregivers were not asked about whether they received them. Those cells are empty in the table.

Discussion

There has been growing interest in providing tailored or adaptive interventions to family caregivers as a way of addressing their heterogeneity of risk factors and other needs (Belle et al., 2006; Burgio, Collins, Schmid, Wharton, McCallum & DeCoster, 2009)). Prior work had not considered whether program fidelity can be achieved, whereby participants receive the intended amount of a program, and if such adaptive intervention plans are acceptable to the caregivers themselves. Although limited by a small sample size, the present study found evidence of fidelity of implementation and relatively high degrees of acceptability and satisfaction among caregivers with the amount and type of program they received.

Counselors’ reports immediately following the sessions indicated that they successfully differentiated the intervention plans for caregivers by using more techniques associated with a particular domain and giving more emphasis to that domain when the assigned level of a program component was higher. Conversely, when the level of intensity for an intervention domain was assigned to be low or medium, counselors appropriately did not cover as much of the material. This differentiation was confirmed by ratings of session recordings. The exception to this pattern was Domain 1, Care Demands, which our results showed was emphasized equally across the two higher levels of intensity. Counselors may have felt that behavior problems and other care demands required more attention and monitoring, regardless of what the intervention plan indicated. It may also be that counselors were not able to spend as much time as needed for caregivers with high levels of behavior problems, since they had to attend to the other program domains.

Even though counselors were able to differentiate successfully the level of program coverage to each caregiver, the assigned levels of intervention were not significantly related to caregivers’ evaluation of acceptability, specifically, their ratings of counselors’ competence or of the helpfulness of specific program components. These results are similar to findings at the outset of the sessions, where caregivers’ ratings of acceptability did not vary by their assigned level of stress or plans for the intervention. Prior research has suggested that the number of sessions might be associated with participants’ ratings of competence of their counselors (Simons et al., 2010), but that was not the case in the present study. It may be that utilizing an adaptive intervention plan that provides people with the knowledge and skills that they need (and not what they do not need) leads to high ratings of counselor competence even when only a few sessions are delivered. We did, however, expect that caregivers would rate an intervention strategy more helpful when they received more of that specific component. Again, it may be that matching the intervention plan to the needs of the individual will lead to a perception that the program was helpful, regardless of the intensity of the program.

The intensity of the program that was delivered also reflects a caregiver’s initial stress levels. Although confounded with program intensity, initial stress levels did not affect initial ratings of acceptability of the intervention. Other factors might have affected acceptability of the program, including caregiver and care receiver characteristics and exposure to specific types of stressors. The small sample size, however, precluded a more thorough investigation of individual differences in acceptability.

The findings also point to the reliability of counselors’ reports of their use of various strategies within sessions. Although ratings of session recordings can provide unique evidence about program fidelity and process, it can be cumbersome and costly to implement. The addition of ratings by caregivers following individual sessions would complement the counselors’ ratings and provide a rich set of observations for assessing fidelity.

The present study has some limitations that must be considered in interpreting the results. First, it is difficult to generalize the results because of the small sample size. The small sample also limited our ability to examine how different subgroups of caregivers responded to the intervention. Notably, kin relationship has previously been found to affect outcomes of psychosocial interventions with caregivers (e.g., Gitlin et al., 2003) and could also have an effect on fidelity and acceptability. Second, it was not possible to record all the program sessions. Therefore, we cannot rule out if concordance of ratings between counselors and raters differed for recorded sessions compared to non-recorded sessions. Third, we could have included other intervention domains as part of the overall program. Some caregivers, for example, noted that they would have liked a discussion of spirituality as it applied to their caregiving role. Fourth, one of the measures of program adherence had only minimally acceptable reliability. Lastly, the present study was not able to examine the relationship between program adherence and program outcomes. Designed for the purposes of program development, the present study did not utilize a randomized design and did not have enough statistical power to detect treatment and control group differences, nor to consider differences in outcome related to the skill or competence of individual counselors. Further studies investigating adherence-outcome and competence-outcome relations would provide a more powerful test of determining the effectiveness of adaptive intervention for dementia caregivers.

In sum, the study provided demonstrable evidence that an adaptive intervention approach was acceptable to caregivers and could be implemented by counselors with an appropriate level of differentiation. Confirming the acceptability and fidelity of an adaptive approach is a valuable step toward examining the efficacy of these types of intervention. Establishing these criteria holds future promise that short and long-term benefits of an adaptive intervention for caregivers can be assessed in a scientifically rigorous manner.