Judging Outcomes in Psychosocial Interventions for Dementia Caregivers: The Problem of Treatment Implementation

Abstract

Purpose

In published dementia caregiver intervention research, there is widespread failure to measure the level at which treatment was implemented as intended, thereby introducing threats to internal and external validity. The purpose of this article is to discuss the importance of inducing and assessing treatment implementation (TI) strategies in caregiving trials and to propose Lichstein’s TI model as a potential guide.

Design and Methods

The efforts of a large cooperative research study of caregiving interventions, Resources for Enhancing Alzheimer’s Caregiver Health (REACH), illustrates induction and assessment of the three components of TI: delivery, receipt, and enactment.

Results

the approaches taken in REACH vary with the intervention protocols and include using treatment manuals, training and certification of interventionists, and continuous monitoring of actual implementation.

Implications

Investigation and description of treatment process variables allows researchers to understand which aspects of the intervention are responsible for therapeutic change, potentially resulting in development of more efficacious and efficient interventions.

It is estimated that 4 million Americans suffer from Alzheimer’s disease and related disorders (ADRD), with 80% of these individuals living at home and being cared for by family caregivers (Czaja, Eisdorfer, & Schulz, 2000). The number of people with ADRD is expected to grow exponentially as the U.S. population ages (Schulz & O’Brien, 1994). Potential stressors associated with family caregiving are numerous and can include managing behavioral disturbances, attending to physical needs, and providing seemingly constant vigilance (Gold et al., 1995; Vitaliano, Russo, Young, Teri, & Maiuro, 1991; Wright, Clipp, & George, 1993). The effects of these stressors on family caregivers can be catastrophic. Family caregiving has been associated with increased levels of depression and anxiety as well as higher use of psychoactive medications, poorer self-reported physical health, compromised immune function, and increased mortality (Light, Niederehe, & Lebowitz, 1994; Schulz & Beach, 1999; Schulz, O’Brien, Bookwala, & Fleissner, 1995).

Over the past 20 years, researchers have examined a plethora of psychosocial interventions aimed at alleviating distress associated with dementia caregiving. Intervention programs have included intensive personalized counseling, supportive group counseling, providing knowledge about ADRD through educational programs, specific therapeutic skills training, enhancing problem-solving skills, and teaching behavior management techniques (see reviews by Bourgeois, Schulz, & Burgio, 1996; Kennet, Burgio, & Schulz, 2000; Knight, Lutzky, & Macofsky-Urban, 1993; Toseland & Rossiter, 1989; Zarit & Teri, 1992). The conclusions from these reviews are varied. However, there is a growing consensus in the literature that interventions that are comprehensive, intensive, and individually tailored to caregivers’ needs are likely to be more effective than those lacking these characteristics (Haley, Ehrbar, & Schonwetter, 1997; Kennet et al., 2000; Mittelman, Ferris, Shulman, Steinberg, & Levin, 1996).

Another conclusion of these reviews is that existing dementia caregiving intervention studies are fraught with methodological weaknesses that present threats to their internal validity, that is, their ability to infer that the intervention was related to therapeutic changes in caregiver and care recipient outcomes. These deficiencies include, but are not limited to, the use of weak experimental designs, problematic sampling and randomization strategies, and the use of inadequate outcome measures. In their review, Bourgeois and colleagues (1996) focused on the importance of describing treatment process variables and the lack of attention caregiving intervention researchers have devoted to describing these variables.

The investigation and description of treatment process variables allows researchers to understand which aspects of the intervention are responsible for therapeutic change; this, in turn, can result in the development of more efficacious and efficient interventions. Much has been written over the past two decades on the therapeutic process, particularly surrounding psychotherapy outcome research. Early writings by Cook and Campbell (1979) and Sechrest and colleagues (Sechrest, West, Phillips, Redner, & Yeaton, 1979; Sechrest & Yeaton, 1981) helped focus researchers’ attention on critical process variables such as the quality of the therapeutic relationship (therapeutic alliance), the level of therapist skill, and treatment dosage. However, the sine qua non of investigating treatment process variables is the description in precise terms of what the intervention involved (i.e., the treatment components) and establishing through empirical means that the intervention was presented as intended. This is commonly referred to as establishing treatment integrity. In the absence of assessing treatment integrity, conclusions regarding treatment efficacy cannot be made with any confidence (Sechrest et al., 1979; Sechrest & Yeaton, 1981).

However, over the past decade clinical researchers have come to recognize that establishing treatment integrity (i.e., that the treatment was delivered as intended), though necessary, is insufficient for asserting that an unbiased test of the treatment was conducted. The participant’s mastery of treatment (termed receipt) and the participant’s application of treatment beyond the boundaries of the therapy session (termed enactment) are no less critical (Lichstein, Riedel, & Grieve, 1994). Thus, the path of the independent variable (treatment) has been partitioned into three components—delivery, receipt, and enactment—and their summative impact on the client has been termed treatment implementation (TI).

The purpose of this article is to discuss the importance of assessing and reporting treatment implementation in dementia caregiver intervention research. Although we focus on caregiver interventions, these issues have relevance to all geriatric intervention research (e.g., comprehensive geriatric assessment, psychosocial nursing home interventions). We propose Lichstein’s TI model (Lichstein et al., 1994) as a means of conducting unbiased tests of intervention efficacy, and we describe the efforts of the Resources for Enhancing Alzheimer’s Caregiver Health (REACH) cooperative group to apply this model in ongoing caregiver intervention trials. Lichstein’s approach was chosen because it is the only existing system for evaluating treatment along its entire path from presentation to the client to the client’s use of treatment in the natural environment.

Treatment Implementation

TI strategies are used to facilitate and monitor activities between two actors, the interventionist and the client, so that the action of an intervention can be understood. Typically, the term treatment implementation refers to a class of strategies to document the implementation of individual treatment components.

TI strategies can be classified according to three fundamental aspects of intervention application: delivery, receipt, and enactment (Lichstein et al., 1994). Treatment delivery targets the actions of the interventionist, specifically his or her ability to present the intervention to the client as intended. The focus here is on the therapist’s therapeutic skills and his or her ability to engage the client in the treatment protocols. Thus, delivery is concerned with including all intended parts of the treatment, but also with excluding inadvertent introduction of parts of other treatments.

Treatment receipt refers to the degree to which the client actually received the intended treatment, as indicated by mastery of concepts and/or skill development. The focus of treatment receipt is on the client’s actions, for example, the extent of the client’s knowledge or skill level in the area being targeted.

Treatment enactment targets the degree to which the client demonstrates changes in therapeutic behaviors related to the intervention in the natural environment, that is, does the client use appropriately, in his or her daily life, the skills and knowledge that define the particular intervention? In theory, enactment should be closely associated with desired treatment outcomes, but it is important to note that enactment indicators are distinct from the primary outcomes of the study.

Lichstein and associates (1994) differentiated between strategies that are used to induct and those used to assess TI. Induction refers to the methods researchers use to enhance the probability that proper TI occurs. Assessment refers to either quantitative or qualitative measurement of their occurrence. This distinction is important because clinical researchers can use several TI strategies but might assess only a portion of those strategies (Lichstein et al., 1994).

TI Induction

Formal and informal induction methods for delivery, receipt, and enactment should be instituted on an ongoing basis to help ensure appropriate implementation of the intervention. One mechanism for ensuring appropriate and accurate delivery is the use of a detailed written treatment protocol. Other induction procedures include didactic instruction, role playing, performance feedback, and discussions of each of the interventions and how they differ during structured therapist training and recalibration sessions.

Although delivery induction is aimed at improving interventionist skills, receipt and enactment induction focus on encouraging the client’s understanding and performance of therapeutic skills. Receipt and enactment induction methods include instructions and reminders by the interventionist to use the treatment components. Other methods include written materials related to the intervention, role playing, and performance feedback provided to the client during therapeutic sessions.

TI Assessment

Unlike data obtained from study outcome measures, information from TI assessments is collected continuously and is inspected and interpreted as part of the ongoing intervention. Direct measures of client performance of intervention components yield more reliable judgments than indirect measures but are generally more difficult to obtain. Direct measures are more common in delivery and receipt assessments, whereas enactment assessment often relies more strongly on indirect assessment.

Delivery assessment focuses on the skills of the interventionist and his or her ability to deliver the intervention as intended, without additions, and within the amount of time allotted for the intervention. Direct measures of delivery include the frequency, format, and content of all interactions between interventionist and client. One standard methodology for directly assessing delivery is to specify the components of the intervention that are intended, along with plausible confounding components that should not occur, and rating the interventionist on the basis of the occurrence of each. One method for assessing delivery is to obtain independent ratings of an intervention session, using audio- or videotapes or an observer.

Treatment receipt is often assessed by documenting the client’s knowledge or skill level, frequently through the use of pen-and-paper surveys or questionnaires. However, for clients who have low literacy, some direct methods of assessing intervention receipt, such as written tests of knowledge or understanding, may be burdensome or impossible. Other direct measures of receipt that are less obtrusive than written measures include asking the client to demonstrate a skill or asking him or her to recall intervention suggestions. Two “soft” indirect measures of treatment receipt, often used as the sole measures of receipt, are to confirm that the client has the intervention materials in his or her possession and to ask if he or she has any questions about the intervention. Direct and indirect measures of treatment receipt can be scored by the interventionist or by an independent rater, again using tapes or observation.

Assessment of enactment is more difficult but critical to establishing the internal validity of an intervention. Written self-reports of client use of therapeutic components provide a direct assessment of enactment; however, as in receipt assessment, these may prove difficult for clients with low literacy. Indirect assessment of enactment is more common and can include questioning the client regarding the use of intervention techniques. Enactment assessment is more commonly completed by the interventionist but can be scored by an independent observer.

TI assessment provides information concerning the process of treatment as the study unfolds. It provides critical information to the investigator regarding the current state of the intervention protocol and creates the opportunity for corrective action if the interventionist “drifts” from the treatment protocol. Such assessments can also be useful in determining why an intervention did or did not achieve the desired outcomes with specific individuals.

Strategies to induce and assess treatment delivery, receipt, and enactment must be customized to the specific intervention. In the remainder of the article we use examples from the six REACH intervention projects to illustrate several approaches, as well as complications, to TI induction and assessment. However, before we describe specific TI strategies, we present a brief general description of the REACH cooperative effort. A more detailed description can be found in Coon, Schulz, and Ory (1999).

REACH

REACH is a unique, 5-year program sponsored by the National Institute on Aging and the National Institute for Nursing Research. REACH grew out of a National Institutes of Health initiative that acknowledged the well-documented burdens associated with dementia caregiving, as well as the emergence in the literature of promising dementia caregiver interventions. In 1995, NIH funded six intervention sites and a coordinating center to develop interventions for family caregivers of individuals at the moderate level of impairment (see Appendix). Each site investigated a different intervention, although there were commonalities in the treatment components used across sites. The sites investigated the following Alzheimer’s disease (AD) caregiver interventions: (a) home-based behavioral skills training and problem solving (Birmingham/Tuscaloosa), (b) telephone-based system offering voice mail support and advice (Boston), (c) training in behavior and stress management provided through a primary care setting (Memphis), (d) home-based, family-focused multisystem intervention with computer-telephone integration (Miami), (e) group-based coping skills training and enhanced support (Palo Alto), and (f) home-based environmental and behavioral skills training (Philadelphia). Because the caregiving experience in ethnic minority families has been particularly neglected in the field, all of the sites included substantial minority participation. The cooperative agreement fostered the development of standardized outcome measures to assess the impact of comparable strategies on caregivers and their care recipients and the creation of a common database to help compare the efficacy of these different interventions across the range of identified populations.

REACH TI Strategies

Strategies to Induce and Assess Treatment Delivery

Accurate and consistent delivery of the intended treatment is critical to intervention effectiveness as well as to the interpretation of both significant and null findings. Although few will disagree with this fundamental statement, mechanisms to ensure the accurate and consistent delivery of a treatment protocol can be difficult to achieve and are often omitted by investigators. Common induction and assessment strategies used in the REACH projects were designed to combat potential threats to delivering treatment as intended. Table 1 shows some of the induction and assessment strategies used in the initiative. Many of the strategies used in REACH served both induction and assessment functions.

Table 1.

Examples of Resources for Enhancing Alzheimer’s Caregiver Health Treatment Implementation Strategies

Treatment Implementation Component	Induction Strategy	Assessment Strategy
Treatment Delivery
Treatment manuals	X
Structured training	X
Supervisory monitoring and feedback	X
Interventionist certification		X
Delivery checklists used by therapist	X	X
Delivery and accuracy checklists used by supervisors	X	X
Treatment Receipt
Record of contacts and system utilization	X	X
Assessing caregiver knowledge of key treatment concepts and skills	X	X
Interventionist documentation	X	X
Feedback from caregiver		X
Treatment Enactment
Direct observation of the caregiver	X	X
Caregiver self-report	X	X
Interventionist documentation	X	X

There are multiple threats to consistency of inducing treatment delivery, particularly in large, complex intervention trials. For example, all of the REACH interventions included multiple treatment components. Ensuring consistent, accurate application by an interventionist of a single component intervention (e.g., imparting knowledge about ADRD) was simpler than an intervention involving knowledge plus behavioral skills training. Similarly, the complexities of treatment delivery assessment multiply as a function of the number of interventions being compared in a multisite trial. Length of intervention is also a factor. Some of the REACH sites extended their intervention phase over 12 months or longer. It is not uncommon for interventionists to drift from intervention protocol when lengthy intervention phases are used (Moncher & Prinz, 1991). Moreover, although the use of multiple interventionists is advisable to control for “extratherapeutic” factors (e.g., therapist personality), therapist attrition during multiyear therapy trials presented complications for ensuring consistent treatment delivery throughout the trial.

Strategies to induce (i.e., enhance) and assess treatment delivery included (a) using treatment manuals, (b) interventionist training and certification, and (c) providing monitoring and feedback on performance. Each is discussed below.

Treatment Manuals

To guarantee a consistent level of accuracy in treatment delivery, all REACH interventions were guided by detailed treatment manuals. Each of the manuals was examined by the coordinating center for consistency across sites in format and level of detail. These extensive manuals described all aspects of treatment delivery and assessment. Manuals were used as training tools and to maintain accurate delivery over time. Interventionists were given a treatment manual that provided a detailed account of each treatment component and a step-by-step timeline to ensure timely delivery of all intervention activities. In many of the sites, therapists also carried into each session treatment delivery checklists to remind them of the necessary treatment components. Manuals were also a convenient and accurate source of guidance regarding special circumstances that might occur during intervention.

Structured Training and Certification

Methodical interventionist training is a critical step to ensure accurate delivery of treatment. A formal procedure for training interventionists was developed by the investigators at each site. Training consisted of independent readings, didactic instruction, hands-on demonstration, and performance feedback to enable interventionists to implement treatment with AD caregivers. Training was followed by an evaluation procedure, supervised by the coordinating center, that certified the individual to serve as an interventionist. Interventionists were provided with certificates acknowledging that they had acquired the skills necessary for delivering the intervention.

Ongoing Monitoring and Feedback

Although certification procedures confirmed that interventionists had a standardized level of expertise prior to delivering interventions, ongoing monitoring of the interventionists’ performance during the trial was critical. Thus, all REACH sites conducted periodic assessments of the interventionists’ performance. This was accomplished largely by audiotaping interactions with caregivers, either at every therapeutic session or on a random basis. These audiotaped interactions were coded by an individual at the site who was knowledgeable about the intervention to ensure accurate and consistent adherence with protocol. Coding of each interventionist’s performance was guided by a treatment delivery and accuracy checklist on which the coder rated the interventionist’s performance according to previously identified key treatment components. The completed checklist was then used to provide feedback to the interventionist on accuracy of performance. Thus, both the therapist and the supervisory checklists were used for delivery induction and assessment.

In addition, some REACH sites incorporated the practice of weekly group supervision sessions to allow very careful and consistent feedback and monitoring of their interventionists. This was done particularly at sites where multiple and, at times, rotating interventionists (e.g., psychological interns) were used, to ensure that treatments were delivered in a consistent manner by all involved throughout the life of the project. As an example, three of the sites used a similar “minimal support condition” (MSC). In this condition, therapists contacted caregivers by telephone to offer limited social support; however, only very general therapeutic information was provided. In the MSC, the handling of serious issues (such as possible elder abuse) often required a delicate balance between the constraints of the protocol itself and the ethical mandate to respond appropriately to the problem under discussion. In group supervision, all current interventionists and those in training would discuss possible ways to handle a situation, and a plan of action would be agreed on and followed up on. For those sites that used group supervision, attendance was required for all staff, including those functioning in outlying areas who were connected by phone to the face-to-face weekly group meetings. To increase the consistent application of the MSC protocol across sites, the project managers from each site participated in regular phone conferences where problems in delivery were discussed.

Cultural Diversity and Language Issues in Treatment Delivery

Four of the six REACH sites heavily recruited African American caregivers (Birmingham/Tuscaloosa, Boston, Memphis, and Philadelphia), and two focused on Hispanic/Latino caregivers (Miami and Palo Alto). Diversity issues were often complex and required special attention. At those sites serving the Hispanic/Latino populations, language issues—notably language preference and translation issues—added to the complexity of delivering the REACH interventions as intended. We raise this issue, and describe experiences handling it, to alert other researchers to some important considerations as it is anticipated that in the future more intervention programs will be designed for diverse cultural and ethnic groups.

Language preference and translation issues go hand in hand: interventions cannot be delivered in the caregivers’ language of choice unless suitable translations of the material are available, along with bilingual (and preferably bicultural) staff to offer the programs. Because two of the six sites worked with Spanish-speaking caregivers (in Miami, Cuban Americans were the majority of Hispanics seen; in Palo Alto, Mexican Americans were the majority), efforts were made to coordinate translation processes across these sites.

The multiple translation processes were lengthy, complex, expensive, and often frustrating for the investigators for a variety of reasons. First, accepted practices of forward and back translation needed to be implemented to arrive at a consensus regarding the meaning and intent of the treatment protocols. A professional translation company had to be used first for the forward translations, to get them into “generic Spanish” that would provide a culturally appropriate starting point. Then panels of bilingual and bicultural Hispanics representing different Hispanic subgroups were convened to do the back translations. Discrepancies in meaning had to be discussed until consensus was achieved. Some use of idioms (which vary regionally in their meaning) was permitted to facilitate accurate comprehension, but this was kept to a minimum.

Overall, the process took about a year to accomplish. Second, once these translations had been accomplished effectively, interventionists who were bilingual (and in most instances, bicultural) had to be selected and trained to use the manuals and the treatment delivery assessment forms. These were relatively unfamiliar concepts for most of the staff at these two sites: Even those with a significant background in the social sciences were unaccustomed to following very detailed protocols when interacting with people in distress. Third, many culturally sensitive issues arose over the course of time that needed to be addressed by the interventionists. For example, depression was very common among the caregivers enrolled at both Spanish-speaking sites, and at times it was present to a clinically significant degree. However, management of significant clinical depression was beyond the scope of the REACH protocols. Therefore, the interventionists had to be trained to locate appropriate referral sources, make the actual referral, and follow up to encourage the caregiver to accept the referral.

In summary, providing standardized, manual-guided interventions in languages other than English poses challenges in treatment delivery, specifically in adequacy of translation, effective training and supervision of staff, and handling of off-protocol topics and situations. Despite the challenges involved, however, this type of work will be more and more common in the future, particularly in areas of the United States that have high immigrant populations who often do not speak English well enough to communicate their distress and benefit from help that is not delivered in their native language.

Strategies to Induce and Assess Treatment Receipt

Even if treatment is delivered by the interventionist in exemplary fashion, the investigator should not assume that the caregiver has received the intervention as intended. Numerous threats to treatment receipt are present in caregiver intervention research. For example, burdened caregivers might be distracted and inattentive during training sessions due to events in their environment. Also, older adult caregivers might have diminished hearing, vision, or memory abilities that can hamper their learning. The REACH cooperative group induced and assessed treatment receipt through various methods, including maintaining a record of intervention contacts, assessing caregiver knowledge, documenting intervention sessions, and eliciting caregiver feedback. Each is discussed in more detail below.

Record of Contacts and System Utilization

A minimal requirement for any intervention is an accurate record of the frequency, duration, and types of contact between interventionist and client. This information is essential for assessing both treatment delivery and receipt. At all REACH sites, a standard form was used to systematically document information related to contact with caregivers. This information included the number of contacts, duration, and method (e.g., telephone, face-to-face, group, access to computerized information system). The form was completed by the staff member involved in the contact; this individual also documented whether the contact was scheduled or unscheduled, who initiated the contact, who was involved in the contact (e.g., other family members, other professionals, the care recipient), and if the contact was “off protocol.” This information was immediately entered into the REACH core database by data entry staff. A record of contacts allowed investigators to analyze outcomes based on type, number, and duration of contacts. It also allowed the supervisor to provide feedback to the interventionists about the degree to which the intervention was delivered to and received by the client.

It is important to note here the potential for overlap and the interface between treatment delivery and receipt. For example, interventionists in sites using the MSC were asked to engage the caregivers in phone conversations lasting 15 min. If the contact of record form showed that conversations were consistently much briefer in duration, the supervisor asked the interventionist to redouble his or her efforts to lengthen the conversations (delivery); however, the caregiver’s willingness to engage in conversation also influenced the duration of contact (receipt). This example also shows how a TI strategy can serve both induction and assessment functions. The recording process provided data on treatment delivery and receipt; however, supervisory feedback based on the data induced the interventionist to adhere more closely to the protocol.

Assessing Caregiver Knowledge of Key Treatment Concepts and Skills

It is important to assess changes in the caregiver’s knowledge of the key concepts and skills involved in the targeted intervention. Some of the sites that included didactic caregiver instruction in group or face-to-face format used a formal pre–post knowledge test. Most of the sites achieved this goal by audiotaping intervention contacts. Audiotaped intervention contacts were scored for caregiver understanding of fundamental knowledge of therapeutic behavior using a standard form.

Interventionist Documentation

A rich source of information about treatment receipt in REACH was the interventionists’ field notes and documentation. Both qualitative and quantitative methods were used by the REACH sites and included written summaries of dementia management strategies developed during therapy sessions and documentation of behavioral or environmental problems addressed during treatment. Supervisory feedback from both the knowledge assessment and interventionist documentation was used to induce as well as assess treatment receipt.

Feedback From the Caregiver

To collect information about caregiver perspectives on the intervention, all REACH sites used a standard 17-item survey. The questions asked the caregivers to rate their experiences in several topical areas, including education about dementia, caregiver skill building, and perceived benefit of the intervention. In addition, each site included up to 10 site-specific questions to gain information about topics that were unique to their intervention. For instance, 2 of the sites used new technology to deliver the treatment; these sites asked caregivers to rate their experiences of learning to use the system and frequency of problems with the system. The survey was administered to caregivers after the primary endpoint, the 12-month assessment. Thus, the information was used to assess but not induce treatment receipt. Use of this standard form directly accessed information about the caregivers’ level of treatment receipt and provided a mechanism for comparing receipt across sites.

Strategies to Induce and Assess Treatment Enactment

Strategies for inducing and assessing treatment enactment are designed to assess the level at which caregivers actually use the knowledge and skills acquired in treatment and apply these skills to situations outside of the therapy session. Enactment suggests that mechanisms of change are in operation. It is preferable to assess enactment by collecting data from various perspectives. Specifically, combining direct observations with reports from both the caregivers and the interventionists is most advantageous. In the REACH initiative, these strategies were used to both assess and induce treatment enactment.

Direct Observation of the Caregiver

Observation of caregivers is the most direct and reliable means of determining the level of treatment enactment. However, for some interventions, such as home-based treatments, this method might also be costly for the researchers and burdensome for the caregiver. At the Alabama site, one of the goals of intervention was to improve caregiver communication skills. Subsequently, all caregiver-care recipient dyads were asked to participate in structured, staged social activities. Each dyadic interaction was videotaped for a total of 90 min, 30 min during baseline followed by four 15-min observations completed over the first 10 weeks of the intervention. Specific feedback was provided to the caregivers at the end of each session regarding communication skills used during the session. Project staff then completed a detailed assessment of these videotaped social interactions. Observing the caregiver’s behaviors during these staged social activities provided an opportunity to examine the effects of communication skill training on the caregiver’s verbal and nonverbal social behaviors.

Group and office-based interventions provide greater opportunities for observation of caregivers, and several REACH sites used this method. Intervention enactment during the sessions was documented on a checklist or rating scale by research staff. At the Philadelphia site, which focused on environmental redesign, caregivers consulted with the interventionist to decide which adaptive equipment addressed their management issues. The purchase of the adaptive equipment was then recorded on a log sheet. Information from these methods for assessing enactment was augmented by interventionist and caregiver reports, as described below.

Caregiver Self-Report

Investigators can use caregiver self-reports to collect data regarding which aspects of the intervention are enacted and the frequency with which they are used. Self-reports can also provide valuable information regarding the barriers to enactment and the length of time that strategies are effective (this is particularly important in the case of a progressive condition such as dementia). Consequently, the REACH investigators used several types of self-reports. These included caregivers’ reports concerning frequency of strategy use, ability to generalize skills and knowledge to newly emerging situations, and number of weeks that a strategy is in use. The use of self-report forms allows an opportunity for caregivers to reflect on the benefits or consequences of the intervention, to make suggestions for changes that are allowed within the protocol (i.e., individualizing the intervention), and to comment on the usefulness of specific components of the treatment.

Interventionist Documentation

Assessing the interventionists’ perspectives on enactment may be as simple as obtaining information similar to clients’ self-reported use of interventions or as complex as a thematic analysis of field notes. REACH sites used various strategies, including the interventionists’ record of caregiver adherence, rating of intervention effectiveness, and progress notes documenting use of treatment strategies. Interventionists at several sites also documented the degree to which caregivers used knowledge and skills to address newly emerging caregiving issues.

Summary and Conclusions

Dementia caregiving research has progressed from investigations of factors contributing to the stress of caregiving to the development of interventions to alleviate these stressors. Generally, reviews of the efficacy of these interventions have been equivocal, although comprehensive, intensive, and individually tailored interventions appear to be more efficacious than those lacking these characteristics. One problem with existing intervention studies is that the issue of treatment implementation has largely been ignored, making it difficult to interpret findings and draw conclusions regarding treatment efficacy.

This article discussed some of the efforts undertaken in the NIH-funded REACH initiative to induce and assess TI. The approaches taken by the research sites vary with the intervention protocols and include using treatment manuals, training and certification of interventionists, and continuous monitoring of actual implementation by audiotaping therapist interactions with caregivers. A unique feature of REACH is that the approaches are based on Lichstein’s TI model, which captures three fundamental aspects of TI: delivery, receipt, and enactment (Lichstein et al., 1994). Few caregiver intervention studies have reported systematic methods for evaluating treatment along its entire path from presentation to the caregiver to the caregiver’s use of the treatment in the natural environment.

Various strategies for inducing and assessing treatment delivery, receipt, and enactment have been discussed and are available for use by researchers investigating psychosocial interventions. The examples used from the REACH initiative highlight the necessity of adapting TI strategies to specific interventions and make apparent the potential complexities involved in their application. The amount of complexity is dependent, in part, on the type of intervention used. For example, tracking implementation of computer- or telephone-assisted information and support can be done easily by programming these devices to record usage. Conversely, obtaining quantitative data on caregiver skill usage in the home can be quite challenging, forcing the researcher to infer usage from multiple “softer” measures such as caregiver self-report and interventionists’ impressions.

As shown in Table 1, strategies can be used solely to induce or teach the systematic implementation of an intervention (e.g., treatment manuals) or they can be used solely to assess accurate implementation (e.g., interventionist certification), but most can serve a dual function of inducing and assessing TI. For example, audiotapes from a treatment session will provide information on the therapist’s use of intended treatment components (assessment); however, the extent to which this information is used to induce the therapist to deliver the intervention more in line with the intended treatment depends on the study’s basic aims. The REACH projects are efficacy trials wherein a specific aim is to clarify whether novel interventions are feasible and whether specific therapeutic activities are related to beneficial changes in caregiver outcomes. Thus, there is an emphasis within the initiative on both assessing TI and frequently using that information to induce the intended treatment.

All intervention studies fall somewhere along the efficacy–effectiveness continuum. Effectiveness trials are designed to evaluate the use of interventions in the “real world” after efficacy has been established. An extreme example of an effectiveness trial would involve the evaluation of a manual-guided psychosocial treatment intended for use by therapists in existing community mental health clinics. In such a trial, ongoing induction techniques would be present, but deemphasized. In essence, TI becomes a major outcome variable in effectiveness trials.

There are three issues pertaining to the generalizability of interventions incorporating TI strategies that require discussion. First, the process of inducing and assessing TI can and often does produce reactivity in the individual receiving the intervention. Thus, induction and assessment techniques become a part of the intervention itself. It is incumbent on intervention researchers to acknowledge this point when describing their methods and procedures. Second, although the use of induction and assessment techniques is essential, the specific techniques selected can have profound implications for the feasibility of implementing the intervention in the real world. For example, videotaping client behavior in the home, although an excellent enactment assessment strategy, may be too intrusive for some types of interventions and may even undermine intervention goals. In general, it is preferable in naturalistic settings to use strategies that are unobtrusive and maximally feasible. Finally, although we have stated that TI strategies are often deemphasized in effectiveness trials, if induction and assessment techniques are neglected this can have serious implications for the effectiveness of the trial and/or the researcher’s ability to draw any conclusions from the study outcome.

The availability of TI data will ultimately allow us to have greater insight into findings regarding the efficacy and effectiveness of caregiver interventions. TI data from the REACH initiative will provide information regarding needed modifications in future treatment protocols. If we find, for example, that caregivers have difficulty comprehending a particular aspect of a treatment protocol it would suggest that this component of the protocol requires modification. TI data should also yield valuable information regarding future requirements for interventionist training. Finally, we are planning to assess the impact of variations in TI on targeted outcome measures. If there are differences between actual and intended dosage in the various treatments, we can evaluate the impact of these differences on relevant outcomes (see Henggeler, Melton, Brondino, Scherer, & Hanley, 1997, and Huey, Henggeler, Brondino, & Pickrel, 2000, for examples of clinical studies that include TI data in their analyses).

The importance of assessing TI in clinical studies cannot be overstated. Policymakers, researchers, and study participants pay high costs in time and money to develop and test interventions. Without an accurate assessment of whether the intervention was delivered as intended, received by the client, and enacted by the client, conclusions regarding treatment outcomes are suspect. The intervention that was described may not be the intervention that made the difference in caregivers’ lives, or interventions that hold great promise for alleviating caregivers’ burdens may be discarded altogether.

Appendix

REACH Research Group —Participating Institutions and Principal Staff

University of Alabama (Tuscaloosa and Birmingham, AL): Louis Burgio, PhD (Principal Investigator—Tuscaloosa), Alan Stevens, PhD (Principal Investigator—Birmingham), Alfred Bartolucci, PhD, Delois Guy, PhD, William Haley, PhD, David Roth, PhD, David Vance, PhD

Hebrew Rehabilitation Center for Aged Research and Training Institute (Boston, MA): Diane Mahoney, PhD (Principal Investigator), Robert Friedman, MD, Brooke Harrow, PhD, Timothy Heeren, PhD (former participant), Ting Lin, PhD, Barbara Tarlow, PhD, Sharon Tennstedt, PhD, Ladislav Volicer, MD, PhD

University of Tennessee, Memphis (Memphis, TN): Robert Burns, MD (Principal Investigator), Marshall Graney, PhD, Kenneth Lichstein, PhD, Jennifer Martindale-Adams, EdD, Linda Nichols, PhD, Grant Somes, PhD

University of Miami (Miami, FL): Carl Eisdorfer, MD, PhD (Principal Investigator), Soledad Arguelles, PhD, Trinidad Arguelles, MS, Sara Czaja, PhD, David Loewenstein, PhD, Mark Rubert, PhD, Jose Szapocznik, PhD

Veterans Affairs Medical Center (Palo Alto, CA): Dolores Gallagher-Thompson, PhD (Principal Investigator), David Coon, PhD, Helena Kraemer, PhD, Ana Menendez, MS, Larry Thompson, PhD

Thomas Jefferson University (Philadelphia, PA): Laura N. Gitlin, PhD (Principal Investigator), Mary Corcoran, PhD, Susan Klein, PhD, Sue Marcus, PhD, Laraine Winter, PhD

Project Office, National Institutes of Health: Marcia Ory, PhD, Mary Leveck, PhD

Coordinating Center, University of Pittsburgh (Pittsburgh, PA): Richard Schulz, PhD (Principal Investigator), Steven H. Belle, PhD, Joy Herrington, MEd, Jason Newsom, PhD (former participant), Galen Switzer, PhD (former participant), Stephen R. Wisniewski, PhD

External Advisory Committee: Patricia Archbold, DNSC, Oregon Health Sciences University; Larry Beutler, PhD, University of California, Santa Barbara; Joel Greenhouse, PhD, Carnegie Mellon University; J. Neil Henderson, PhD, University of South Florida; Ira Katz, MD, PhD, University of Pennsylvania; Powell Lawton, PhD, Philadelphia Geriatric Center; Len Pearlin, PhD, University of Maryland; May Wykle, PhD, Case Western Reserve University