Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Mar 27.
Published in final edited form as: Am J Phys Med Rehabil. 2012 Aug;91(8):715–724. doi: 10.1097/PHM.0b013e31824ad462

Measuring Treatment Fidelity in a Rehabilitation Intervention Study

Mary W Hildebrand 1, Helen H Host 2,3, Ellen F Binder 3, Brian Carpenter 4, Kenneth E Freedland 5, Nancy Morrow-Howell 6, Carolyn M Baum 7, Peter Doré 1,6, Eric J Lenze 5
PMCID: PMC3967862  NIHMSID: NIHMS448678  PMID: 22377824

Abstract

Attaining and demonstrating treatment fidelity is critical in the development and testing of evidence-based interventions. Treatment fidelity refers to the extent to which an intervention was implemented in clinical testing as it was conceptualized and is clearly differentiable from control or standard-of-care interventions. In clinical research treatment fidelity is typically attained by intensive training and supervision techniques and demonstrated by measuring therapist adherence and competence to the protocol using external raters. Yet, in occupational therapy (OT) and physical therapy (PT) outcomes research, treatment fidelity methods have not been utilized, which in our view is a serious gap that impedes novel treatment development and testing in these rehabilitation fields. In this article we describe the development of methods to train and supervise therapists to attain adequate treatment fidelity in a treatment development project involving a novel OT and PT-based intervention. We also present a data-driven model for demonstrating therapist adherence and competence in the new treatment and its differentiation from standard-of-care. In doing so, we provide an approach that rehabilitation researchers can use to address treatment fidelity in OT and PT-based interventions. We recommend that all treatment researchers in rehabilitation disciplines use these or similar methods as a vital step in development and testing of evidence-based rehabilitation interventions.

Keywords: Treatment Fidelity, Research Design, Randomized Controlled Trial, Evidence-Based Treatment

Occupational therapy (OT) and physical therapy (PT) are rehabilitation disciplines involving complex patient interaction due to their behavioral and therapist-dependent nature. As a result one of the most important issues in research is the need to define OT and PT interventions, measure their implementation during the research study, demonstrate in research publications that they are being carried out as defined, and demonstrate that they are different from standard-of-care interventions. This issue is known as treatment fidelity. Attaining and demonstrating treatment fidelity in outcomes research enable researchers to isolate the active ingredients of an intervention, determine its efficacy, contrast it with a control or standard treatment, and replicate findings.1,2 This is challenging in OT and PT, as rehabilitative treatments are multi-faceted, involve many different professions, and as such are difficult to quantify.3

As a model for rehabilitation research, Whyte and Hart recommend the model of psychotherapy research in developing and testing theoretically based treatments.4 They observe that the fields of psychotherapy and rehabilitation have analogous challenges in attaining and demonstrating treatment fidelity, since both types of interventions are inherently difficult to quantify and are dependent on the clinician’s delivery and interactions with the patient.4 Despite these challenges, psychotherapy research has come to a consensus for addressing treatment fidelity in clinical trials and as a result, this field has been successful in developing evidence-based treatments in many contexts. Much of this success is owed to methods to attain and demonstrate treatment fidelity, which result in more consistent application of skills and knowledge.5,6 These methods include development of treatment manuals and corresponding scales of fidelity, and therapist training and supervision techniques.2,7–9 The psychotherapy field has come to such a consensus regarding the value and importance of these treatment fidelity methods –manualization of treatment, training and supervision techniques, and measuring treatment fidelity – that a psychotherapeutic intervention will not be considered evidence-based unless published studies have addressed these treatment fidelity issues. 10–12

Similar methods should be feasible in rehabilitation research. For example, OT has historical roots in mental health, and its supervision texts cite psychotherapy supervision models for education of fieldwork students.13 There have been calls for addressing treatment fidelity in OT and PT, as well as other rehabilitation disciplines, 2, 14–21 but to move forward in developing and improving evidence-based OT and PT-based treatments, methods need to be established to attain and demonstrate treatment fidelity for these treatment modalities.

Treatment fidelity comprises two key aspects: 1) treatment integrity, that is, demonstrating that therapists carry out the intervention with adequate levels of adherence and competence to the treatment model or protocol; and 2) treatment differentiation, that is, ensuring that the experimental intervention condition differs from a control condition (i.e., showing much higher adherence and competence to the treatment model).22 Adherence refers to the degree to which therapists deliver the intervention as it is specified in the manual, whereas competence refers to the skill with which these techniques are implemented.23 In order to clearly demonstrate that their interventions attain aspects of treatment fidelity, researchers must collect and examine measurable treatment features such as therapist statements and therapy duration, repetition, or intensity and abstract features such as therapist empathy, demeanor, or style. Such measurement then becomes an important way of not only demonstrating but also providing feedback to attain adequate treatment integrity. Thus, attainment of treatment integrity is a process that begins with development of the treatment protocol, training and supervising therapists to deliver the intervention, measuring their adherence and competence to the protocol, and, finally, using measurement results to provide targeted supervision to therapists to further enhance adherence and competence. Treatment differentiation can be demonstrated by measuring and comparing adherence and competence to the new treatment intervention of both trained therapists and standard-of-care or untrained therapists. 22

Researchers in behavioral medicine have reviewed the use of treatment fidelity methods in outcomes research and found that many studies have not verified the two aspects of treatment fidelity. 24–26 The absence of demonstrating treatment fidelity can undermine internal and external validity of outcomes studies posing major obstacles to the development of new rehabilitation methods. 1,2 Some researchers have reported assessing treatment fidelity in OT and PT interventions. Bovend’Eerdt et al. described how low therapist adherence to a novel PT and OT program of motor imagery to improve functional task performance restricted conclusions researchers could draw on the efficacy of the treatment. 27 Leeuw et al. reported designing and utilizing successful methods for assessing treatment delivery and treatment differentiation in a study of interventions for chronic low back pain. 20

We report here on a treatment development project that provided an opportunity to develop and test methods for training, supervision, and treatment fidelity testing for OT and PT, and discover what could be successfully applied to measure and optimize treatment integrity and demonstrate treatment differentiation. The purpose of this report is to 1) describe the development of training and supervision methods applicable to occupational and physical therapists carrying out rehabilitation research, and 2) present a data-driven model for attaining and demonstrating treatment fidelity.

METHODS

We carried out a treatment development project, “Enhanced Medical Rehabilitation” (EMR), from 2009–2011.28 The project’s goal was to develop an OT and PT-based intervention protocol that provided high-intensity and high-engagement daily therapy to improve depression and functional outcomes for older adults who are in short term skilled nursing facilities (SNF) following a disabling medical event. The present manuscript focuses on optimizing the manual, training and supervision, and fidelity testing. Details of the EMR intervention are available elsewhere.28

Manual Development

Daily OT and PT in EMR were designed to be much more physically intense, more cognitively stimulating, and more emotionally engaging than is standard therapy, thereby overcoming depression-related barriers (e.g., amotivation, passivity) to rehabilitation success. To accomplish this, all study participants’ OT and PT sessions had three interacting components: a patient-directed approach; high intensity activities; and frequent feedback on effort and progress. These interacting components, described further below, were the foci of the manual and subsequently were foci of training and supervision of therapists and of treatment fidelity measurement.

  1. A patient-directed approach may improve patient participation and have antidepressant effects, as is embodied in problem-solving therapy concepts.29–32 Therefore, therapists were taught to have the patient decide which goal to work on and which activities to perform, and elicit the patient’s input on solutions to identified problems.

  2. There is evidence that high intensity OT and PT is better at reducing disability and may provide antidepressant effects because it involves more vigorous activity.33–35 Therefore, therapists were trained to maximize intensity by monitoring intensity, encouraging more rigorous activities, and increasing activity intensity (e.g., add weight or complexity to an activity or exercise).

  3. For patients to benefit maximally from rehabilitation, they must be actively engaged in the process, which requires them to believe that therapy is beneficial and that they are able to achieve therapy goals.31 Therefore, we trained therapists to make frequent comments on the patient’s effort and progress and to review this systematically with the patient.

The research team included faculty members in OT, PT, psychology, geriatric medicine, social work, and psychiatry. The process of defining and refining the components of EMR was undertaken via weekly investigator meetings to agree on the treatment protocol and create a manual. Unlike psychotherapy manuals, the EMR manual was mainly a single page prompt sheet (see Appendix 1), along with a 9-page manual of examples. The prompt sheet (personalized for each patient) listed the key components of the intervention, similar to a checklist used in implementation studies involving infectious disease control or operating room procedures.1,36

Training and Supervision

Four registered and licensed therapists (OT, OT assistant, PT, and PT assistant) employed by a SNF, volunteered to provide the EMR protocol to study participants. The therapists ranged from 29 to 45 years of age with between 5 and 25 years of experience in the SNF setting. Initial training of therapists included discussion of theoretical underpinnings of the protocol with research team members, reading the manual draft, and a one-day training to review specifics of the intervention. The study coordinator videotaped therapy sessions for review by team members. The weekly team meetings focused on evaluating the therapists’ fidelity to the protocol by viewing preselected segments of recordings and on revising the manual as needed. The video reviews served multiple purposes: to highlight therapists’ fidelity successes or failures, to offer feedback to the therapists, and to identify simplifications and clarifications to the manual or procedures. At the conclusion of the pilot study, interviews were conducted with the supervisors and SNF therapists with the aim of uncovering both successful and problematic training and supervision methods.

Treatment Fidelity Measurement

After the research team members came to a consensus on a treatment manual and training and supervision techniques, we developed a checklist to rate treatment adherence and competence that quantified the following behaviors that are consistent with EMR: (1) the frequency with which the therapist performed specific tasks during the session, (2) therapist-patient communication, and (3) competency items (scored as low, medium, or high competence). The research coordinator, supervising OT (MWH), and supervising PT (HHH) developed scoring criteria for each item on the adherence and competence checklist. To measure both treatment integrity and treatment differentiation, we rated a total of 60 therapy sessions: 35 EMR sessions (18 OT, 17 PT) conducted by the four therapists who were trained and supervised in EMR and 25 standard therapy sessions (11 OT, 14 PT) carried out by ten therapists at the same SNF who were not trained in EMR. These observations for fidelity ratings were done weekly at random, approximately 12 months after the therapists’ initial training in EMR and while they continued to receive ongoing supervision. All sessions were videotaped and rated independently by the research coordinator. To determine inter-rater reliability of the treatment fidelity measure, the supervising OT rated all OT sessions and the supervising PT rated all PT sessions, and their ratings were compared to the research coordinator’s ratings of the same session.

RESULTS

Development of Training and Supervision Techniques

One of the goals of the pilot study was to develop robust methods for training and supervision of therapists to ensure accuracy and consistency and minimize “drift”; therefore, these techniques evolved based on early successes and failures. Doctoral-level faculty in OT and PT (MWH and HHH) trained and supervised the SNF therapists, consistent with recommendations in psychotherapy supervision guidelines.8

The first step in training was to distribute the first and ensuing drafts of the treatment manual to the SNF therapists to study and implement. Soon thereafter, the supervisory OT and PT carried out training sessions in which they observed therapy, provided feedback, and modeled new techniques.

Videotaping sessions is not standard in OT and PT supervision (either in research or elsewhere), but it is a central part of psychotherapy supervision.8. Videotaping provides a direct observation of actual therapy practices, and we decided to adapt it for OT and PT in order to provide supervision and carry out treatment fidelity monitoring. We did not video all therapy sessions due to limited resources and feasibility concerns. We randomly chose and videotaped one OT and one PT session per week for each study participant for the research team to generate feedback that the supervising OT and PT provided at weekly meetings with SNF therapists. Audio recordings substituted for videotapes of OT sessions when there were patient privacy concerns during performance of activities of daily living (e.g., bathing and dressing). SNF therapists reported unease with the taping initially but acclimated to it quickly. Research subjects reported no concerns. It was necessary to place a microphone on the OT or PT to hear the therapy techniques clearly, and it was necessary to have a research staff present at all times during the videotaping to ensure that therapy activities were actually caught on tape. One advantage of this was that the research staff could also directly observe the therapy while videotaping and report back to the supervising OT and PT any concerns. The three theoretical components of EMR led the research team to recommend training therapists in specific techniques. We trained therapists to ensure a patient directed approach with techniques such as giving choices of which activities to perform in each session and to reduce directive language (e.g. “This morning in OT, you will work on bathing and dressing yourself.”) We trained therapists to get patients to perform activities at a higher intensity with techniques such as minimizing “down” time and not limiting patients by asking them to perform an arbitrary number of exercise repetitions (e.g. “Do 10 repetitions of biceps curls with this 1 pound weight.”) We trained therapists to give feedback to the patient on effort and progress such as commenting when the patient was working hard enough to achieve a training effect or when activities (e.g. dressing) were getting easier to perform as an indication of progress.

Supervision methods evolved based on the research team’s observations and the understanding that the initial training, by itself, would be insufficient to ensure treatment integrity. Therefore, weekly group meetings (30 minutes) were held for the duration of the pilot study for the four EMR SNF therapists and alternately led by the supervising OT and PT. Weekly meetings were structured and focused on reviewing one or two intervention concepts. We addressed problems that had been identified by the research team (e.g., during videotaping), showed video clips to highlight examples of excellent adherence, performed role-plays to rehearse ways to improve treatment delivery, and devoted a portion of each meeting to therapists’ concerns about current study patients and to problem-solving about barriers to participation in therapy. Qualitatively, the research team and the SNF therapists were satisfied with these techniques.

Additionally, to ensure that feedback was given in a timely and consistent manner, the supervising OT and PT continued weekly direct observations of sessions and gave feedback immediately after these sessions regarding the adequacy of treatment fidelity and any concerns. We also continued to videotape sessions as backup for missed direct observations and for fidelity rating.

Supervision of therapists that is designed to attain high treatment fidelity in a research protocol, is atypical for rehabilitation disciplines. Therefore, throughout the pilot study, the research team (which included several mental health experts) provided oversight and suggestions to the OT and PT supervisors (MWH and HHH) on their supervision techniques. As part of this, the 30 minute weekly therapist supervision meetings were videotaped and reviewed by the lead researcher for 3 months. Thus, we also had “supervision of the supervisors”, in which the team discussed the content and structure of the weekly supervision meetings, reviewed good practices based on psychotherapy supervision and other management principles, and highlighted examples of good supervision techniques. After 3 months, the supervisors were sufficiently comfortable with the supervision techniques that these additional reviews became unnecessary.

Development and Validation of a Treatment Fidelity Measure

A validated adherence and competence measure, in which a member of the research team systematically rates directly-observed therapy sessions using a scale that closely follows the principles of a treatment manual, is considered critical in psychotherapy research.37,38 This scale performs two important tasks in a clinical trial: 1) verification of treatment integrity via a reliable method to measure attainment and maintenance of sufficient levels of protocol adherence and competence, and 2) collection of data for evidence of treatment differentiation from the control treatment.1,39

We developed an objective fidelity measure to measure adherence, in which raters (independent members of the research team) tallied when an EMR principle was observed. After measuring adherence, raters then gave subjective competence ratings with respect to the three treatment components: using a patient-directed approach, high intensity activities, and continuous feedback on effort and progress. Appendix 2 shows the final treatment fidelity measure. The results here focus on the fidelity measurement of observed EMR principles; fidelity data for the missed opportunities and fidelity violations are available upon request.

The treatment fidelity measurement also provides a framework and data for supervision. We were able to use this instrument not only to collect data on therapists’ treatment integrity, but also as a supervision tool to provide feedback to SNF therapists, either reinforcing areas in which they were showing good adherence and competence, or to target further supervision in areas where they were showing inadequate adherence or competence.

To demonstrate the inter-rater reliability of the treatment fidelity measurement, two study investigators independently rated each session with the adherence and competence scale (one investigator rated the session in person, and the other rated a video recording of the session). We measured EMR-trained and supervised SNF therapists’ sessions as well as the standard-of-care comparison group therapists’ sessions (N=60; 29 OT and 31 PT sessions). Because we found that adherence data appeared bimodal, appropriate cutoffs were determined and Kappa statistics were calculated as an index of inter-rater reliability. Kappas ranged from 0.24 to 1.00 on adherence items with acceptable Kappas of 0.70 or greater in 4 of 12 items in OT sessions and 6 of 12 items in PT sessions. When Kappas were low on a particular item, we evaluated its importance to the core treatment principles and determined whether we should retain the item and rewrite it and/or recalibrate our rating criteria to achieve greater inter-rater reliability, or, if the item was deemed less important, leave it out of subsequent rating forms. As an example, our measurement of one item, “ensured patient understood link between activity and goals”, initially had Kappas of 0.62 in PT and 0.49 in OT sessions; this suggested that our measurement of this item required further calibration. Therefore, we re-examined this item, established that it exemplified two core principles (patient-directed approach and feedback on effort and progress) and needed to be retained. We redefined the rating criteria to achieve higher inter-rater reliability (i.e. “Did therapist ask patient to explain how the activity related to goal?”). Similarly, we removed some items from inter-rater reliability measurement because they were too difficult to rate (e.g., “missed opportunity to guide patient to more rigorous activity”) or they did not occur frequently enough for analysis (e.g., “dealt appropriately with a patient’s emotional distress”). In this way, we found this process useful for further defining the intervention itself.

The three competence items were rated on a 1 to 3 scale representing low, medium, and high competence for each of the three core principles of EMR. We achieved an acceptable Kendall’s Tau of 0.70 or greater in 2 of 3 competence items in OT, “patient-directed approach” and “high intensity”, and 3 of 3 in PT. Based on these reliability data, we further refined the assessment of “feedback on effort and progress” in OT but did not otherwise change our measurement.

Testing Treatment Fidelity with the Adherence and Competence Measure

Once the treatment fidelity measure was finalized, we assessed treatment integrity and treatment differentiation by examining the mean number of times a therapist behavior occurred (e.g., “gave feedback on effort”), and mean competence scores in therapy sessions, in both the EMR group and the standard of care group. Figure 1 displays levels of adherence and Figure 2 displays levels of competence on the three EMR core treatment components, for all therapists. As the figures show, sessions conducted by EMR therapists included components of the treatment model at a significantly higher frequency and with significantly higher levels of competence than sessions conducted by standard of care therapists.

FIGURE 1.

FIGURE 1

Open in a new tab

Occupational and physical therapy sessions mean adherence scores for EMR, comparing EMR intervention-trained and supervised therapist sessions (n = 35) to standard of care nontrained therapist sessions (n = 25). EMR, enhanced medical rehabilitation.

FIGURE 2.

FIGURE 2

Open in a new tab

Occupational and physical therapy sessions mean competence scores for EMR, comparing EMR intervention-trained and supervised therapist sessions (n = 35) to standard of care nontrained therapist sessions (n = 25). EMR, enhanced medical rehabilitation.

DISCUSSION

In this study, we successfully developed empirically tested methods for therapist training, supervision, and treatment fidelity monitoring in a rehabilitation intervention study involving OT and PT-based treatment. We trained and supervised therapists to an adequate level of adherence and competence with a protocolized intervention, and we were able to demonstrate this adequacy of treatment fidelity using a validated measure. The adequacy of treatment fidelity was demonstrated by the therapists showing good treatment integrity (i.e., high levels of adherence and competence) and treatment differentiation (i.e., much higher levels of adherence and competence than non-trained therapists). Moreover, we were able to develop and successfully apply these training, supervision, and treatment fidelity measurement techniques with OT and PT staff from a SNF who themselves were not researchers. We conclude that these techniques are a critical step in the development and testing of evidence-based rehabilitative intervention; therefore these methods have great potential for use in rehabilitation research.

Treatment manuals do not prevent drift.1 Our initial training techniques of asking therapists to read the treatment manual and of modeling new techniques were insufficient to achieve and maintain adequate treatment integrity. To define the EMR intervention and train the SNF therapists in it, the development of training and supervision procedures was an essential first step. The treatment protocol needed to be simplified to a brief prompt sheet, accompanied by a brief manual; supervision then focused on heavy repetition to recall and use these prompts. We successfully used direct observation and feedback, group supervision meetings, videotaping, role-play, the one-page prompt sheet, and adherence and competence forms to rate therapists. Our most effective supervision meetings were structured, led by either the supervising OT or PT, focused on one or two topics of concern, and highly repetitious of these concepts.

As for direct observation and videotaping, we recommend that they be initiated early in the pilot phase of treatment development to allow therapists to become comfortable with them. We assume that videotaping and direct observation by research team members made therapists more conscious of the way they interacted with patients and improved therapy in both EMR and SOC therapists. We did not find that videotaping and observation improved SOC therapists’ adherence and competence ratings over time, but EMR therapists, who also received feedback and training, achieved significantly better adherence and competence ratings. We assert that videotaping and direct observation by research team members did not negatively influence results of the study.

Although short video clips can illustrate appropriate treatment techniques, we also found that case studies with discussion and role-play were similarly helpful while being less time intensive. We conclude that both video clips and case studies should be developed during a pilot study and used for training and supervision in subsequent RCTs. We found that direct in-person observation and videotaping of treatment sessions were also essential both to attain treatment fidelity and to prevent drift. Direct in-person observation provided the opportunity to give immediate feedback to treatment implementers, thus quickly correcting errors, while videotaping allowed for more thorough observation of treatment (because the researcher can review segments multiple times).

To rate treatment integrity of therapists, the first aspect of treatment fidelity, we were successful in measuring adherence by tallying observable statements and actions by therapists that demonstrated intervention principles as outlined in the prompt sheet. Although closely related to adherence, the competence rating was a more subjective analysis of therapist administration of the three components of EMR. With this rating we were able to take into consideration therapist skills that were not amenable to tallying such as giving the patient appropriate activity options, motivating the patient to engage in vigorous activity, or providing feedback on effort to encourage participation in therapy. Using the adherence and competence ratings forms enabled us to supervise SNF therapists, give them objective and consistent feedback, and led to improved treatment fidelity. We also found that the measurement of reliability of the treatment fidelity scale was useful for further defining the intervention itself. With a validated adherence and competence checklist and high treatment integrity, researchers can have greater confidence in the results of their research and also be able to demonstrate this to the scientific community. Doing so is considered an essential part of developing and testing an evidence-based behavioral intervention. 10–12

We also demonstrated differentiation, the second aspect of treatment fidelity, between standard-of-care and EMR sessions. We were able to demonstrate large differences in the use of EMR intervention components (e.g., patient-directed approach, high intensity, and feedback on effort and progress) between the EMR-trained therapists and non-trained therapists. Thus, our data show that research-quality training and supervision of therapists can be successful and probably is necessary in order for OTs and PTs to achieve adequate fidelity with these or any principles underlying research interventions. As a result, we recommend continued testing of treatment fidelity throughout all stages of the research, including pilot and confirmatory RCTs, similar to methods in psychotherapy research.40–42

One limitation worth noting in this study was our use of non-blinded raters to collect fidelity data. This is acceptable in a treatment development phase, but we note that it is standard practice to use neutral raters in confirmatory RCTs to prevent bias. Although EMR therapists and SOC therapists were often treating patients in the same therapy clinical space, we did not address contamination issues. In spite of the ability to observe EMR therapists, our results showed that SOC therapists did not use the EMR techniques. Another limitation in our study was using a small number of therapists at only one SNF to develop training and supervision techniques and treatment fidelity testing methods. However, as our techniques and methods worked across disciplines (PT and OT) with therapists with a range of education and training and who were not themselves research staff, there is no reason to think that these techniques and methods would not be generalizable to other therapists and rehabilitation sites and contexts.

In summary, our experience with development of a treatment manual, training and supervision methods, and a tool to evaluate treatment fidelity provides an approach that rehabilitation researchers can use to ensure, measure and demonstrate treatment fidelity in complex interventions. For OT, PT, and other rehabilitation fields to improve evidence in developing and improving new behavioral interventions, researchers, journal reviewers, and practitioners must consider these methods of treatment fidelity implementation, data collection, and reporting to demonstrate and evaluate the quality and outcomes of clinical trials.

Supplementary Material

Supplemental Data 1
Supplemental Data 2

Acknowledgments

We would like to thank Cathy Huels, MSCCC-SLP, Rehabilitation Director, Linsey Kraus, OTR/L, Marijon Stern, COTA, Carolyn Carrabine, PT, and Cindy Messinger, PTA, at Barnes Jewish Extended Care (BJEC) for implementing EMR and providing feedback on training and supervision methods; Jacqueline Bitticks, BA, and Grace Snell, MSW, for their work in developing and coordinating EMR; Aliza Smason, OTD, and Emilie Richards, MSOT, for their assistance with this project; all rehabilitation and nursing staff at BJEC for assistance and cooperation with this study; and Kenneth P. Thomas, PhD for helpful editing suggestions.

Appendix 1: EMR one-page prompt sheet

Basics of Enhanced Medical Rehabilitation
GOAL & PLAN before every session DO throughout each activity or exercise CHECK after every activity or exercise

  1. Ask (patient name) to decide which goal to work on.

      (Patient name)’s GOALS:

    • A.

    • B.

    • C.

    • D.

    • E.

  2. PLAN: With (patient name) decide on what activities to perform that will help achieve the goal.

    1. Guide (patient name) toward more rigorous activities (especially at the beginning of the session). Use PRE (progressive resistance exercise) training.

    2. If an exercise, tell (patient name) clearly, so that a non-therapist can understand:

      1. What the activity is.

      2. Why they are doing it.

      3. How it relates to the goal.

    3. Ask (patient name) if they agree with and understand why they are doing the activity.

    4. Tailor activities to simulate patient’s home environment

  1. Focus on good communication

    1. Good eye contact

    2. Be at eye level

    3. Stay close to the patient

    4. Use plain English

    5. Avoid distractions (noise)

  2. Ask (patient name) about or comment on exertion level.

    1. If it is not at least moderate (5–6 out of 10), make the activity harder and say why.

    2. If it is moderate (5–6 out of 10) or vigorous (7–8 out of 10), tell them what benefits they are getting by achieving this effort.

  3. Verbalize (patient name)’s achievement when an activity becomes easier for them.

  4. If there is a barrier & (patient name) refuses to do or complete the activity), ask the patient what is going on. Try to find a reason for the barrier and implement a solution.

  5. Show empathy when (patient name) seems distressed

    1. Comment on their emotions.

    2. Acknowledge if the activity seems hard for them.

    3. Inquire about source of emotion and let (patient name) know that is okay to talk about it.

    4. Follow (patient name)’s lead to continue with therapy or take a brief break

  1. Ask (patient name), “How do you feel you did with that? What do you think about your progress on this? What do you think went well for you? What could have gone better?

  2. Tell (patient name) how their effort and progress on the activity is moving them closer to their goals.

  3. On Mondays, Wednesdays, and Fridays, using the progress binder, review progress with (patient name)

    Copyright: Washington University in St. Louis

    School of Medicine

    PI: Eric Lenze, MD

    US NIH - STIMULUS 5R34MH08386802
Open in a new tab

Appendix 2: EMR adherence and competence rating form

EMR: Adherence and Competence Rating Form Subject ID: Date:
Copyright: Washington University in St. Louis, School of Medicine PI: Eric Lenze, MD US NIH - STIMULUS 5R34MH08386802 Therapist: Day of week: Session start time:
Rater: Day of care at SNF: Session end time:
GOAL & PLAN Did (tally) Missed opportunity (tally) DO Did (tally) Missed opportunity (tally) CHECK Did (tally) Missed opportunity (tally)
1. Asked patient to decide what goal to work on. 14. Asked patient about effort level (including heart rate measurement). 17. Asked for patient feedback after each activity/exercise.
2. Decided with patient what activities/exercises to perform.  a. If effort level was low (<5) to moderate (5–6)…
  i. Commented on progress, if applicable  a. If patient identified a problem, therapist elicited strategies and solutions.
3. Guided patient towards more rigorous activities/exercises.   ii. Made activity/exercise harder, if appropriate  18. Related progress to goal achievement.
4. Therapist ensured that the patient understood the link between exercise/activity and his/her goal(s).  b. If effort level was moderate (5–6) or vigorous (7+), explained the benefits of achieving moderate or vigorous exertion levels. 19. At the end of the session, systematically reviewed patient’s progress. Yes No NA
5. Tailored activities to correspond to patient’s home environment, and explained this to the patient. 15. Were there rehab barriers? Yes No Competence Ratings
 If 15 is ‘yes’, complete a and b below Did (tally) Missed opportunity (tally) 20. Quality of therapy during session:
 low quality= 1
 medium quality = 2
high quality = 3
Areas of concern (if applicable) Did (tally)  a. Asked patient to elaborate on perceived barrier to participating/continuing exercise/activity.
6. Did not address environmental barriers (TV, distractions, lighting).  b. Tried to resolve barrier or find alternative exercise. 21. High Intensity COMPETENCE Rating:
 low competence=1
 medium competence=2
high competence=3
7. Arbitrarily decided upon exercise/activity volume and duration. 16. Did patient show emotional distress during session? Yes No
8. Did not communicate at eye level or at appropriate distance from patient.  If 16 is ‘yes’, complete a through f below Did (tally) Missed opportunity (tally)
9. Used jargon.  a. Commented on their emotions. 22. Patient-as-Boss COMPETENCE Rating:
 low competence=1
 medium competence=2
 high competence=3
10. Divided attention among >1 patient(s).  b. Acknowledged if exercise/activity seemed hard.
11. Used directive language.
12. Responded to patient distress with phrases like, “I know how you’re feeling,” or “Don’t worry”.  c. Inquired about source of emotion.
 d. Let patient know that it is okay to talk about emotions. 23. Feedback on Effort and Progress COMPETENCE Rating:
 low competence=1
 medium competence=2
 high competence=3
13. Therapist distractions (e.g., walking away from patient without explaining, chatting with other therapists, engaging patient in irrelevant chit-chat)  e. Followed patient’s lead to continue therapy or take a brief break.
 f. Used open-ended questions.
Open in a new tab

Footnotes

Editor’s Note:

Supplemental digital content is posted at www.AJPMR.com

Disclosures:

Financial disclosure statements have been obtained, and no conflicts of interest have been reported by the authors or by any individuals in control of the content of this article. Supported by grant US NIH -STIMULUS - 5R34MH08386802

References

  • 1.Moncher FJ, Prinz RJ. Treatment fidelity in outcome studies. Clin Psychol Rev. 1991;11(3):247–266. [Google Scholar]
  • 2.Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443–451. doi: 10.1037/0278-6133.23.5.443. [DOI] [PubMed] [Google Scholar]
  • 3.Wade DT. Editorial: Describing rehabilitation interventions. Clin Rehabil. 2005;19:811–818. doi: 10.1191/0269215505cr923ed. [DOI] [PubMed] [Google Scholar]
  • 4.Whyte J, Hart T. It’s more than a black box; it’s a Russian doll: defining rehabilitation treatments. Am J Phys Med Rehabil. 2003;82(8):639–652. doi: 10.1097/01.PHM.0000078200.61840.2D. [DOI] [PubMed] [Google Scholar]
  • 5.Wheeler S, Richards K. The impact of clinical supervision on counsellors and therapists, their practice and their clients. A systematic review of the literature. Counselling and Psychotherapy Research. 2007;7(1):54–65. [Google Scholar]
  • 6.Rakovshik SG, McManus F. Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance. Clin Psychol Rev. 2010;30(5):496–516. doi: 10.1016/j.cpr.2010.03.004. [DOI] [PubMed] [Google Scholar]
  • 7.Lichstein KL, Riedel BW, Grieve R. Fair tests of clinical trials: A treatment implementation model. Advances in Behaviour Research and Therapy. 1994;16(1):1–29. [Google Scholar]
  • 8.Bernard JM, Goodyear RK. Fundamentals of Clinical Supervision. 4. Upper Saddle River, NJ: Person Education, Inc; 2008. [Google Scholar]
  • 9.McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. Am Psychol. 2010;65(2):73–84. doi: 10.1037/a0018121. [DOI] [PubMed] [Google Scholar]
  • 10.Chambless DL, Baker MJ, Baucom DH, et al. Update on empirically validated therapies, II. Clin Psychol. 1998;51(1):3–16. [Google Scholar]
  • 11.Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annu Rev Psychol. 2001;52(1):685–716. doi: 10.1146/annurev.psych.52.1.685. [DOI] [PubMed] [Google Scholar]
  • 12.Öst LG. Efficacy of the third wave of behavioral therapies: A systematic review and meta-analysis. Behav Res Ther. 2008;46(3):296–321. doi: 10.1016/j.brat.2007.12.005. [DOI] [PubMed] [Google Scholar]
  • 13.Costa D. Clinical Supervision in Occupational Therapy: A Guide for Fieldwork and Practice. Bethesda, MD: AOTA Press; 2007. [Google Scholar]
  • 14.Fuhrer MJ. Overview of clinical trials in medical rehabilitation: impetuses, challenges, and needed future directions. Am J Phys Med Rehabil. 2003;82(10):S8–S15. doi: 10.1097/01.PHM.0000086995.80644.D7. [DOI] [PubMed] [Google Scholar]
  • 15.Nelson DL, Mathiowetz V. Randomized controlled trials to investigate occupational therapy research questions. Am J Occup Ther. 2004;58(1):24–34. doi: 10.5014/ajot.58.1.24. [DOI] [PubMed] [Google Scholar]
  • 16.Johnston MV, Sherer M, Whyte J. Applying evidence standards to rehabilitation research. Am J Phys Med Rehabil. 2006;85(4):292–309. doi: 10.1097/01.phm.0000202079.58567.3b. [DOI] [PubMed] [Google Scholar]
  • 17.Spillane V, Byrne MC, Byrne M, Leathem CS, O’Malley M, Cupples ME. Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. J Adv Nurs. 2007;60(3):343–352. doi: 10.1111/j.1365-2648.2007.04386.x. [DOI] [PubMed] [Google Scholar]
  • 18.Gordon WA. Perspectives on Rehabilitation Research. Arch Phys Med Rehabil. 2010;91(2):169–172. doi: 10.1016/j.apmr.2009.11.001. [DOI] [PubMed] [Google Scholar]
  • 19.Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: Explanation and elaboration. Ann Intern Med. 2008;148:295–309. doi: 10.7326/0003-4819-148-4-200802190-00008. [DOI] [PubMed] [Google Scholar]
  • 20.Leeuw M, Goossens MEJB, de Vet HCW, Vlaeven JWS. The fidelity of treatment delivery can be assessed in treatment outcome studies: A successful illustration from behavioral medicine. J Clin Epid. 2009;62:81–90. doi: 10.1016/j.jclinepi.2008.03.008. [DOI] [PubMed] [Google Scholar]
  • 21.Wade DT, Smeets RJEM, Verbunt JA. Research in rehabilitation medicine: Methodological challenges. J Clin Epid. 2010;63:699–704. doi: 10.1016/j.jclinepi.2009.07.010. [DOI] [PubMed] [Google Scholar]
  • 22.Perepletchikova F, Kazdin AE. Treatment integrity and therapeutic change: Issues and research recommendations. Clin Psychol Sci Prac. 2005;12(4):365–383. [Google Scholar]
  • 23.Sharpless BA, Barber JP. A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clin Psychol Rev. 2009;29(1):47–56. doi: 10.1016/j.cpr.2008.09.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Naleppa MJ, Cagle JG. Treatment Fidelity in Social Work Intervention Research: A Review of Published Studies. Res Soc Work Pract. 2010;20(6):674–681. [Google Scholar]
  • 25.Borrelli B, Sepinwall D, Ernst D, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol. 2005;73(5):852–860. doi: 10.1037/0022-006X.73.5.852. [DOI] [PubMed] [Google Scholar]
  • 26.Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. J Consult Clin Psychol. 2007;75(6):829–841. doi: 10.1037/0022-006X.75.6.829. [DOI] [PubMed] [Google Scholar]
  • 27.Bovend’Eerdt TJ, Dawes H, Sackley C, Izadi H, Wade DT. An integrated motor imagery program to improve functional task performance in neurorehabilitation: A single-blind randomized controlled trial. Arch Phys Med Rehabil. 2010;91:939–946. doi: 10.1016/j.apmr.2010.03.008. [DOI] [PubMed] [Google Scholar]
  • 28.Lenze E, Host H, Hildebrand M, Bitticks J, Morrow-Howell N, Carpenter B, et al. Enhanced Medical Rehabilitation is feasible: preliminary data on a novel treatment for older adults with depression. Am J Geriatr Psychiatry. doi: 10.1016/j.jagp.2012.11.006. In Press. [DOI] [PubMed] [Google Scholar]
  • 29.Wressle E, Eeg-Olofsson AM, Marcusson J, Henriksson C. Improved client participation in the rehabilitation process using a client-centred goal formulation structure. J Rehabil Med. 2002;34(1):5–11. doi: 10.1080/165019702317242640. [DOI] [PubMed] [Google Scholar]
  • 30.Londos E, Boschian K, Lindén A, Persson C, Minthon L, Lexell J. Effects of a goal-oriented rehabilitation program in mild cognitive impairment: A pilot study. Am J Alzheimers Dis Other Demen. 2008;23(2):177–183. doi: 10.1177/1533317507312622. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Lequerica AH, Kortte K. Therapeutic Engagement: A Proposed Model of Engagement in Medical Rehabilitation. Am J Phys Med Rehabil. 2010;89(5):415–422. doi: 10.1097/PHM.0b013e3181d8ceb2. [DOI] [PubMed] [Google Scholar]
  • 32.Guidetti S, Andersson K, Andersson M, Tham K, Koch LV. Client-centred self-care intervention after stroke: A feasibility study. Scand J Occup Ther. 2010;17(4):276–285. doi: 10.3109/11038120903281169. [DOI] [PubMed] [Google Scholar]
  • 33.Munin MC, Seligman K, Dew MA, et al. Effect of rehabilitation site on functional recovery after hip fracture. Arch Phys Med Rehabil. 2005;86(3):367–372. doi: 10.1016/j.apmr.2004.10.004. [DOI] [PubMed] [Google Scholar]
  • 34.Horn SD, DeJong G, Smout RJ, Gassaway J, James R, Conroy B. Stroke rehabilitation patients, practice, and outcomes: is earlier and more aggressive therapy better? Arch Phys Med Rehabil. 2005;86(12):S101–S114. doi: 10.1016/j.apmr.2005.09.016. [DOI] [PubMed] [Google Scholar]
  • 35.Binder EF, Brown M, Sinacore DR, Steger-May K, Yarasheski KE, Schechtman KB. Effects of extended outpatient rehabilitation after hip fracture. JAMA. 2004;292(7):837–846. doi: 10.1001/jama.292.7.837. [DOI] [PubMed] [Google Scholar]
  • 36.Pronovost P, Needham D, Berenholtz S, Sinopoli D, et al. An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. N Engl J Med. 2006;355(26):2725–32. doi: 10.1056/NEJMoa061115. http://search.proquest.com/docview/223924546?accountid=10639. [DOI] [PubMed] [Google Scholar]
  • 37.Salend SJ. Minimizing integrity-of-treatment violations. Except Child. 1983;50(2):179–181. [Google Scholar]
  • 38.Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. J Consult Clin Psychol. 1993;61(4):620–630. doi: 10.1037//0022-006x.61.4.620. [DOI] [PubMed] [Google Scholar]
  • 39.Sechrest L. Program evaluation: The independent and dependent variables. Clin Psychol. 1982;10(4):73–74. [Google Scholar]
  • 40.Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: Getting started and moving on from stage I. Clin Psychol Sci Prac. 2001;8(2):133–142. [Google Scholar]
  • 41.Carroll KM, Nuro KF. One size cannot fit all: A stage model for psychotherapy manual development. Clin Psychol Sci Prac. 2002;9(4):396–406. [Google Scholar]
  • 42.Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2010;45(5):626–629. doi: 10.1016/j.jpsychires.2010.10.008. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Data 1
NIHMS448678-supplement-1.pdf (56.6KB, pdf)
Supplemental Data 2
NIHMS448678-supplement-2.pdf (40.8KB, pdf)

RESOURCES