Abstract
A process fidelity assessment was conducted as a nested study within a home-based randomized clinical trial teaching self-management to 101 long-term indwelling urinary catheter users in the treatment group. Our hybrid model combined external assessments (outside observations and tape recordings) with internal evaluation methods (through study nurse forms and notes) for a comprehensive process fidelity assessment. Barriers, patient-related issues, and nurse perspectives were identified demonstrating the complexity in home care intervention research. The complementary and synergistic approaches provided in depth information about the context of the delivery and the impact of the intervention on study outcomes.
Keywords: process fidelity, intervention research, home healthcare, urinary catheterization, nursing
A key component of implementation science is related to the processes of the intervention. However, it is often difficult to discern the particular contribution of select intervention components in home healthcare (HHC) studies because evaluated studies are often multi-faceted and conducted in home settings with inherent complexity and variation. Even where it is possible to attribute effectiveness to the home visits, it is difficult to know which elements had the greatest impact on study outcomes, positively or negatively.
Thus, there is a need for comprehensive process evaluation in randomized clinical trials (RCT) to critically evaluate program fidelity, including standardization of intervention protocols, description and monitoring of intervention activities and measurement of variability in program delivery (dose) associated with expected outcomes (Scheirer, 1994; Sidani, 1998)(Faw, Hogue, & Liddle, 2005). The goal of these activities is to provide the best evidence available to promote improved interventions, care, and patient outcomes. Despite the need to ensure evidence- based practice, there has been a virtual lack of comprehensive process evaluation in HHC intervention research, especially from RCTs. Furthermore, studies of self-management interventions seldom provide sufficient detail of implementation strategies or issues related to delivery which could be useful in replication or application to other clinical settings, like HHC. (Schinckus, Van den Broucke, Housiaux, & Diabetes Literacy Consortium, 2014). This information is critical to evaluating intervention implementation and fidelity and to knowing how to sustain quality HHC programs/interventions in the community (Metzelthin et al., 2013).
Therefore, we conducted a process evaluation as a nested study within an RCT that was delivered to people with long-term urinary catheters residing in their homes. The self-care management intervention used nurse home visits to help patients monitor urine flow to improve self-management of indwelling urinary catheters and reduce untoward outcomes. Conducting a process evaluation of this trial provided needed insight into the multiple factors influencing not only the outcomes of this intervention but also the specific barriers and factors associated with intervention fidelity.
Background
Understanding how an intervention is delivered is central to knowing whether the treatment had the desired effect and it is relevant to the internal and statistical conclusion validity of the study (Santacroce, Maccarelli, & Grey, 2004). For a full evaluation of intervention fidelity, both quantitative and qualitative data are of value in providing a range of needed information, such as perspectives of study participants, contextual issues, and whether the approach differed among subgroups or sites (Oakley, Strange, Bonell, Allen, & Stephenson, 2006). Evaluation of the implementation processes requires exploring the “implementation, receipt and setting of an intervention to understand how the intervention was conducted” (Oakley et al., 2006). This knowledge provides insight into what is needed to implement the intervention into HHC or to make refinements of the intervention for further research.
Even in the presence of a well-developed research design and rigorous interventionist training, multiple issues can develop posing barriers to intervention implementation in HHC settings, especially among adults with chronic disease and comorbid conditions. Barriers to care are quite unpredictable. For example, while performing professional services, nurses implementing a research protocol are guests in patients’ homes and social milieu. As a result, there may be multiple conversations and distractions (e.g., unsafe environment; problem with condition) occurring which can take a higher priority over intervention activities.
Self-management education programs are often conducted in diverse populations, have complex research designs, and multicomponent interventions, making process evaluation challenging (Nicolaides-Bouman, van Rossum, Habets, Kempen, & Knipschild, 2007) (Schinckus et al., 2014). Moreover, in order to promote HHC self-care management, patients need to actively participate in decision-making and also collaborate with the nurse to achieve their goals (Liebel & Powers, 2013). This can be very challenging when patients have cognitive challenges, chronic illness, or sensory challenges. Subsequently, it is important for researchers to include process measures to monitor patient responses or issues and barriers in the environment which might compromise the intervention delivery.
We used a hybrid model of evaluation which involved a blend of both external and internal evaluation methods, providing a superior method to evaluate program effectiveness (Bourgeois, Hart, Townsend, & Gagné, 2011; Conley-Tyler, 2005). External evaluation was conducted by a contracted expert knowledgeable about the study, but not part of implementing the intervention. The internal evaluation was performed by the study nurses after each of their four encounters. Thus, the combination of external and internal assessments provided us with a more comprehensive evaluation than one form of evaluation alone.
In this paper we report on how we used both forms of evaluation, internal and external, with quantitative and qualitative data to better elucidate contextual issues in the study. Our purposes are to: 1) demonstrate how process fidelity evaluation is integral to a full and comprehensive assessment of treatment fidelity and 2) illustrate the value in using a hybrid model of both internal and external assessments.
Essentially our plan to assure fidelity to the intervention included activities in four areas to: 1) assure that the intervention was delivered as it was designed, 2) learn what factors may have affected the delivery and/or receipt of the intervention, 3) identify how participants’ responses affected outcomes, and 4) understand the study nurse perspectives of strengths in the intervention and what might be changed in the future.
Intervention Summary
The nursing intervention was conducted in two sites, one in a university setting in Central/Western New York State and the other in a large home care agency (that often conducts research) in a large metropolitan area in the same state. The participants were community dwelling individuals. Seventy-four percent of the sample of 202 (half intervention and half usual care) completed the full 12 months of the study, and 87% were referred through home care agencies. The intervention involved teaching self-management skills to 101 long-term adult indwelling urinary catheter users as a part of a 12 months’ randomized clinical trial, using a urinary diary (intake and output record and journal) and an educational booklet. The initial home visit (HV) was received by 99/101 allocated to the intervention (98%), 95% received the second HV in the first month, 93% received the follow up phone call two weeks later, and 91% received the final HV at 4 months. The two study nurses, who were trained together at the beginning, delivered all the intervention components. The sample of equal numbers of men and women was diverse by race (57% white, 30% black, 4% other, 9% mixed race), ethnicity (11% Hispanic), age (ranged 19-96 years), and medical diagnoses [often spinal cord injury (40%) or multiple sclerosis(23%)]. A full description of the intervention and the sample at baseline have been published elsewhere (Wilde et al., 2013a; Wilde et al., 2013b).
Methods for Hybrid Process Evaluation
External Assessment
External evaluation was defined as an objective assessment through observation of the intervention delivery through either tape recorded or directly observed home visits of study nurses with their study participants. For our study's external assessment, there were two key components, adherence to protocols and competence of the study nurse. Adherence was defined by whether the intervention was administered as outlined in the planned protocol for each encounter in the training manual, including use of verbal scripts (which correspond to the theoretical framework of Bandura (Bandura, 1997) and the completeness in addressing each component (listed in the left column of Table 1). In addition, the study participants’ response of interest were noted and rated, using a simple scale from 0 to 3. Competence was defined as therapeutic communication, patient centered approach, using terminology appropriate to the individual's needs, and encouraging confidence in self-management. The study nurses and raters were provided with two pages of information describing each term, such as the following link which illustrates a lack of therapeutic communication in a nurse who appears rushed, harried, distracted and not hearing the patient. http://ezinearticles.com/?Therapeutic-Communication-in-the-Nursing-Profession&id=594747. Coding for adherence and competence were on a five point scale of performance from 1- not at all, 2- a little, 3- somewhat, 4- considerably to 5- extensively. Competence requires that the skill of the interventionist be evaluated and this includes “communication, technical abilities and skills in responding to the participants receiving the intervention” (Breitenstein et al., 2010), p. 165). Therefore, we informed the raters to feel free to make comments in the sections on the form or another page, and that they might be asked to provide additional feedback during the conference calls with the study nurses. In person assessments were completed by masters’ prepared nurses familiar with the study and intervention content who evaluated each component related to the above criteria; no coaching was allowed.
Table 1.
External assessment: Delivery of Intervention of study nurses’ home visits
| University-Mean (SD) | Home care agency-Mean (SD) | ||||||
|---|---|---|---|---|---|---|---|
| Components of study nurse assessment | Observation | n a | A | C | n a | A | C |
| Adherence (A) & Competence (C) b | |||||||
| I & O entries discussed and compared with a standard | HV 1 | 4 | 4.50 (0.58) | 5 (0) | 8 of 9 | 4.75 (0.46) | 4.75 (0.46) |
| HV 2 | 4 | 4.75 (0.50) | 4.75 (0.50) | 8 | 4.63 (0.74) | 4.88 (0.35) | |
| HV 3 | 1 of 2 | 4 | 5 | 2 of 3 | 5 (0) | 5 (0) | |
| Journal used to understand urine flow relationships | HV 1 | 3 of 4 | 5 (0) | 4.67 (0.58) | 8 of 9 | 4.75 (0.46) | 4.75 (0.46) |
| HV 2 | 4 | 2.75 (1.26) | 3.25 (1.71) | 7 of 8 | 4.29 (1.50) | 4.43 (1.51) | |
| HV 3 | 1 of 2 | 3 | 5 | 0 of 3 | |||
| Problems identified | HV 2 | 4 | 4.25 (.50) | 4 (0.82) | 8 | 5 (0) | 4.75 (0.46) |
| HV 3 | 2 | 5 (0) | 5 (0) | 2 of 3 | 5 (0) | 5 (0) | |
| Educational strategies identified and listed on action plan | HV 2 | 4 | 4 (0.82) | 5 (0) | 8 | 5 (0) | 5 (0) |
| HV 3 | 2 | 5 (0) | 5 (0) | 3 | 5 (0) | 5 (0) | |
| Motivational bookmark | HV 2 | 4 | 1.5 (1.00) | 1.5 (1.00) | 6 of 8 | 4 (1.55) | 3.57 (1.27) |
| HV 3 | 1 of 2 | 1 | 1 | 2 of 3 | 4 (0) | 4.5 (0.71) | |
| Identified barriers | HV 1 | 4 | 5 (0) | 5 (0) | 6 of 9 | 4.83 (0.41) | 4.67(0.52) |
| HV 2 | 4 | 4 (1.16) | 3.75 (1.50) | 8 | 4.50 (0.76) | 4.50 (0.76) | |
| HV 3 | 2 | 4.5 (0.71) | 5 (0) | 3 | 5 (0) | 5 (0) | |
| Assist in identifying goals | HV 2 | 4 | 3.50 (1.73) | 4.75 (0.50) | 8 | 4.63 (0.74) | 4.5 (0.76) |
| HV 3 | 2 | 5 (0) | 5 (0) | 2 of 3 | 5 (0) | 5 (0) | |
| Congratulate on successes | HV 2 | 4 | 5 (0) | 5 (0) | 8 | 4.25 (0.89) | 4.38 (0.74) |
| HV 3 | 2 | 5 (0) | 5 (0) | 3 | 5 (0) | 5 (0) | |
| Follow Scriptsc –(0-2) higher is better | |||||||
| HV 1 | 4 | 1.75 (0.50) | 7 of 9 | 1.57 (0.53) | |||
| HV 2 | 4 | 1.25 (0.50) | 7 of 8 | 1.57 (0.53) | |||
| HV 3 | 2 | 2.0 (0) | 3 | 1.67 (0.58) | |||
| Study participant level of interest d | |||||||
| HV 1 | 4 | 2.25 (0.96) | 6 of 9 | 2.67 (0.82) | |||
| HV 2 | 4 | 2.75 (0.50) | 7 of 8 | 2.71 (0.76) | |||
| HV 3 | 2 | 3 (0) | 3 | 3 (0) | |||
To clearly show missing data, we identified the n as completed observations. But when depicted e.g., like this: 2 of 3, data were collected for 2 of 3 observations, with the result of 1 observation having missing data for that item.
1 to 5 (higher is better): 1 not at all; 2 a little; 3 somewhat; 4 considerably; 5 extensively.
0 to 2 (higher is better): 0 not at all; 1 somewhat; 2 completely.
0 to 3 (higher is better): 0 not interested; 1 somewhat; 2 moderately; 3 very.
To assure consistency of intervention delivery over time and between sites (for instance “drift” in which an interventionist might modify the approach too much), 10% of the 300 home visits were selected at random (by our statistician) for audio-taping or home visit observation (5% each), with participant permission. Observations were adjusted for sample size by site and time of the encounter. For example, more observations were scheduled early in study, so that adjustments could be made, and for HVs 1 and 2, when key teaching took place. At the much larger home care site, there were 9 observations of HVs #1, 8 for HV 2, and 3 for HV #3 (20 total). At the university site, there were 4 evaluations for each of HVs #1 & 2, and 2 for HV #3 (10 total). Audio tapes were evaluated by the principal investigator (PI), the first author, or the project coordinator (PC). In a few instances, when the ratings were not easily determined, both raters listened to the audiotape and agreed on the scores. Feedback was given to the study nurses shortly after the observations, either individually or during conference calls depending on the issue and whether the information would be useful to both study nurses.
Internal Assessment
In our trial internal evaluation was related to information elicited from the study nurses in checklists and assessments that they completed after conducting the intervention. Forms were completed by the study nurse after each of four encounters, answering yes or no to the questions about presence or absence of barriers and participant issues/concerns related to the encounter that affected the nurses’ ability to complete all planned components as scheduled and/or that impacted the patient's ability to follow up with the self-management teaching. There was a place to write comments if desired. The questions are depicted in results in Tables 2-3.
Table 2.
Internal assessment of implementation delivery--barriers
| Barriers | HV 1 n=98 (# Yes) | HV 2 n=93 (# Yes) | TC n-92 (# Yes) | HV 3 92 (# Yes) | Comments # persons affected (times reported) |
|---|---|---|---|---|---|
| 1. Unsafe environment (elaborate) | 9 | 10 | 10 | 8 | Dirty-cluttered- 9(14) Elevator or stairs problem 5(8) Small space 2(4) Poor exit access/No ramp 7(10) Top floor 2(2) Door unlocked 5(10) |
| 2. Had to reschedule entire visit for later home visit (explain reason) | 10- | 25 | 1 | 22 | Time conflict 15(16) Sick or hospitalized 10(11) Tired 5(5) Other 14(15) |
| 3. Had to reschedule part of visit for later phone call (explain reason) | 3 | 4 | 10 | 6 | Family-visitors present 2(2) Hospitalized 1(1) Other 2(2) |
| 4. Home care nurse or other health care worker visited | 36 | 26 | 10 | 31 | Nurse visits 10(12) Home health aide 41(68) Specialist visit 10(10) Family 3(3) |
| 5. Family or caregiver issues (describe briefly) | 10 | 16 | 9 | 10 | Lives alone 3(4) Unhelpful caregivers 7(11) Family works(involved 5(7) Family member disabled 2(2) Family not helpful 7(9) Friend(family nearby & help 5(7) |
| 6. Competing demands or commitments (e.g., caregiver role strain and stress, death of spouse or caregiver, disruptive adult or child in home) | 3 | 7 | 6 | 3 | Worry about family 7(8) Personal issues 4(7) Care for children 4(4) Other 5(5) |
| 7. Limited social support (e.g., no caregiver, few extended family, friends, resistance from social supports) | 12 | 14 | 14 | 8 | Limited caregivers 5(8) Few extended family 12(25) Having home attendants 23(31) Unsupportive family 7(8) Supportive family 13(20) Unhelpful caregivers 3(4) |
| 8. Scare(limited community resources and referrals (e.g., lack of transportation, no insurance, financial difficulties, few community agencies) | 5 | 12 | 8 | 7 | Lack transportation 4(4) Lack health care-insurance 16(20) Lack assistance 10(12) |
Note: TC = telephone call.
Table 3.
Internal assessment of implementation delivery--participant-centered issues
| Participant centered issues | HV 1 n= 98 (# Yes) | HV 2 n=93 (# Yes) | TC n-92 (#Yes) | HV 3 92 (# Yes) | Comments # persons affected(times reported |
|---|---|---|---|---|---|
| 1. Lack of support by family(caregivers in self-management practices (describe *) | 6 | 16 | 16 | 9 | Lack support of family 12(18) Lack support of caregivers 14(18) Being alone 5(7) Not enough encouragement 5(5) |
| 2. Lack of confidence in ability to modify behavior in various settings (state why this seems to be the case and describe steps to address in participant action plan) | 7 | 7 | 10 | 9 | Lack knowledge 2(2) Physical limitations 7(7) Memory deficit or depression 2(2) Lack of interest or motivation 2(2) Lack of self-responsibilities 3(3) Not ready for change 3(2) |
| 3. Nurse-pt. relationship and communication (e.g., lack of trust, participant not involved or unwilling to collaborate in decision-making with nurse) | 5 | 3 | 3 | 5 | Poor relationship with provider 4(4) Lack of trust 4(5) |
| 4. Physical health challenges (e.g., acute illness, co-morbidities, severity of disability, high level of pain) | 65 | 68 | 66 | 73 | Severity of disability 69 (224) Co-morbidities 15 (26) High level of pain 6 (13) Acute illness 5(8) |
| 5. New cognitive or mental health challenges (e.g., dementia, presence of depression and(or other mental health diagnoses) | 5 | 4 | 1 | 4 | Memory deficit 5(5) Depression 5(8) Dementia 1(2) Confusion(bitterness 2(2) |
| 6. Unhealthy behavior or addictions (e.g., alcohol abuse, drug abuse, history of physical abuse) | 10 | 5 | 3 | 4 | Alcohol abuse 2(5) Smoker 9(23) |
| 7. Lack of motivation to engage in goal attainment and health behavior change (e.g., unwilling to engage in self-management behaviors; not ready to change health behaviors; participant unable to learn skills) | 3 | 11 | 15 | 12 | Not ready or unsure 7(10) No perception what is needed 3(4) Not willing to change 7(8) Needs support-encouragement 5(5) |
| 8. New communication challenges (e.g., speech or language processing difficulties, visual and(or hearing impairments) | 4 | 3 | 1 | 4 | Speech 4(4) Vision or hearing 7(11) |
| 9. Transitions in care (e.g., nursing home placement, hospitalization, in-patient rehabilitation needed) | 3 | 5 | 7 | 13 | Hospitalization 13(17) Nursing home 2(2) Rehabilitation 3 (3) |
Participant responses in relation to outcomes
Study nurses also recorded information about how the participants responded to the intervention including: participant overall interest, adherence to the agreed upon plan for catheter self-management, awareness of urine flow, self-management behaviors, goals, completeness of the urinary diary, and use of the intervention booklet and motivational bookmark. These were coded using simple rating scales (with comments as desired). The scales are depicted in the results in Table 4.
Table 4.
Internal assessment: Implementation delivery- Participant responses
| Participants’ responses | HV 1 N = 99 (# Yes) | HV 2 N = 96 (# Yes) | TC N =94 (#Yes) | HV 3 N = 92 (# Yes) |
|---|---|---|---|---|
|
What was the response of participant to the intervention, e.g., seemed interested, asked questions, participated in activities?
| ||||
| Did not seem interested | 3 | 1 | 5 | 1 |
| Somewhat interested | 9 | 8 | 9 | 8 |
| Moderately interested | 18 | 16 | 24 | 19 |
| Very interested | 69 | 71 | 56 | 64 |
| Participant adherence to plan | ||||
| Voiced awareness (observations of own urine flow issues | N/A | 90 | 90 | 89 |
| Stated has changed self-management behaviors | N/A | 38 | 62 | 67 |
| Set initial goal(s) | N/A | 82 | N/A | N/A |
| Progress toward goals | N/A | N/A | ||
| Met initial goal(s) | 65 | 69 | ||
| Did not meet goal(s) | 18 | 13 | ||
| Revised goals | 7 | 8 | ||
| Completeness of I & O form (asked only at 2nd HV) | N/A | N/A | N/A | |
| Completed for 3 days | 62 | |||
| Partial but good enough | 13 | |||
| Very incomplete | 4 | |||
| Did not do | 12 | |||
| Other comments | 2 | |||
| I & O form used (asked only at TC and booster visit) | N/A | N/A | ||
| Used | 12 | 7 | ||
| Not Used | 82 | 83 | ||
| Intervention booklet used | N/A | N/A | 32 | 23 |
| Motivational bookmark used | N/A | N/A | 9 | 6 |
Coding process forms
We realized that coding direction for the responses (high to low or low to high) was inconsistent among questions, so we recoded the answers for more consistent ordinal scales. “Following scripts” was recoded as 2=completely, 1=somewhat, and 0= not at all. “Response to the intervention” was coded as 3=very interested, 2=moderately interested, 1=somewhat interested, and 0=did not seem interested. “Progress toward goals” was recoded as 2=met goals, 1=revised goals, and 0=did not meet goals. “Intake and output completeness” was recoded as 3=complete, 2=partially complete, 1=very incomplete, and 0=did not do.
To further understand how interest in the intervention and completion of the study activities contributed to study outcomes, we used data collected from the nurses’ process forms and bivariate correlation analysis to explore whether participant interest, achievement of goals, and completion of intake and output (I & O) were associated with study outcomes (catheter-associated urinary tract infection [UTI] and blockage) at 6 and 12 months.
Qualitative assessment through study nurses’ reflections
A few months after the intervention for the study had been completed, the researchers held an in depth conversation with the study nurses to discuss their experiences and reflections. The interview data are based on an hour and half tape-recorded interview between the two study nurses and the PI and university site PC. Questions were asked about the process forms (e.g., which they found helpful), use of theory (Bandura, 1997) including scripts, intervention materials (urinary diary, educational booklet), and suggestions for the intervention.
Data Analysis
Data were transcribed to SPSS, v19 (http://www-01.ibm.com/software/analytic) by the study coordinators; numerical and narrative data were coded/analyzed by research team members (the first author and two doctoral student authors). Scores for competence and adherence were calculated for each site and verified against the original forms. Due to the small samples for each observable component and the unequal samples per site, we did not test for significance, but instead displayed the data in Table 1. Narrative comments about barriers and patient-related issues were analyzed using Miles and Huberman's approach through an iterative process of coding, data reduction, display in tables, verification, and conclusions (Miles & Huberman, 1994). This involved reducing the nurses’ brief comments into one or two word descriptive entries (narrative coding). Then the codes (comments) were organized into content categories in tables. Codes were agreed upon by both coders.
Bivariate analysis of participant response in relation to study outcomes
A correlation matrix was completed using Sprearman's Rho to examine the relationships among participant responses, completion of activities and outcomes. Four new variables of urinary outcomes of UTI and blockage were created to sum the counts of adverse events at six and twelve months which had been recorded bimonthly for months 2-6 as well as months 2-12 respectively. After creating the new variables, one person's blockage data were removed as an outlier and recoded as missing. The variable Mean Response was created to reflect the average response (interest) to the intervention at the four study nurse contacts. The variable Mean Progress was created to reflect the average progress towards goals at two time points. The variable I & O form completeness was available at one time point and used to reflect the level and progress in completing the I & O form.
The interview with study nurses (for qualitative data) was listened to in its entirety and an iterative process between the PI and the university site PC was used to identify themes in the data using a simple content analysis approach by writing several drafts of the key points in the conversation until iteratively a final summary was constructed and agreed upon. The summary was validated by the study nurses (also co-authors), clarifying when needed so that an accurate description and interpretation was conveyed of their perspectives.
Results
Findings related to External Assessment (Table 1)
Most ratings were well over 4 on the 5-point scale for adherence and competence, with higher scores being better. Slight increases in scores over time were notd in the nurse's “following scripts” and “study participants’ level of interest.” Observations indicated the need for additional training related to more consistent use of the journal and motivational bookmark during the second self-management teaching visit. There was little variability in the performance of the two study nurses, providing evidence that the intervention was delivered consistently across sites and over time.
Findings related to Internal Assessments
Barriers
(Table 2) Some home visits needed to be rescheduled fully (ranging from 10% for HV1, 27% for HV 2, and 24% for HV 3) or rescheduled partly for a phone call follow up later as per study protocol (3-6%). Eleven percent of the planned phone call encounters, which took place two weeks after the second home visit, also were interrupted and completed later, but only 1% were entirely rescheduled. Frequently other healthcare workers were present, including nurses, home health aides, and specialists, but we did not ask whether others visiting had caused rescheduling of visits.
There were multiple comments related to barriers to the delivery of the intervention or to the completion of the recommended self-management strategies. Some issues were related to barriers identified by smaller numbers of individuals. For instance, family or caregivers/attendants were sometimes perceived as unhelpful or not available, such as having a disability or illness themselves. Frequently a lack of social support was noted, and comments indicated that this involved either not having enough support from paid caregivers or family, or that they were in fact unsupportive. However, in relation to the item about limited social support, there were positive comments also which depicted supportive families (in 13 persons, reported 20 times), and there were positive comments about friends/family who lived nearby or were helpful (in 5 persons, reported 7 times). Issues such as transportation or healthcare insurance were reported often (in 16 persons, reported 20 times).
Participant centered concerns
(Table 3) Severe disabilities were identified 224 times in 69 persons, and pain and co-morbidities were frequently identified as physical health challenges. A lack of motivation was noted in 12%, 16%, and 13% during the 2nd HV, telephone call, and 3rd HV respectively, which were times for learning or refining self-management strategies. 18 persons were dealing with their recent return home from the hospital or nursing home/rehabilitation setting. A wide range of issues were noted in a small number of individuals, including lack of confidence or deficits in participants’ abilities (e.g., in hearing, speech, vision, cognition), which could impact their self-management capacity. As in barriers above, there was a similar report of a lack of support related to self-management related to families or caregivers, being alone, or not receiving encouragement.
Participants’ responses
(Table 4) Some of the components were assessed only during certain encounters, so the responses in the table reflect this. In general, there was variability in responses, with about 66% of the sample responding rather positively to intervention components (such as the diary and goal setting), another 10-11% seemingly less interested or engaged, and the others in between. The bivariate analysis results (Table 5) show that study participants’ (level of interest) had a significantly positive relationship with both their I & O diary completion (p=0.022) and progress towards goals (p<0.001). The association between progress towards goals and I & O completion are also positive and with a trend, but it is not statistically significant (p=0.068). In addition, there is a significant positive relationship between the response to the intervention and the total UTI counts at 12 months (p=0.005) but not at 6 months. Blockage counts were not related to the response to the intervention.
Table 5.
Correlation matrix of participant responses related to study outcomes (Spearman's Rho test)
| Mean Response to intervention | Mean Progress towards Goals | # UTIs in 6 months | #UTIs in 12 months | Blockages in 6 months | Blockages in 12 months | |
|---|---|---|---|---|---|---|
| I &0 form completeness | ||||||
| Correlation | .246 | .198 | –.124 | –.111 | –.057 | –.015 |
| Sig. (2 tailed) | .022 * | .068 | .232 | .283 | .586 | .889 |
| N | 87 | 86 | 95 | 95 | 93 | 93 |
| Mean Progress towards Goals | ||||||
| Correlation | .379 | –.008 | .063 | .067 | –.023 | |
| Sig. (2 tailed) | <.001* | .944 | .562 | .542 | .837 | |
| N | 83 | 87 | 87 | 85 | 85 | |
| Mean Response to intervention | ||||||
| Correlation | .174 | .299 | .039 | .010 | ||
| Sig. (2 tailed) | .104 | .005* | .719 | .930 | ||
| N | 88 | 88 | 86 | 86 | ||
Significant correlations.
Findings related to Study Nurse Interview
Use of study forms to guide activities
The checklist used by study nurses was helpful in planning for each encounter, and the action plan (like a nursing care plan) systematized the process further to ensure that every part of the educational booklet was addressed. The nurses said it could be easy to just focus on one problem and miss other issues.
Intervention suggestions
The nurses said that participants adapted the intervention to their own needs, for instance using their own calendar for tracking catheter problems instead of the study calendar. Some participants offered suggestions on self-management, for example how sock ribbing or leg warmers can be used to anchor the catheter and prevent skin irritation.
Some persons wanted more information about catheters such as, links to websites in the educational booklet for catheter supplies, insurance reimbursement options, and support for living with a common medical problem like spinal cord injury. More pictures were requested for people with low literacy, and diagrams depicting catheter placement in the body. The nurses thought that the motivational bookmark was not used much, but some people liked the quotes from catheter users because they do not ordinarily meet others with catheters.
The nurses suggested adding probing questions to their “scripts” for each encounter, e.g., when a person had a UTI, asking “what was meant” to elicit which symptoms were noticed. Other suggestions from the study nurses included: 1) carry a copy of the educational booklet during home visits as people can misplace theirs, 2) incorporate the I and O sheets and space for notes (instead of a journal) into the educational booklet, 3) add numbers to the urine color scale for comparisons over time, and 4) turn the booklet into an attractive presentation through desktop publishing and offer it as a CD at the end of the study.
Theory application
There were familiar nursing elements in the intervention related to advocacy and empowerment of the patient in promoting self-efficacy (confidence) related to catheter self-care. The use of scripts helped the nurses learn what to address and how, and this promoted consistency of theory application over time. The study nurses thought nurses in practice could learn to use the guiding scripts and that catheter self-efficacy evaluation also would be especially useful for patients new to a catheter. To facilitate individualizing the content on adjustment, they proposed that participants be asked “What does adjustment to living with a catheter mean to you?”
Other reflections about catheter self-management
The nurses made the distinction between social/psychological and physical adjustment. Some people considered the catheter a problem, and for others it was a blessing to not be wet any longer. For others, it might be interpreted as a sign that the person might not get better. Although many participants had used catheters for years, several stated that no one really talks about the catheter with them; the usual focus is on the technical aspects of changing it. The nurses recognized that some participants experienced a sense of isolation, shame, but also relief over having a catheter. Overall, this intervention gave participants a chance to have an in-depth dialogue with a health professional about how a catheter impacts their lives. One of the study nurses said that the catheter “is important and it defines them.”
Discussion
This hybrid process evaluation demonstrated that the study was implemented as planned and had the necessary infrastructure to support the requirements of the study. Most components related to the nurses’ competency and adherence to intervention delivery were consistently rated high or improved over time, indicating the potential benefit of feedback provided to nurses during intervention delivery as well as ongoing training to prevent intervention “drift.” There was also consistent enactment of the intervention delivery across persons seen by home visit process forms (Tables 2-4) and checklists of completed nurse activities (not shown in this paper). Overall, our external evaluations, internal nurse self-reported process data, interviews with study nurses, and analysis of study participants’ responses in relation to outcomes provided multiple insights about what worked and how to improve intervention delivery.
The bivariate analysis looking at the responses to the intervention and study outcomes indicates that people who responded more to the interventions appeared to have more self-management behaviors (I &O completion) and achieved their goals better. Also people who had more response to the intervention (interest in) might have had more UTIs and thus were more interested or motivated toward the intervention activities. However, it could be that people who responded more to the intervention gained more knowledge of UTI, and thus paid attention and became more aware of their UTI symptoms and sought treatment for UTIs more.
A number of process evaluations have been conducted as part of nurse-led research studies that provided similar self-care interventions in community dwelling persons with multiple co-morbidities (Liebel, Powers, Friedman, & Watson, 2012; Metzelthin et al., 2013; Nicolaides-Bouman et al., 2007). Similar to our study these trials used mixed methods and included qualitative interviews to enrich the scope of the quantitative process data. However, these self-management studies seldom include theories to plan the intervention implementation components (e.g. self-efficacy theory), frameworks to conduct process evaluation (e.g., hybrid model), or include sufficient detail to accurately appraise treatment fidelity and program effectiveness (Liebel et al., 2012;Nicolaides-Bouman et al., 2007). Finally, multiple factors that can impact outcomes were often not considered during intervention design and development planning (e.g., target population, intervention characteristics, staff carrying out the visits, and the compliance to self-care management advice (Schinckus et al., 2014). Thus we believe that this process fidelity assessment provides some much needed information which could improve home visiting trial research.
Our research used a novel feedback loop by completing the process form immediately after encounters, thus giving the study nurses an opportunity to reflect on processes and barriers to care. Feeback loops like this are known to improve practitioners’ performance and can impact targeted outcomes of the intervention (Breitenstein et al., 2010). Furthermore, the nurses’ use of scripts (which were assessed externally) related to identifying and supporting sources of self-efficacy. This, combined with goal setting and self-monitoring fluid intake and output (assessed internally), were presumed to increase self-efficacy (our theoretical framework). The scripts also might have increased the nurses’ abilities toward therapeutic relationships related to self-efficacy. Therefore, we believe that completing the process forms not only helped us understand the issues after the study was over but also helped the nurses maintain the quality of the intervention while it took place.
Comparison of External and Internal assessments
Scheduling joint home visits for the external raters was not easy because of competing time demands and the preset randomization plan for observing different parts of the intervention. While the external assessments were critical to a more objective assessment of the fidelity of the intervention delivery, the internal assessments gave us information that was not otherwise available, such as details related to barriers and participant issues and their responsiveness (Tables 2-4). The study nurse interviews gave information into the contextual issues of the delivery in homes, affirmed the value of the theoretical framework, and helped us identify how we may improve the intervention.
Time and financial constraints are often considered when planning for evaluation of intervention fidelity. Our internal assessments were easier to complete because they did not need a statistically-based plan for the outside evaluations and listening to and coding tape recordings was not needed. However, there was a considerable amount of time spent by study nurses who completed the process forms and by coordinators who collected them and input their data into a spread sheet and data analysis program (SPSS). Verification of data and analysis were similar for both forms of fidelity data.
Intervention Research in the Home
Process evaluation is particularly valuable in research taking place in the home, including feedback from the home-based interventionists (Nicolaides-Bouman et al., 2007). In our qualitative analysis, we retrieved relevant information from interviews with the study nurses regarding fidelity that addressed overall levels of competence. Similar to other HHC studies we found that implementation fidelity was influenced by the nurses’ commitment to the study (Hasson, Blomberg, & Dunér, 2012; Stijnen, Jansen, Duimel-Peeters, & Vrijhoef, 2014). Specifically, in this study nurses appeared committed to helping patients effectively adjust to living with their catheters, as well as their ability to execute the intervention protocol.
Prior research has demonstrated that multiple barriers exist to intervention delivery in HHC interventions due to the complexity of the home environmental as well as the changing social milieu (Liebel, Friedman, Watson, & Powers, 2009;Markle-Reid et al., 2014). For example, one study (Nicolaides-Bouman et al., 2007) found that patients were less likely to engage in collaborative self-care management with nurses when there were contextual challenges such as unsafe home environments or cognitive decline. Additionally, participants also reported dissatisfaction when nurses missed visits; however, in our study nurses performed visits without absence, and rarely needed to reschedule the visit. In fact only 15% of the visits were rescheduled fully, most often for time conflicts or illness.
Our intervention was somewhat individualized and reflects this type of challenge (Radziewicz et al., 2009) as well as demands related to delivering multiple components in a home setting (Nicolaides-Bouman et al., 2007). There are often competing demands related to other diseases, or comorbidities, which are common in patients with chronic illness, such as those with indwelling catheters. One such challenge for us was a lack of motivation in over 10% of the participants in our study during most of the encounters. This is an important concern because it could affect use of intervention materials and evaluation of study effectiveness. We believe that this might be related to multiple co-morbidities which could affect some people's participation. Some of the less motivated may have benefitted from other individualized techniques, e.g. motivational interviewing or more frequent visits, which could support self-care management and improve clinical outcomes (Bodenheimer & Grumbach, 2007) (Chen et al., 2013).
Limitations
While all of the 30 observations planned of the external assessment were completed, during data verification we realized that there were items which were scored incorrectly (2.4%); therefore, some items were recoded and verified by two members of the research team. Five items were missing out of 80 items expected at the university site, and 22 were missing out of 155 items expected at the home care site of (6% at the university site and 14% at the home care agency site, 11% overall). (See Table 1.) Besides more missing data in the home care site, the journal component was often not addressed during HV#2 and HV#3. We believe this was related to a general lack of interest in the journal when we assessed the perceived value of intervention components with those in the self-management group (Wilde et al., 2013a).
Verification of the forms for the external assessment taught us some important lessons. Our use of three different scales with varying directions and metrics for the external reviews most likely confused some of the raters. In addition, on the assessment form related to identifying barriers, there may have been some confusion that the raters were to list the barriers that they observed; this was incorrect as they were to evaluate the study nurse for adherence and competence related to the task of identifying barriers. We did not anticipate that the reviewers might need closer supervision to assure complete and accurate coding. The 1-5 scale we used for external evaluations has not been evaluated for reliability, and thus it is a limitation of the study. The use of not validated scales is common in intervention fidelity research and it demonstrates why there is a need for development of psychometrically robust scales (Breitenstein et al., 2010).
In our study we did not learn what types of home support were present because the study nurses identified only barriers on their process forms. Identifying the types and range of caregiving supports could be useful in strengthening research with community-dwelling chronically ill persons.
Conclusion
Conducting research with a population with high levels of disability in a home setting is a complex undertaking and it requires knowledge of pertinent issues. Knowing more about the process fidelity related to the delivery of an intervention in a clinical trial can provide a more complete understanding of what might have impacted the delivery, how processes might have affected outcomes, and modifications needed for future interventions. Therefore, we recommend a comprehensive approach to process fidelity assessment in the use of both internal and external forms of evaluation, including some qualitative approaches, such as interviews with interventionists or eliciting comments from participants while the delivery is taking place. Importantly we believe that external assessment could be done less often because of the overlap and synergy in using both forms of evaluation. This would save time and expense, thus making the process of evaluating process fidelity of intervention delivery more feasible, efficient and comprehensive.
ACKNOWLEDGMENTS
The authors acknowledge the following: at the Visiting Nurse Service of New York, Center for Home Care Policy Research, Penny Feldman, PhD, Senior Research Scientist and Director Emeritus, Seon Lewis-Holman RN, MS-C, Director, Education and Clinical Development, and Yanick Martelly, RN, MSN, CWOCN; and at the School of Nursing, University of Rochester, Karen Farchaus Stein, PhD, RN, FAAN, Brody Endowed Professor, who kindly reviewed this paper.
FUNDING
Funding for this study was provided by the National Institute of Nursing Research, National Institutes of Health (U.S.) #R01 NR01553.
Footnotes
Disclosures: Dr. Wilde has been a consultant for NovaBay Pharmaceutical since June 2013. No other authors have anything to disclose.
Contributor Information
MARY H. WILDE, School of Nursing, University of Rochester, Rochester, New York, USA.
DIANNE LIEBEL, School of Nursing, University of Rochester, Rochester, New York, USA.
EILEEN FAIRBANKS, School of Nursing, University of Rochester, Rochester, New York, USA.
PAULA WILSON, Visiting Nurse Service of New York, Center for Home Care Policy and Research, New York, New York, USA.
MARGARET LASH, School of Nursing, University of Rochester, Rochester, New York, USA.
SHIVANI SHAH, Visiting Nurse Service of New York, Center for Home Care Policy and Research, New York, New York, USA.
MARGARET V. McDONALD, Visiting Nurse Service of New York, Center for Home Care Policy and Research, New York, New York, USA.
JUDITH BRASCH, School of Nursing, University of Rochester, Rochester, New York, USA.
FENG ZHANG, School of Nursing, University of Rochester, Rochester, New York, USA.
EILEEN SCHEID, Neurology-Neuroimmunology, Strong Memorial Hospital, University of Rochester Medical Center, Rochester, New York, USA.
JAMES M. McMAHON, School of Nursing, University of Rochester, Rochester, New York, USA.
References
- Bandura A. Self-efficacy: The exercise of control. W. H. Freeman; New York: 1997. [Google Scholar]
- Bodenheimer T, Grumbach K. Improving primary care: Strategies and tools for a better practice. McGraw-Hil; New York: 2007. [Google Scholar]
- Bourgeois I, Hart RE, Townsend SH, Gagné M. Using hybrid models to support the development of organizational evaluation capacity: A case narrative. Evaluation and Program Planning. 2011;34(3):228–235. doi: 10.1016/j.evalprogplan.2011.03.003. [DOI] [PubMed] [Google Scholar]
- Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Research in Nursing & Health. 2010;33(2):164–173. doi: 10.1002/nur.20373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Carpenter J, Burns D, Wu J, Yu M, Ryker K, Tallman E, Von Ah D. Strategies used and data obtained during treatment fidelity monitoring. Nursing Research. 2012;62(1):59–65. doi: 10.1097/NNR.0b013e31827614fd. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chen L, Chuang LM, Chang CH, Wang CS, Wang IC, Chung Y, Lai F. Evaluating self-management behaviors of diabetic patients in a telehealthcare program: Longitudinal study over 18 months. Journal of Medical Internet Research. 2013;15(12):e266. doi: 10.2196/jmir.2699. doi:10.2196/jmir.2699 [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Conley-Tyler M. A fundamental choice: Internal or external evaluation. Evaluation Journal of Australasia. 2005;4(1/2):3–11. [Google Scholar]
- El-Mallakh P, Chlebowy DO, Wall MP, Myers JA, Cloud RN. Promoting nurse interventionist fidelity to motivational interviewing in a diabetes self-care intervention. Research in Nursing & Health. 2012;35(3):289–300. doi: 10.1002/nur.21472. [DOI] [PubMed] [Google Scholar]
- Faw L, Hogue A, Liddle H. Multidimensional implementation evaluation of a residential treatment program for adolescent substance abuse. American Journal of Evaluation. 2005;26:7794. [Google Scholar]
- Gearing RE, El-Bassel N, Ghesquiere A, Baldwin S, Gillies J, Ngeow E. Major ingredients of fidelity: A review and scientific guide to improving quality of intervention research implementation. Clinical Psychology Review. 2011;31(1):79. doi: 10.1016/j.cpr.2010.09.007. [DOI] [PubMed] [Google Scholar]
- Hasson H, Blomberg S, Dunér A. Fidelity and moderating factors in complex interventions: A case study of a continuum of care program for frail elderly people in health and social care. Implementation Science. 2012;7(23):1–11. doi: 10.1186/1748-5908-7-23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Horner SD. Best practices for improving intervention fidelity that every nurse should know. Journal for Specialists in Pediatric Nursing. 2012;17(2):171–174. doi: 10.1111/j.1744-6155.2012.00327.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Liebel DV, Friedman B, Watson NM, Powers BA. Review of nurse home visiting interventions for community-dwelling older persons with existing disability. Medical Care Research and Review: MCRR. 2009;66(2):119–146. doi: 10.1177/1077558708328815. doi:10.1177/1077558708328815 [doi. [DOI] [PubMed] [Google Scholar]
- Liebel DV, Powers BA. Home health care nurse perceptions of geriatric depression and disability care management. The Gerontologist. 2013 doi: 10.1093/geront/gnt125. doi:gnt125 [pii] [DOI] [PubMed] [Google Scholar]
- Liebel DV, Powers BA, Friedman B, Watson NM. Barriers and facilitators to optimize function and prevent disability worsening: A content analysis of a nurse home visit intervention. Journal of Advanced Nursing. 2012;68(1):80–93. doi: 10.1111/j.1365-2648.2011.05717.x. doi:10.1111/j.1365-2648.2011.05717.x [doi] [DOI] [PubMed] [Google Scholar]
- Markle-Reid M, McAiney C, Forbes D, Thabane L, Gibson M, Browne G, Busing B. An interprofessional nurse-led mental health promotion intervention for older home care clients with depressive symptoms. BMC Geriatrics. 2014;14:62–2318-14-62. doi: 10.1186/1471-2318-14-62. doi:10.1186/1471-2318-14-62 [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Metzelthin SF, Daniëls R, van Rossum E, Cox K, Habets H, de Witte LP, Kempen GI. A nurse-led interdisciplinary primary care approach to prevent disability among community-dwelling frail older people: A large-scale process evaluation. International Journal of Nursing Studies. 2013;50(9):1184–1196. doi: 10.1016/j.ijnurstu.2012.12.016. doi:10.1016/j.ijnurstu.2012.12.016. [DOI] [PubMed] [Google Scholar]
- Miles M, Huberman A. Qualitative data analysis. 2nd ed. Sage; Newbury Park, CA: 1994. [Google Scholar]
- Nicolaides-Bouman A, van Rossum E, Habets H, Kempen GI, Knipschild P. Home visiting programme for older people with health problems: Process evaluation. Journal of Advanced Nursing. 2007;58(5):425–435. doi: 10.1111/j.1365-2648.2007.04235.x. doi:10.1111/j.1365-2648.2007.04235.x. [DOI] [PubMed] [Google Scholar]
- Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Health services research: Process evaluation in randomised controlled trials of complex interventions. BMJ: British Medical Journal. 2006;332(7538):413. doi: 10.1136/bmj.332.7538.413. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Radziewicz RM, Rose JH, Bowman KF, Berila RA, O'Toole EE, Given B. Establishing treatment fidelity in a coping and communication support telephone intervention for aging patients with advanced cancer and their family caregivers. Cancer Nursing. 2009;32(3):193–202. doi: 10.1097/NCC.0b013e31819b5abe. doi:10.1097/NCC.0b013e31819b5abe; 10.1097/NCC.0b013e31819b5abe. [DOI] [PubMed] [Google Scholar]
- Resnick B, Inguito P, Orwig D, Yahiro JY, Hawkes W, Werner M, Magaziner J. Treatment fidelity in behavior change research: A case example. Nursing Research. 2005;54(2):139–143. doi: 10.1097/00006199-200503000-00010. [DOI] [PubMed] [Google Scholar]
- Robb SL, Burns DS, Docherty SL, Haase JE. Ensuring treatment fidelity in a multi-site behavioral intervention study: Implementing NIH behavior change consortium recommendations in the SMART trial. Psycho-Oncology. 2011;20(11):1193–1201. doi: 10.1002/pon.1845. doi:10.1002/pon.1845; 10.1002/pon.1845. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nursing Research. 2004;53(1):63–66. doi: 10.1097/00006199-200401000-00010. [DOI] [PubMed] [Google Scholar]
- Scheirer MA. Designing and using process evaluation. In: Wholey HS, Hatry H, Newcomer K, editors. handbook of practical program evaluation. Josey Bass; San Francisco: 1994. pp. 40–68.pp. 40–68. [Google Scholar]
- Schinckus L, Van den Broucke S, Housiaux M, Diabetes Literacy Consortium Assessment of implementation fidelity in diabetes self-management education programs: A systematic review. Patient Education and Counseling. 2014;96(1):13–21. doi: 10.1016/j.pec.2014.04.002. doi:10.1016/j.pec.2014.04.002. [DOI] [PubMed] [Google Scholar]
- Sidani S. Measuring the intervention in effectiveness research. Western Journal of Nursing Research. 1998;20(5):621–635. doi: 10.1177/019394599802000508. [DOI] [PubMed] [Google Scholar]
- Spillane V, Byrne MC, Byrne M, Leathem CS, O'Malley M, Cupples ME. Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. Journal of Advanced Nursing. 2007;60(3):343–352. doi: 10.1111/j.1365-2648.2007.04386.x. doi:10.1111/j.1365-2648.2007.04386.x. [DOI] [PubMed] [Google Scholar]
- Stijnen MM, Jansen MW, Duimel-Peeters IG, Vrijhoef HJ. Nurse-led home visitation programme to improve health-related quality of life and reduce disability among potentially frail community-dwelling older people in general practice: A theory-based process evaluation. BMC Family Practice. 2014;15(1):173. doi: 10.1186/s12875-014-0173-x. doi:s12875-014-0173-x [pii] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wilde MH, Zhang F, Fairbanks E, Shah S, McDonald MV, Brasch J. Perceived value of a urinary catheter self-management program in the home. Home Healthcare Nurse. 2013a;9:465–473. doi: 10.1097/NHH.0b013e3182a89791. doi: 10.1097/NHH.0b013e3182a89791. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wilde MH, McDonald MV, Brasch J, McMahon JM, Fairbanks E, Shah S, Scheid E. Long-term urinary catheter users self-care practices and problems. Journal of Clinical Nursing. 2013b;22(3-4):356–367. doi: 10.1111/jocn.12042. doi: 10.1111/jocn.12042. [DOI] [PMC free article] [PubMed] [Google Scholar]