Enhanced fidelity of an educational intervention on skin self-examination through surveillance and standardization

Abstract

Background

Melanoma can metastasize but is often successfully treated when discovered in an early stage. Melanoma patients and their skin check partners can learn skin self-examination (SSE) skills and these skills can be improved by practice. The purpose of this study is to determine the degree of fidelity with which educational in-person SSE intervention can be delivered by trained research coordinators to patients at risk of developing another melanoma and their skin check partners.

Methods

The in-person intervention was performed in two iterations. In phase 1 (2006-2008), the research coordinators were trained to perform the intervention using a written script. In phase 2 (2011-2013), the research coordinators were trained to perform the intervention with a PowerPoint aid. Each research coordinator was individually counseled by one of the authors (KM) to insure standardization and enhance fidelity of intervention delivery. Phase 1 and Phase 2 were compared on 16 fidelity components. Further, Phase 2 fidelity was assessed by comparing mean scores of fidelity across the five research coordinators who delivered the intervention.

Results

Phase 2, which utilized a PowerPoint aid, was delivered with a higher degree of fidelity compared to phase 1with four fidelity components with significantly higher fidelity than Phase 1: 1) Explained details of melanoma, χ² (1, n = 199)= 96.31, p < .001, 2) Discussed when to call doctor, χ² (1, n = 199) = 53.68, p < .001 3) Explained assessment at month 1, χ² (1, n = 199)= 12.39, p < .01, and 4) Explained assessment at month 2, χ² (1, n = 199) = 117.75, p < .001. Further, no significant differences on mean fidelity were found across research coordinators in Phase 2.

Discussion

When using the PowerPoint aide, the research coordinators delivered the intervention with high fidelity (all scores >14) and there were no mean differences in fidelity across research coordinators, indicating consistency in fidelity. This can be attributed to the standardization and cueing that the PowerPoint program offered. Supervision was also a key component in establishing and maintaining fidelity of the patient educational process. This method of intervention delivery enables trained healthcare professionals to deliver an educational intervention in an effective, consistent manner.

1 Introduction

Self-management of health is fundamental in controlling chronic conditions, yet the fidelity of education and support given to enable patients to obtain greater independence of management are seldom evaluated ^[1]. In-person interventions educate individuals about their health issue and encourage behavioral changes they can make to better manage a variety of chronic conditions, including venous leg ulcers and diabetes which require daily attention [^{2, 3}]. Interventions are most effective when they are standardized ^[2]; however, little research has examined the fidelity of standardized in-person educational interventions aimed at early detection of melanoma.

Melanoma is a serious skin cancer that can metastasize but is often successfully treated when discovered at an early stage. In those diagnosed with Stage IA melanoma, the survival rate 10 years after diagnosis was estimated as higher than 95% and declined to less than 60% when diagnosis occurred later (Stage IIB, C). People who are diagnosed later rapidly progress to the advanced stage of the disease with metastasis to internal organs (Stage IV). Previous research has found that melanoma patients who conduct deliberate skin self-examinations (SSE's) are more likely to survive than those who discover melanoma incidentally [^{4, 5}]. Therefore, enhanced surveillance for melanoma patients has the greatest potential to detect future melanomas in their earliest stages where treatment prognosis is optimal. Robinson found that melanoma patients and their skin check partners can learn the SSE skills and that these skills can be improved by practice ^[6]. This kind of education is essential for early detection and prevention of potential further spread of the cancer. However, due to increased patient load and the short duration of clinic visits, there are barriers for health care providers to deliver an educational intervention on SSE. Past research has shown that healthcare professionals other than the physician can effectively deliver interventions on SSE. For example, a study of nurse-delivered interventions with melanoma patients on conducting SSEs found that a brief teaching session increased the patients’ adherence to completing the SSEs over 50% during a four month period ^[7].

Ensuring interventions are delivered in a consistent, standardized and accurate manner to participants is essential to the interpretability of intervention outcomes. Specifically, measuring the fidelity, which is the ability to deliver the intervention as planned with accuracy, is important because it assesses the standardization of the delivery across deliverers of the intervention [^{8, 9}]. The purpose of this study is to determine the fidelity of a patient/partner SSE educational intervention delivered by trained research coordinators

2 Methods

Participant accrual

The study was conducted in the outpatient clinics of the Northwestern Memorial Faculty Foundation in Chicago, Illinois where researchers had access to patients, who had previously been diagnosed with melanoma. Prior to their appointments at the clinic, melanoma patients with Stage 0 to IIB disease were identified by review of electronic medical records. The inclusion criteria were having a history of Stage 0 to IIB melanoma and being at least 6 weeks after surgical treatment of melanoma, able to see to read a newspaper, fluent in English, age 21-80, and having a significant other person (spouse, partner, close relative) who was willing to participate in the research. Exclusion criteria were a history of Stage III or greater melanoma; ocular, genital, and oral melanoma; being overburdened with co-morbid disease, unable to see to read a newspaper, not fluent in English, unable to participate in conversation at a sixth grade language level due to cognitive impairment, and did not have a significant other person (spouse, partner, close relative) who was willing to participate in the research. At the visit with their dermatologists and surgical oncologists, research coordinators interviewed eligible patients, explained the research and made an appointment for the initial study visit. The Institutional Review Board of Northwestern University approved the study.

Training and cues to delivering the educational intervention

The in-person educational intervention was performed in two iterations. In phase 1 (2006-2008), the Northwestern (NW) research coordinators were trained to perform the intervention using a written script and each coordinator participated in three role playing sessions. During the intervention, the coordinator was cued by the ABCDE rule as presented on the card given to the participants and showed 2 additional color printed examples demonstrating evolution of pigmented lesions in one year to the participants (patient with melanoma and partner), answered questions about the content, gave a skills demonstration using a magnifying hands lens to look at moles and pointed out irregular borders and uneven colors, and administered a pre- and post-skills training quiz to assess the immediate efficacy of the intervention and to promote self-efficacy. At the conclusion of the intervention, the research assistant demonstrated how to use the enabling kit consisting of the card with the ABCDE rule, a lighted hand held magnifying glass, and a set of body maps to use as a diary to record areas of concern found during the monthly SSE. The kit was given to the pair and an appointment for a visit with the dermatologist (JR) was made in 4 months ^[6]. Over the two-year course of phase 1, two research coordinators (both female college graduates) performed interventions with 65 participant pairs (see Table 1, N = 65 pairs).

Table 1.

Comparison of Fidelity Components Across Phases

Fidelity Component	Phase 1 (N = 65 pairs) n (%)	Phase 2 (N = 134 pairs) n (%)	χ 2
Introduced topics for discussion	52 (80)	116 (86.6)	1.435
Inquired about couples knowledge of melanoma	33 (51)	75 (56.0)	0.477
Explained details of melanoma	18 (28)	126 (94.0)	96.309**
Explained the likelihood of developing melanoma	59 (91)	127 (94.8)	1.151
Explained aspects of moles/importance of partner examination	62 (95)	128 (95.5)	0.002
Introduced ABCDE rule	63 (97)	128 (95.5)	0.223
Explained “A” assess	63 (97)	128 (95.5)	0.223
Explained “B” border	63 (97)	131 (97.8)	0.109
Explained “C” color	65 (100)	131 (97.8)	1.477
Explained “D” diameter	65 (100)	131 (97.8)	1.477
Explained “E” evolution	65 (100)	131 (97.8)	1.477
Discussed when to call doctor	39 (60)	132 (98.5)	53.679**
Introduced Body Map	65 (100)	131 (97.8)	1.477
Explained assessment at month 1	55 (85)	131 (97.8)	12.388*
Explained assessment at month 2	17 (26)	131 (97.8)	117.748**
Addressed and engaged both patient and partner during presentation	65 (100)	132 (98.5)	0.98

Note.

^*p < .01

^**p <.001

In phase 2 (2011-2013), the intervention was expanded to include a system of scoring the border, color and diameter with a supporting scorecard diary ^[10]. The research coordinators were trained to perform the intervention using a written script and each coordinator participated in three role playing sessions. During the role playing sessions, the lack of consistent delivery was observed by the research team. A PowerPoint presentation was developed for delivery with a laptop computer (Dell Inspiron N5110) and a 12 × 16 inch monitor placed onto the conference room table to show patients and partners the images and interactive elements, thus, cueing the delivery of the educational intervention. Over the two-year course of accrual of participants in phase 2, five research coordinators (2 female college graduates and 3 males having master degrees in health related fields) performed interventions with 134 participant pairs (see Table 1, N = 134 pairs).

Thus, with the exceptions of the scorecard diary and the PowerPoint presentation delivery aid in phase 2, the other elements of the intervention remained essentially the same as phase 1. This consistency permitted a formal comparison of the fidelity across the two phases.

Monitoring fidelity of delivery of the intervention

In phase 1, the dermatologist observer (JR) remotely monitored the delivery of the intervention by viewing and listening to the research coordinator. As the intervention was delivered, the observer indicated on a 16-item checklist if the task was completed adequately (see Table 1). After observing five interventions, the dermatologist individually counseled the research coordinator about how to better deliver the intervention in a standardized manner with fidelity.

In phase 2, the research coordinators audio recorded intervention sessions with patients and partners. During the first several weeks of phase 2, audio recordings were coded by an independent team of three researchers at Pennsylvania State University using the 16–item checklist (see Table 2). Inter-rater reliability between the coders was high (r = .95). Following the analyses of these initial intervention sessions, NW research coordinators were individually counseled by one of the lead study authors at Penn State (KM) on how to better deliver the intervention in a standardized manner with fidelity. Each session took approximately 30 minutes and consisted of addressing research coordinators’ questions and providing feedback on areas of potential improvement. All subsequent phase 2 intervention sessions were coded in the same manner and NWresearch coordinators were provided with feedback over the two years on an ad hoc basis to insure standardization and fidelity.

Table 2.

Mean Fidelity Scores of the Research Coordinators in Phase 2

	Research Coordinator
	1	2	3	4	5
Mean	15.41	15.47	14.68	14.68	15.20
SD	1.85	0.87	3.20	1.33	0.42
N	32	17	38	37	0.42

Data analyses

First, phase 1 and phase 2 intervention delivery was compared on each of the sixteen fidelity components (1= delivered, 0 = not delivered) using chi-square analyses. Second, fidelity between the research coordinators in phase 2 was examined for differences. The first step in this process entailed scoring the sixteen fidelity components (1= delivered, 0 = not delivered) and deriving a total score for each intervention delivered. The second step involved deriving a mean fidelity score for each research coordinator. These mean fidelity scores ranged from 0 to 16 and signified the average number of components delivered during an intervention for a given research coordinator. The final step was to examine differences between mean fidelity scores for the 5 research coordinators using the F test in a one-way ANOVA.

3 Results

Although no significant differences were found for 12 of the 16 components, four components were delivered significantly more often in Phase 2: 1) Explained details of melanoma, 2) Discussed when to call doctor, 3) Explained assessment at month 1, and 4) Explained assessment at month 2 (see Table 1).

There were no significant differences between the mean fidelity scores, F (4, 129) = 0.98, p > .05 (see Table 2). Further, the mean fidelity scores were universally high across the research coordinators (all mean scores > 14). A non-significant F value and high mean fidelity scores (e.g., mean scores > 14) are reflective of high implementation standardization and fidelity.

4 Discussion

This study demonstrated that the use of a PowerPoint program as a delivery aid added structure and increased fidelity of the in-person SSE intervention. Continuing supervision and feedback to those who delivered the intervention over the entire course of the intervention (phase 2) may have helped maintain consistent fidelity. In phase 2, fidelity was consistently high among all the research coordinators, with mean scores all above 14 out of 16. Further, interventions delivered with the aid of the PowerPoint presentation were more consistently delivered, especially components that explained details of melanoma, discussed when to call the doctor, and information about performing one and two month assessments. Consistent with past studies, these findings suggest that a well-defined highly structured program can be delivered by less experienced staff ^[11].

The greater proficiency in the delivery of the elements in phase 2 may also be attributed to the longer period of supervision and individual feedback. This finding is consistent with prior studies that established ongoing supervision of trainees successfully sustained proficiency for deliverers of motivational interviewing for alcoholism [^{12, 13}]. Further research should identify the optimal timing for how long and/or how often supervision and feedback is necessary to keep high fidelity of intervention delivery. In addition, future studies should assess how fidelity affects actual patient and partner SSE behavior.

5 Future directions

As the prevalence of melanoma continues to rise, healthcare professionals will be confronted with how to best deliver quality care and assessment in the out-patient clinical setting. Using the nursing process as a model of practice, the nurse can assess, plan and implement the SSE intervention. Nurses have the opportunity to identify risk factors and assess patients’ level of understanding of SSE and willingness and ability to perform SSE. If the assessment indicated the patient and partner (if available) were willing and able to learn the skill, the nurse could determine if it was appropriate to demonstrate SSE, re-state the instruction, and provide a quiet, safe environment for the patient and partner to ask questions or verbalize concerns ^[14].

In-person interventions have the potential to vary widely based on the interaction between the deliverer of intervention and the patient(s), the concerns of the patient(s), and what components are actually discussed. This study demonstrates the benefit derived from using a PowerPoint presentation as an aid to deliver and reinforce education in health care matters. The global acceptance of electronic medical records has placed computers in most clinical examining rooms. Now that the computers are available, they can be utilized to improve health care by assuring consistent delivery of education interventions. The current study also emphasized the importance of supervision to establish and maintain fidelity of the patient educational process.