The Development and Validation of a tool to assess nurse performance in dual protection counseling

Abstract

Objective

To present preliminary evidence for the reliability and validity of the Dual Protection Counseling Checklist (DPCC), an instrument designed to evaluate nurses’ fidelity to high quality dual protection counseling in a family planning setting.

Methods

During a trial comparing a dual protection (DP) nurse counseling intervention to standard of care (SOC), client-counselor sessions were audiotaped. Following good inter-rater reliability, 78 audiotaped interviews were coded from the two conditions using the DPCC. We constructed indices from a subset of codes to capture three domains: Promotion of DP (PDP), Relapse Prevention Counseling (RPC), and Quality of Nurse-Client Interaction (QNCI). The association between scores on these indices and client outcomes was evaluated using logistic regression.

Results

The DPCC and indices were reliable. Construct validity of indices was supported by greater frequency of target behaviors by the DP nurse. Validity of the QNCI was further supported by its association with clients’ sexual risk reduction six months post counseling.

Conclusion

The DPCC and indices hold practical utility for evaluation, monitoring, and supervision of nurse-client counseling sessions.

Practice Implications

The Dual Protection Counseling Checklist provides a user-friendly tool for assessing nurses’ and other providers’ counseling behaviors in dual protection.

1. Introduction

The integration of family planning (FP) and sexually transmitted infection (STI) services is rapidly gaining momentum in reproductive health following the 1994 International Conference on Population and Development in Cairo, Egypt. Family planning providers are seen as well-positioned to assume a more influential role in increasing women’s safer sex practices and thereby reducing the rate of STIs, including HIV. In the United States (US) as well, there has been an effort, supported by Title X funding, to incorporate STI diagnosis, treatment, and prevention into traditional family planning settings.

Dual protection (DP) counseling is an emergent and innovative approach to client-counseling in which the counselor aims to address simultaneously a woman’s need for protection against both unintended pregnancy and HIV/STIs. It extends the physical integration of family planning and STI-prevention services under one roof to include an integrated conceptual approach to client-counseling. Dual protection counseling is more directive than traditional family planning options counseling in that it seeks to help women identify and address risks for disease that they may be reluctant to acknowledge. Counselors indicate that DP can be achieved in three ways: by use of a male or female condom alone; by use of two methods including a condom and some non-barrier contraceptive; or by use of an effective contraceptive in the context of long-term mutual monogamy [1], and they help women determine which approach best addresses their needs.

To date, efforts to integrate family planning and STI services have been evaluated via observations of provider-client interactions and client exit interviews to determine providers’ utilization of selected behaviors, and have been guided by models of ideal client-provider interactions and quality of care in family planning settings [2,3]. Standardized checklists require the observer to assess whether or not the providers engaged in the target behaviors during their consultations with clients. Although several checklists are available, for example, the 4HCS [4] and MISC [5], they require direct observation of counseling, can be complex and time-consuming to code, were not specifically designed to examine DP counseling, and evidence of validity– specifically, whether they assess behaviors that meaningfully relate to clients’ behavioral response to counseling– are scant.

In this paper, we present the results of an evaluation of provider-client interactions from a proof-of-concept Phase II trial of DP counseling. The DP counseling in this study was a one-session intervention conducted by a FP nurse provider who was trained in the counseling technique, and was compared to standard of care counseling provided by nurses in four study clinics in New York City. We describe the development of a dual protection counseling coding checklist used to evaluate nurse performance, testing of the instrument for reliability, use of the instrument to evaluate nurse performance in the trial, and the association between coded behavior with clients’ subsequent sexual risk reduction, an indicator of the validity of the indices.

2. Methods

2.1 Study Overview and Research Setting

“Project REACH” – Research and Education About Contraceptive Health – was a Phase II trial that tested the potential efficacy of an individualized, clinic-based, provider-delivered intervention designed to increase adoption and maintenance of DP. The study was undertaken collaboratively with the Community Healthcare Network (CHN), a New York City consortium of clinics offering family planning, reproductive health, pediatric, general medical and HIV primary care services in medically under-served areas. The study was conducted in four of the nine clinics, with two sites located in Brooklyn, one in the Bronx, and one in Manhattan.

2.1.1 Clients

Clients enrolled in the study were 101 low-income, primarily African American, Caribbean, and Latina 15–32 year-old HIV-seronegative women. Eligible and interested participants completed informed consents prior to study participation. To be eligible for the study, women had to report at least one occasion of vaginal intercourse in the preceding three months that was not protected by male or female condoms and be attending the clinic for routine (6-month or yearly) counseling and check-up. Study recruitment efforts were spread across the four clinics. Four hundred and eighty-two clients were approached in the waiting areas of the respective study sites and escorted to a private office or a more private section of the waiting area to be screened for eligibility. Of the 482 clients, 306 were found to be ineligible and 75 were eligible but declined study participation. The study participation rate was 57%. Clients were assigned to condition based on day of visit to see either the study (Risk and Decision-Making, RAD) nurse, who rotated on a 4-month schedule to each of the clinics, or one of the SOC nurses available at that clinic. Clients were evaluated pre- and post-nurse counseling, and 6 months later using a structured assessment instrument. Assessment of sexual risk behavior and contraceptive use relied on an adaptation of the Sexual Risk Behavior Assessment Schedule for Adults (SERBAS-A; [6]), an instrument with demonstrated reliability in diverse populations [7–12]. Areas assessed included number of partners, number of occasions of vaginal intercourse, and number of vaginal intercourse occasions protected by male or female condoms, as well as use of other contraceptive methods. The main outcome was whether clients had reduced the number of condom unprotected sex occasions between baseline and 6-month follow-up, measured using a dichotomous variable of “0” if women reported no change or an increase in the number of unprotected sex occasions, and “1” if women reported a decrease in the number of unprotected sex occasions. Assessments were via face-to-face interview by carefully trained and supervised field interviewers. Institutional Review Board approval was obtained from the New York State Psychiatric Institute and the Community Healthcare Network.

2.1.2 Nurses

Fifteen Licensed Practical nurses (LPNs) already working at the clinics, were included in the study. Of the fifteen, 11 self-identified as African-American or Black, two as Latina, one as White, and one as Asian Pacific Islander. Only one of the 15 nurses, who was Black, was trained in Risk and Decision-Making (RAD) counseling. As the salary of the nurse administering this untested counseling approach was partially supported by grant funds, it was only possible to include one RAD nurse. The choice of the RAD nurse was not random, but based on pre-implementation discussions with the host agency. The criteria for selection were the nurse’s willingness to participate in a study and openness to try a different form of counseling.

2.1.3 Counseling

The RAD is a manualized intervention that is DP-focused and grounded in motivational interviewing [13] and relapse prevention [14]. The study nurse was extensively trained prior to launching the study. HIV/STI risk assessment followed a standard protocol, with normalizing prefaces and targeted probes designed to increase client comfort and enhance evaluation of risk. During method selection, the study nurse used a semi-structured counseling and decision-making guide to help a client simultaneously select both a method of contraception and HIV/STI prevention. The client’s values framed the decision-making process, and provided both direction for the nurse provider and a counterweight to the nurse’s own values [15,16]. Once a preferred option was identified by the client during the consultation, the study nurse discussed the positive and negative aspects of use. She also helped clients anticipate difficulties they might encounter in consistently and correctly using the selected method and helped them brainstorm solutions. These challenges and solutions were incorporated into “Action Plans”; the client left with a written copy of her individualized plan. At the end of the session, the study nurse also scheduled clients for subsequent phone or in-person follow-up appointments to reinforce method use.

The remainder of the clinic nurses followed the clinic’s standard of care (SOC). The SOC policy in the clinics required that all clients seen for their initial, semi-annual, or annual exam visits be screened for HIV risk by the nurse, using a structured checklist that conformed to New York State mandated pre-test counseling guidelines. The checklist assessed risks of clients and their partners with response options of “Yes”, “No” or Do not Know (DK). Contraceptive and STI-related needs of the clients also were addressed at each visit. SOC nurses did not use a standard protocol when counseling clients on method selection and STI risk. All nurses in the study were instructed to audiotape all their counseling sessions with study participants. Seventy-eight counseling sessions (42 SOC and 36 RAD collected between June 2001 and April 2004) were successfully audiotaped with adequate quality for scoring/coding.

2.2 Instrument Development

2.2.1 Dual Protection Counseling Checklist (DPCC)

We developed the Dual Protection Counseling Checklist (DPCC) instrument to evaluate nurse-client interactions through observation or listening to audiotapes of sessions in a family planning setting. It was based on parameters of contraceptive decision-making identified by Kim, Kols, and Mucheke [2], on Bruce’s [3] quality of family planning services framework, and on items adapted from a counseling checklist for use in a Nigerian study [17]. The DPCC comprises 175 checklist items across six domains: (1) introduction to counseling, which includes the manner in which nurses greeted clients and determined their reasons and expectation for the visit; (2) risk assessment of clients and their partner(s), which includes the extent to which providers reviewed clients’ medical histories, evaluated clients’ and partner(s)’ HIV/STI and pregnancy risk, and probed personal circumstances; (3) motivational assessment, in which providers review clients’ previous history with pregnancy prevention methods and determined if clients were currently using a method to prevent STI; (4) method review, whereby providers discuss and review with clients the advantages, disadvantages, and side effects of contraceptive methods, their history or experiences with the method(s) selected, whether clients were taught how to use the method(s) selected appropriately, and whether providers explored reasons for particular method(s) selected; (5) problem-solving solutions, in which providers assist clients with solutions for any problems they encountered around method use, their ability to introduce or re-introduce method(s) to their partner(s), and whether providers schedule clients to return for a re-supply or check-up; and (6) overall nurse performance, in which the nurse providers’ overall performance and the clients’ input in the consultative process are assessed. For each item the scoring options are “yes” if the behavior was present, “no” if the behavior was absent. We attempted to avoid the asymmetrical design of many approaches towards coding communication during clinically-based interventions by simultaneously analyzing the content of counseling and the level of communication between the provider and clients [18]. Asymmetrical study designs only assess client input as a response to provider expressions or only account for provider contributions. This method does not equally qualify client utterances. The DPCC assessed not only what a client might say in response to a provider’s “monologue,” which is an unfair representation of an actual medical exchange [18], but contains items that span a variety of client communication behaviors– e.g., codes were included about the types of questions a client may ask, what kind of information she volunteered, or whether she asked to skip any part of the counseling session.

2.2.2 Reliability

Six coders piloted the scoring of the DPCC by using a random sample of 14 audiotape recorded interviews (seven RAD and seven SOC). A joint review process of this instrument helped to develop and refine scoring rules and to produce a procedure manual for using the DPCC. Next, in order to assess reliability, two of the six coders independently rated ten randomly selected audiotaped interviews from the two nurse conditions. The Intraclass Correlation (ICC) for the total nurse performance score using scores for all 175 items was .976, using a two-way random effects model. Among the 175 items, we identified 65 items endorsed on at least 30% of the audiotapes that captured key elements of DP counseling. Inter-rater reliability of the 65 verbal behavior items ranged from (Cronbach α= −.32 to 1.00). Thirty of 65 items (46.1%) had a Kappa of .75 or greater; of these items, 18 (27.7%) scored perfect agreement, and 17 (26.2%) had satisfactory to very good agreement (with scores between .40 and .75)[19]. The median Kappa was .78. Eleven items (17%) had a poor level of agreement, with Kappas below .40. These were not used in constructing face-valid indices described below, because of poor inter-rater reliability and because they also overlapped with other more reliable items. After reliability of the DPCC was established, the nurses’ audiotaped interviews were manually coded by two trained masters- level research assistants and the Project Director, and data were entered into an SPSS database.

2.3 Reliability Testing

2.3.1 Face Valid Indices

Through a consensus process that involved a careful review of the reliable DPCC items, the authors constructed face valid indices by selecting items to reflect three DP counseling elements – Promotion of DP incorporating MI techniques (PDP), Relapse Prevention Counseling for dual protection (RPC), and Quality of Nurse-Client Interaction (QNCI). After selecting the items, internal consistency reliability was calculated using Cronbach’s alpha. [See Table 1 for items included in each index and internal consistency statistics]. The PDP index, comprising two items (Alpha .89), reflects whether the nurse discussed methods of STI prevention and described how DP can be achieved. There were three RPC items (Alpha .80), reflecting whether the nurse helped the client to identify barriers to method use, to formulate an action plan, and whether a follow-up appointment was made. Items on the QNCI (Alpha .81) attempted to capture nurses’ use of counseling techniques that can facilitate client engagement (use of open-ended questions, reinforcement, empathic responses and a non-judgmental attitude), and client behaviors indicative of engagement (sharing concerns, expressing opinions and volunteering pertinent information). We also tested the nurse-defining and client-defining items within the QNCI separately for reliability and found the nurse-defining index (Alpha .65) to be slightly low but acceptable given that the instrument is new, and the client-defining item (Alpha .79) to be reliable.

Table 1.

Study indices and respective DPCC items

Indices	Index Items/Questions	Range	Alpha
Promotion of Dual Protection (PDP) incorporating MI techniques	Did the provider: Discuss how STI prevention can be achieved by abstinence, consistent condom use, or mutual monogamy Discuss how DP can be achieved by the use of one barrier method or two methods	0–2	.89
Relapse Prevention Counseling (RPC) for DP	Did the provider: Help the client identify possible problems in using selected methods Ask the client to formulate an “action plan” for ensuring correct and consistent method use Attempt to schedule a telephone or in-person or action plan	0–3	.80
Quality of Nurse- Client Interaction (QNCI)	Did the provider: Ask open-ended questions Provide reinforcement to the client Give empathetic responses Have a non-judgmental attitude	0–4	.65
	Did the client: Express worry or concern about pregnancy risk Express worry or concern about STD risk Volunteer relevant information about her personal situation Volunteer relevant information about her medical situation Express her own ideas or preferences about a method Give thorough answers	0–6	.79

2.4 Data Analysis

We addressed the following research questions: (1) Were there baseline differences by condition among the 78 clients included in this analysis; (2) Was there a difference in quality of nurse-client interaction, frequency of discussion of DP, and utilization of RP elements when comparing the counseling behavior of the RAD nurse versus the SOC nurses; and (3) To what extent did quality of nurse-client interaction, discussion of DP and use of RP elements predict safer sex behavior among clients?

To assess the first question we conducted chi-squared tests to compare clients in the two groups on demographic characteristics. To address the second question, we conducted independent samples t-tests or chi-squares (for continuous and categorical variables, respectively) to assess differences in the utilization of index-defining behaviors between the nurse trained in the RAD and those who engaged in SOC counseling. To address the third question, we examined whether there was an association between nurse behavior and client outcome, controlling for condition. As each nurse can potentially serve multiple participants, the outcomes were correlated for those participants served by the same nurse. To account for such correlation, we used the Generalized Estimating Equation (GEE) methodology with a log link function.

3. Results

3.1. Baseline Client Characteristics

As shown in Table 2, in this analysis there were no significant differences between clients who received RAD versus SOC counseling.

Table 2.

Demographic Characteristics of the clients by group (n=78)

Variable	Statistics
	Control N=42 (%)	Intervention N=36 (%)	Chi-Square or t-test	df	p-value
*Age
Mean	22	23	−1.08	76	.55
Race/Ethnicity
African-American/Black	15 (50)	15 (50)	1.57	3	.67
Caribbean/Caribbean American	16 (64)	9(36)
Latina/Hispanic	8 (47)	9 (53)
White/API/Other	3 (50)	3 (50)
Marital Status
Single/Never Married	32 (50)	32 (50)	2.53	2	.28
Married/Common Law	7 (78)	2 (22)
Separated/Divorced	3 (60)	2 (40)
Level of Education
Less than High school	13 (65)	7 (35)	1.63	2	.44
Completed High School/GED	14 (54)	12 (46)
At least 1 year College	15 (47)	17 (53)

^*A t-test was used to calculate the mean difference in age between intervention and control group participants.

3.1.1. Nurses’ Utilization of Defining Behaviors

As shown in Table 3, mean scores for the RAD nurse were significantly higher than that of the SOC nurses across all indices, indicating that on average the RAD nurse used more of the index-defining behaviors in her consultations than the SOC nurses. The median number of audiotaped consultations per SOC nurse was 3, with a range of 1–20.

Table 3.

Comparison of Mean scores (and standard deviations) for nurse indices by condition

NURSE TECHNIQUE INDICES	STUDY CONDITIONS
	RAD		SOC
	Mean	sd	Mean	sd	t (df)
Promotion of DP (PDP)	.75	.36	.09	.29	−8.69* (76)
Relapse Prevention Counseling (RPC)	.73	.37	.06	.23	−9.61* (76)
Quality Nurse Client Interaction (QNCI)	7.61	1.64	3.33	2.01	−10.19*(76)

^*p < .001

The RAD nurse addressed both pregnancy and STI prevention (the behaviors defining PDP) with a significantly greater proportion of clients than did SOC nurses (64% versus 10%). Only three (21.4%) of the 14 SOC nurses exhibited at least one of the PDP-defining behaviors. Of these three nurses, one exhibited such behavior with 66% of clients (N=3), one with 50% of clients (N=2), and one with 5% of clients (N=20). The RAD nurse used all three RPC-defining behaviors with 56% of clients, compared to SOC nurses, who used all three behaviors with only 5% of clients. Of the 14 SOC nurses, five of them (36%) used at least one of the techniques with clients; three (21.4%) did so with only one client each, and two (14.3%) others did so with two clients each.

The RAD nurse also demonstrated greater utilization of the four defining QNCI behaviors (mean=3.47 out of a possible 4) when compared to the SOC nurses (mean=1.90; t= −7.60, df= 76, p<.001; See Table 4). Clients of the RAD nurse also evidenced more of the six behaviors in the QNCI (mean=4.14 out of a possible 6) as compared to the SOC clients (mean=1.43; t=−8.11, df=76, p<.001). In comparing the mean total score of ten items for the quality of nurse-client interaction for the RAD nurse versus the SOC nurses, on average the RAD nurse outperformed the SOC nurses by more than two to one, mean=7.61 and 3.33 respectively, t=−10.19, df=76, p<.001.

Table 4.

Comparison of nurse providers’ utilization of index-defining behaviors

Indices	Condition		Chi-Square/df*
	RAD (N=36) %	SOC (N=42) %
Promotion of DP (PDP) using MI techniques			46.51 (2)*
1 Defining behavior	22%	0%
Both Defining behaviors	64%	10%
Relapse Prevention Counseling (RPC)			60.62 (3)*
1 Defining behavior	14%	12%
2 Defining behaviors	31%	0%
All 3 defining behaviors	56%	5%
Quality Nurse-Client Interaction (QNCI)	Mean	Mean	t-test/df
# Client behaviors present of 6	4.14	1.43	−8.11 (76)*
# Nurse behaviors present of 4	3.47	1.90	−7.60 (76)*
Total score, 10 items	7.61	3.33	−10.19 (76)*

^*p < .001

Theoretically, conducting DP counseling with a client (PDP), using relapse prevention techniques (RPC), and quality of nurse-client interaction (QNCI) should have a positive impact on client’s future sexual risk behavior. In order to examine this, we conducted analyses to determine the relationship between the indices and client risk behavior six months later (i.e., whether or not the client had reduced unprotected vaginal intercourse). Findings indicate that the client-defining and nurse-defining index behaviors of the QNCI were significantly associated with clients’ reduction of sexual risk behavior 6 months post nurse-counseling (OR=1.57; 95% CI=1.25, 1.97; p<.0001 and OR=1.60; CI=1.04, 2.44; p<.05 respectively). Thus, there was approximately a 60% greater odds of clients’ reducing sexual risk behavior for each client and nurse-defining QNCI behavior. However, the PDP and RPC indices were not associated with clients’ reduction in sexual risk behavior 6 months post nurse counseling (OR=1.67; CI=0.76, 3.63 and OR=1.00; CI=0.80, 1.26 respectively).

4. Discussion and Conclusion

4.1 Discussion

This study has several limitations. The nurses were not randomly selected. Thus, there is the issue of self-selection bias since it is possible that the nurses who agreed to participate in the study might be more skilled and willing to undertake the new counseling program than those who did not participate. Only one nurse was trained in the enhanced model versus 14 standard of care nurses. We did not conduct a pre-test of the experimental study nurse prior to her training which meant that her counseling sessions may have differed from those of the control nurses at the onset of the study. Because of this, we were not able to distinguish between training in DP counseling as compared with other characteristics of the RAD nurse in accounting for our findings. The coders were not blinded to condition because the RAD nurse counseling intervention was scripted and this increased the chances of voice recognition. It is possible that since the coders were not blinded that this could have introduced bias in favor of the study nurse and thereby overestimate the results. We therefore underscore the need to explore this further in future studies. In addition, the number of audiotapes coded for this study was relatively small, which meant that our power was limited and that only large effect sizes could be confidently detected.

We used a dichotomous variable to measure women’s self-reported change in unprotected sex occasions from baseline to follow-up as our outcome. Since the outcome of interest is based on self-report, this lends itself to bias. Also, measuring the outcome as a dichotomous variable decreases variance and may actually underestimate associations.

Despite these limitations, we have provided initial evidence of the overall reliability of the DPCC and of the construct validity of the PDP, RPC, and QNCI indices as supported by the greater use of such behaviors by the RAD nurse versus SOC nurses. The fact that the RAD nurse exhibited a greater number of the target behaviors makes sense, given the RAD’s focus on these elements. Construct validity of the QNCI was supported adequately by its association with sexual risk reduction by clients of both RAD and SOC nurses six months post counseling, but should be interpreted only as a trend due to non-significance. The lack of significant association between the promotion of dual protection (PDP) and relapse prevention (RPC) indices and client sexual risk reduction may in part have been due to low power.

One of the unique features of this study is that it went beyond developing and testing the reliability of a tool to assess nurse performance. It examined whether such behaviors influenced clients’ behaviors post counseling, and found evidence that quality of the nurse-client interaction was indeed related to client sexual risk reduction 6-months post-intervention. In spite of the brevity of the client communication behaviors assessed, we believe it is critical to credit clients for the contribution they make in the counseling session that is not sparked by providers’ expressions. These client communication behaviors may in fact be stimulated by nurse behaviors, but the clients are credited with the contribution to the counseling dynamic in the coding process. Thus, we need to explore further how nurse counselors’ effective communication skills shape clients’ communication behaviors and therefore appropriately credit the counselors in the coding process. The six client-defining items reflected in the QNCI index do capture a range of meaningful behaviors as evidenced by the acceptable reliability and construct validity. However, other meaningful behaviors such as non-verbal communication expressions also must be incorporated in the nurse-client assessment to ensure that important behaviors do not go unnoticed.

Before we can generalize our findings, additional work will need to be conducted with a different case mix and a greater number and variety of providers in settings where reproductive counseling is conducted.

4.2 Conclusions and Practice implications

Our results indicate that the DPCC is a reliable coding tool that shows promise of practical utility in family planning settings. Coding nurse providers’ adherence to the index-defining behaviors would make such coding even easier by limiting the number of items. The face valid indices had good internal consistency, with a total of only fifteen items across all indices. Thus, an abbreviated DPCC, focusing only on the index behaviors, could provide an efficient alternative to more cumbersome coding instruments.

Another example of the practical utility of the DPCC relates to the fact that the audiotaped interviews did not require transcribing. The DPCC employs a checklist format in which coders mark nurse and client behaviors as “present” or “absent”, instead of the likert-type responses used in other instruments [4,5]. Raters using the DPCC coded the data directly from the audiotapes - with high degree of inter-rater reliability - instead of relying on sifting through lengthy transcriptions, or relying on potentially intrusive observation of sessions that could influence client and/or provider behavior.

The DPCC instrument does not necessarily require high skill or educational level, making it relatively easy to train risk reduction counselors, supervisors, and other providers in its use. Coupled with evidence that the QNCI index, constructed from only 10 DPCC codes, can be used to predict reduction in client sexual risk behavior, this suggests that an abbreviated DPCC holds practical utility for evaluation, monitoring, and supervising of nurse-client counseling sessions in real world settings.

Fig. 1.

Dual protection counseling checklist (DPCC) reliability and selection of itms for final indices