Development of motivational interviewing skills in immunization (MISI): a questionnaire to assess MI learning, knowledge and skills for vaccination promotion

ABSTRACT

Objective: Vaccine hesitancy is a complex problem. We previously demonstrated that motivational interviewing (MI) could be helpful to enhance parents’ motivation to vaccinate their child. The aim of this study is to develop a new, simple and robust evaluation tool that is suitable for evaluating MI learning of vaccination health professionals.

Methods: We designed the Motivational Interviewing Skills in Immunization (MISI), a short written questionnaire to evaluate the MI knowledge and skills of participants in an immunization context. It covers three key areas: knowledge of MI, ability to apply MI-related skills, participant self-confidence in using MI. Questionnaire content and face validity were assessed by MI experts and internal consistency, reliability and effect size were analyzed using a multiple pretest-posttest design.

Results: Psychometric measures showed good to excellent internal consistency of the questionnaire for all three areas (Cronbach’s and KR coefficient: 0.70 to 0.88). Test-retest reliability showed good measurement stability (ICC: 0.53). Good sensitivity to change was also obtained (Cohen’s d: 0.80 to 1.66).

Conclusion: The MISI questionnaire is the first paper/pencil evaluation method to assess MI training specific to immunization. Psychometric measures showed high reliability.

Practice implications: This questionnaire could provide a convenient and inexpensive method to evaluate knowledge and competencies following immunization-specific MI training.

Introduction

Despite vaccine hesitancy being a global and complex problem decried by the World Health Organization, the vast majority of vaccination promotion strategies don’t modify attitudes and behaviors towards vaccination.^1-8 To address this problem, we developed a new strategy to promote vaccination, using motivational interviewing (MI) with parents of newborn infants.^9-12 MI is a communication technique commonly used for addiction treatment,^13,14 and increasingly so in health promotion, including immunization.^11,12,15-21 MI is a patient-centered and directive counseling style, designed to help reinforce a patient’s motivation to change a behavior.^22-24 Our previous study showed that a short MI-based intervention significantly increases parents’ intention to vaccinate their child and significantly improves vaccine coverage of these children.^9-12 With the parents of newborn infants, MI is a convenient and inexpensive way to address both immunization concerns and benefits, in order to help raise motivation to vaccinate offspring.

Given the promising results of our previous study, we developed immunization-specific MI training for nursing health professionals administering vaccines in health facilities, homes, schools, and the community, as nurses mainly perform all vaccinations in Quebec, with an aim to promote vaccination among their patients. The first of day of this MI training is devoted to discovering the MI spirit and acquiring the basic MI skills using theory, observation and active participation of trainees in situational exercises in the context of immunization. After a three-month interval, the second day of training is devoted to revisiting MI skills learned on day and providing feedback to participants. This immunization-specific MI training is based on recurring fears and myths about vaccination where all examples used in the clinical cases are related to immunization. In this specific training, learning and use of the MI skills are tailored according to the patients’ vaccine hesitancy level.

Although this communication technique appeals to most health practitioners and MI training is gaining in popularity,^25,26 certain concerns regarding the evaluation methods of MI skills post training need to be addressed. Standardized validated instruments such as the Motivational Interviewing Treatment Integrity Code (MITI),^27-29 the Behaviour Change Counseling Index (BECCI),³⁰ the Motivational Interviewing Skills Coding System (MISC)³¹ and the Motivational Interviewing Supervision and Training Scale (MISTS)³² already exist and are used to assess MI competencies of trainees in any health topic. Most of the current validated tools for MI skill evaluation involve recorded audio or video sessions of the patient-provider encounter. Not surprisingly, obtaining consent from patients can be challenging. Some difficulties related to the audio or video equipment can also be experienced by the health care practitioners such as access to the material from the institution, poor sound quality and IT security policies. In addition, most evaluation tools are not suitable for large-group evaluation of MI skills following training. Indeed, most are relatively complex and time consuming tools for scoring the items needed to evaluate MI skills among individuals. Some of these tools also require extensive training of the part of the evaluators, an extra challenge for large-group studies requiring several evaluators.³³

To our knowledge, few validated evaluation tools currently exist in the written form to assess the MI learning and skills of trainees. While the Motivational Interviewing Knowledge and Attitude Test (MIKAT) is a written survey assessing knowledge and attitudes about MI, it is limited by the fact that MI competencies are not evaluated.³⁴ Few authors suggest written surveys based on the Helpful Response Questionnaire (HRQ) to evaluate MI-related skills, instead using a different coding method adapted for MI-related skills.^35,36 HRQ is a written open-response questionnaire to evaluate respondent empathy,³⁷ providing an efficient means for assessing respondent ability to generate a response relevant to an open-ended question instead of respondent capacity to discriminate between answers from a list. The HRQ has been validated for use with groups of professionals. As very few validated MI-skill evaluation tools that are suitable for use with large groups, are immunization-specific, the aim of this study is to develop such a written evaluation tool. Ideally, this new written tool should provide a convenient, robust and reliable questionnaire to evaluate the MI knowledge and skills of vaccination professionals, as well as their ability to generate adequate answers in a situational exercise reflecting MI integrity.

Results

Content and face validity

A panel composed of four MI experts identified 18 of our original 32 items as less relevant across all three areas. Of these, 5 were dropped from MI knowledge assessment, 1 from the open-response questions, 11 from ability to apply MI skills, and 1 from self-confidence in using MI skills in clinical practice. Four additional items needed clarification according to the expert panel: 1 in the area of MI knowledge and 3 in that of MI skills. Modifications were made as suggested by the experts. According to the expert panel’s recommendations, 21 new items were written into the questionnaire by our team: 5 for MI knowledge, 12 for MI skills and 4 for self-confidence in using MI skills in clinical practise.

Experts recommended reducing the two open-response questions to a single one to encourage participants to provide a detailed response, instead of two shorter responses. The expert panel also suggested including new clinical cases for the open-response question in each of the three follow-up questionnaires to ensure participants did not provide identical responses to posttest questionnaires.

At the end of this revision process, the new questionnaire contained a total of 31 items: 9 multiple-choice questions for MI-knowledge acquisition, 1 open-response question and seventeen 6-points Likert-scale items for MI-skills application, and 8 items assessing self-confidence in using MI skills in clinical practice.

Participation

Three cohorts of nursing health professionals enrolled for our vaccination-specific MI training in 2017 for a total of 37 nurses, three (3) of whom did not complete the entire training. Among these, 35/37 (94.6%) and 22/34 (64.7%) completed the questionnaire prior to and after the first day of training, respectively. Two months later (i.e. before the 3-month half-day training course), 32/34 (94.1%) completed the questionnaire. Finally, following the second half-day training workshop, 26/34 (76.5%) completed the questionnaire.

Internal consistency

Internal consistency was calculated prior to day 1 and after day 2 for two area (Table 1). Cronbach’s α for the open-response item was 0.70 for both measures, which represents good reliability. The Kuder-Richardson coefficient (KR20) for the categorical items of the MI-skills area was low for the measure at pre-day 1 (KR20 = 0.574). After deleting 2 items that correlated less to other items of this area, the 15 remaining items showed good consistency; KR20 = 0.71 and KR20 = 0.79 at pre-day1 and post-day 2, respectively. Opposite results were observed for the last area; Cronbach’s α were too high at pre- and post-training (α = 0.92 and α = 0.91, respectively). According to analyses, two redundant items were removed, resulting in an excellent internal consistency: α = 0.87 for pre-day 1 and α = 0.88 for post-day 2.

Table 1.

Internal consistency for MI-skills application and self-confidence to use MI.

	Pre-day 1	Post-day 2
MI skills application (1 open-response item)	Cronbach’s α = 0.70 (n = 26)	Cronbach’s α = 0.70 (n = 25)
MI skills application (15 Likert-scale items)	KR20 = 0.71 (n = 33)	KR20 = 0.79 (n = 25)
Confidence in using MI (6 Likert-scale items)	Cronbach’s α = 0.87 (n = 29)	Cronbach’s α = 0.88 (n = 25)

Following this analysis, the final questionnaire contained 9 multiple-choice questions for MI-knowledge acquisition, 1 open-response question and fifteen 6-points Likert-scale items for MI-skills application, and 6 items assessing self-confidence in using MI in clinical practice (See Supplementary Data).

Test-retest reliability

Twenty-one (21) participants answered the items from the MI-knowledge section after training on day 1 and two months later. Mean scores were not significantly different after the 2-month interval (p = 0.565) (Table 2). Scores of this section showed good test-retest reliability, as the intra-class correlation coefficient (ICC) was 0.53.

Table 2.

Mean scores of MI knowledge at post-day 1 and 2 months later.

	Score (/100) (mean ± s.d.)		p	Test-retest reliability (ICC)
	Post-day 1	2 months later
MI knowledge (n = 21)	71.8 ± 14.0	73.0 ± 9.7	0.680	0.53

Intra- and inter-coder reliability

Excellent intra- and inter-coder reliability was calculated for the mean score of the open-response question. Two independent MI-expert evaluators coded 19 random questionnaires and the ICC was 0.74. Two weeks after coding all questionnaires, a single MI-expert coder was asked to recode 19 random questionnaires and the ICC was 0.76.

Effect size

Scores at pre-training day 1 and post-training day 2 were significantly different across each area of the questionnaire (Table 3). The MI-knowledge section showed the greatest effect size (Cohen’d = 1.66) while self-confidence in using MI in the clinic showed the weakest (Cohen’s d = 0.80), although all these results showed very good sensitivity to change.

Table 3.

Mean scores on each questionnaire area at pre-day 1 and post-day 2.

	Score* (mean ± s.d.)		p	Size effet (Cohen’s d)
	Pre-day 1	Post-day 2
MI knowledge (n = 24)	49.3 ± 16.3	76.4 ± 6.8	<0.001	1.66
MI skills application – 1 open-response item (n = 20)	4.4 ± 3.4	9.7 ± 4.2	<0.001	1.56
MI skills application – 15 Likert-scale items (n = 24)	36.9 ± 18.6	58.2 ± 23.2	<0.001	1.04
Self-confidence in using MI (n = 23)	69.3 ± 15.1	80.8 ± 7.9	0.001	0.80

*Scores were converted onto a 100-point scale, except for the open-response item.

Discussion and conclusion

Discussion

In this study, we propose a new written questionnaire to evaluate MI-knowledge acquisition and MI-skills application, following immunization-specific MI basic training. Our questionnaire design allowed evaluating three key aspects: MI-knowledge acquisition, ability to apply MI-related skills and self-confidence in using MI skills in clinical practice. MI-related skills were assessed using two different types of evaluation: a) by writing a detailed MI-consistent real-life dialogue in answering an open-response question and b) by self-rating the frequency of MI-adherent and non-adherent behaviors. These key aspects are consistent with two MI training outcomes described in the Kirkpatrick’s training evaluation model, which are clinical use of MI and MI competencies as the trainee improved ability to use MI elements.³⁸ The open-response question written in the form of a dialogue is a relatively convenient and inexpensive method of evaluation to measure the use of MI in clinical practice. Although the MITI is currently the gold standard to assess MI skills in clinical practice, it is however more complex and quite expensive to administer, with 20-minute audio recordings to code for each trainees. The open-response item of our questionnaire allowed participants to write down what they would respond to a patient in a real-life situation using their MI competencies according to the MI-spirit. This paper/pencil method is convenient for use with large-groups. Therefore, our tool seems highly feasible for MI training workshops and multiple-group trainings (such as our MI training given three times a year) where the use of the gold standard evaluation method, such as MITI, is unlikely to be feasible. With our coding protocol inspired from the MITI scoring,²⁷ it is also easier and faster to code the text response of several participants. Other studies also adapted the HRQ tool and developed their own coding system.^35,39 In this case, we prepared a coding manual for our situational vaccination-specific open-response question so that other coders can evaluate the text response to this question (See Supplementary Data).

According to Kirkpatrick’s training evaluation model, two other MI training outcomes should be assessed. Namely, participants’ reactions to the training and patient health outcomes. We assessed the former by satisfaction surveys administered at the end of each of the training days and very high level of satisfaction was expressed from participants for both days (94% and 93%, respectively). The latter cannot be assessed in this study since the questionnaire only targeted trainees and not patients. However, we have assessed patient health outcomes previously.^9,10,12 Briefly, we showed that parents exposed to an immunization-specific MI intervention had greater intention to vaccinate their offspring and that these children display improved short-term vaccine coverage.

In addition to be the first questionnaire evaluating immunization-specific MI training, this new questionnaire was validated by a panel of international MI experts. Review of the questionnaire by this panel of MI experts was highly useful in discriminating between relevant and non-relevant items, and in order to make appropriate modifications and additions. Their contribution was invaluable since they suggested assessing all MI components such as MI principles, MI processes, MI spirit and MI skills, using several types and sets of questions. For instance, the section covering MI knowledge contains one set of items written in a theoretical manner, and another written in the form of a clinical case. After validation by the MI panel of experts, several analyses were performed to demonstrate the questionnaire’s reliability and responsiveness.

Psychometrics

Internal consistency analyses resulted in the deletion of non relevant items from the section on the ability in applying MI-related skills, as well as that of redundant items from the section covering the self-confidence in using MI in the clinic. The remaining items showed good reliability, as Cronbach’s α and Kuder-Richardson coefficient ranged between 0.70 and 0.90.⁴⁰ Internal consistency was not analyzed for the MI-knowledge section since its aim was not to discriminate between trainees’ level of MI knowledge. To find out if a majority of participants had increased MI knowledge after the training, we asked trainees to answer 9 relatively easy items. We analyzed test-retest reliability between results measured after day 1 and 2 months later. For this section, the questionnaire showed acceptable to good measurement stability over time as ICC ranged between 0.4 and 0.75, as previously described by Fleiss.⁴¹ Test-retest reliability was not assessed for the other two areas covered by the questionnaire, as we expected participants to improve their MI skills and self-confidence in using MI over time.

Intra- and inter-coder reliability calculated for the open-response question was excellent, as suggested by Fleiss.⁴¹ This result is consistent with other studies who adapted the HRQ items for written evaluation of MI skills.^35,39 The inter-coder reliability result confirms that our coding protocol for the open-response is clear and precise. Indeed, a second coder similarly scored the same text responses compared to the first coder.

For each area covered by the questionnaire, a large effect size was observed. For MI knowledge and the two types of questions for MI-related skills, the difference between mean scores from pre-day 1 and post-day 2 was one standard deviation higher. Although Cohen’s d did not reach one standard deviation, questionnaire responsiveness for the “Self-confidence in using MI” section was also qualified as excellent according to Cohen and McDowell.^42,43

Limits

One limit of this study is the absence of a control group. Without a control group, it is difficult to determine if the findings related to MI-skill acquisition are exclusively related to the training protocol or if other factors contributed. However, our questionnaire was completed at 4 different time points during the training, in order to adequately represent trainee learning. Although our written questionnaire can evaluate vaccination-specific MI knowledge and MI-related skills, it cannot show whether there is an actual behavioral change associated to the MI training of the nursing health professionals. Comparison of our questionnaire to the MITI is currently under investigation and will be the object of a forthcoming study. We expect that MISI questionnaire is probably not as precise as the MITI. However, the MISI is more amenable to large-group training and multiple-group training because of its high level of convenience and its low level of cost to use.

Conclusion

The MISI questionnaire is the first paper/pencil evaluation method to assess immunization-specific MI training. Preliminary psychometric measures showed high reliability and future validation of our questionnaire against a gold standard tool will confirm its use as an efficient, convenient and inexpensive evaluation method.

Practice implications

With the increasing proportion of vaccine-hesitant individuals, immunization-specific MI training will offer to immunization providers an effective approach to communicate with vaccine-hesitant patients and promote vaccine uptake. We foresee this questionnaire could be used from now on as a rapid and inexpensive method to evaluate MI-knowledge acquisition and competencies following immunization-specific MI training. This tool can be used among all healthcare professionals administering vaccines and is available as Supplementary Data.

Methods

Training evaluation

We used a multiple pretest-posttest design to evaluate our training protocol. We asked trainees to complete a pretest and a posttest before and after each day of the two-day training. We designed a short written questionnaire to evaluate three key aspects of the participants’ MI training: 1) knowledge of MI theory and principles, and MI-related skills, 2) ability to apply MI-related skills, and 3) self-confidence in using MI in clinical practice.

Questionnaire design

In order to evaluate our immunization-specific MI training protocol, we designed a short written questionnaire to be administered before and after each day of training. Following a careful review of relevant scientific literature on MI evaluation, a preliminary questionnaire was drafted according to the three content areas identified in published studies:^35-39,44 1) MI knowledge acquisition, 2) MI skills application and 3) confidence in practising MI. These three main areas were used to design the Motivational Interviewing Skills in Immunization (MISI) questionnaire. The MISI questionnaire items were constructed de novo in collaboration with a team of experts from the Motivational Interviewing Network of Trainers (MINT) and immunization experts involved in the PromoVac strategy.^9,12 Invitation to participate was sent to several experts and those who were interested were selected in the panel. The first area – MI knowledge, theory and principles, and MI-related skills – is assessed by 9 multiple-choice questions. The next area – ability to apply MI-related skills – was assessed using two separate types of questions. The first set was inspired from the HRQ in order to assess the application of MI-related skills.³⁷ This set consists of 2 vaccination-specific situational, open-response questions that participants are instructed to answer by writing an MI-consistent dialogue. The second set, assessing MI skills application, is a list of sixteen 6-point Likert-scale items allowing for self-measurement of the frequency of MI-adherent and non-adherent skills and behaviors in the participant’s clinical practice. For each of the sixteen statements, participants rate the frequency of the behavior as: “Never”, “Very rarely”, “Rarely”, “Moderately”, “Frequently” and “Extremely”. The last 5 items of our written questionnaire cover self-confidence in using MI in clinical practice. Each of the five consist of a 10-point Likert-scale ranging from 1, meaning a low level of self-confidence, to 10 meaning high level of self-confidence.

Coding protocol for the open-response question (adapted from HRQ)

Two items inspired from the HRQ were included into the questionnaire in order to assess the ability to apply MI-related skills.³⁷ Items are a vaccination-specific situational open-response question that participants are instructed to answer by writing a MI-consistent real-life dialogue. In order to code the question, a coding manual was developed, detailing the coding procedure. This coding manual was adapted from the MITI coding manual version 4.2.²⁷ Briefly, our coding manual comprises two components: 1- scores for four MI skills and 2- a global score of MI-spirit called “adherent score”. The coder must identify and count as MI-related skills each open-ended question (1 point), reflective-listening statement (between 1 and 2 points depending on complexity level), affirmative statement (1 point) and “elicit – provide – elicit” feature (between 1 and 3 points depending on the completion status). The answer to this open-response question must be in the form of a dialogue, otherwise no point is awarded. For the adherent score, the evaluator must appreciate the dialogue as a whole, coding occurs only on vaccinator interventions. Adherent score is evaluated on a scale from 1 to 5, with 3 represent the score to start from. MI-inconsistent behaviors perceived in the written dialogue, shift the adherent score closer to 1. On the other hand, where the respondent respects the MI-spirit, adherent score shifts closer to 5.

Content and face validity

In order to assess the content and face validity of the questionnaire, we contacted four local and international MI experts, who are members of the MINT. These experts were asked to identify the non-relevant and most relevant items of the questionnaire, as well as reformulate any questions needing clarification in their opinion. They were also instructed to identify any relevant items that we may have overlooked. The research team then deleted all items deemed as non relevant by the MI experts, as well as modified and wrote in new items according to the expert panel’s suggestions.

Participants and training protocol

After its content and face validity were assessed by the MI experts, the questionnaire was completed by nursing health professionals providing vaccination in health facilities, homes, schools and/or communities across the Eastern Townships (Quebec, Canada). The vaccination-specific MI training protocol consisted of two in-person training workshops: a one-day workshop followed by a half-day workshop three months later. The contents of the training provided an overview of MI theory and principles, followed by a practical demonstration and exercise to develop vaccination-specific MI-related skills. Training was provided by an MI expert, member of the MINT. Using a multiple pretest-posttest design, trainees were asked to complete the questionnaire before and after each day of training. Pre-test questionnaires were sent out by email two weeks prior to training and participants were invited to return their completed questionnaire either by email, by fax, or in person on the morning of the first day of training. Post-test questionnaires were sent out by email two days after the training and participants were given two weeks to complete and return it back by email or fax.

Intra- and inter-coder reliability

Reliability of the coding for the written open-response item was assessed. Coding of this item was first performed on a small random selection of questionnaires (20%) by a pair of MI experts, members of the MINT. Once good inter-coder reliability was established, the remainder of the questionnaires were coded by a single MI expert. In order to assess intra-coder reliability, the same MI expert randomly coded 20% of the questionnaires two weeks later.

Data analysis

MISI scoring

In the MI-knowledge area, multiple-choice questions were dichotomously coded, with 1 point allotted for each correct answer and 0 for any wrong answers. A dichotomous analysis was also performed for the 6-point Likert-scale items from the MI-skills area. For the MI-adherent items, the two highest categories of responses (“Frequently” and “Extremely”) were grouped and attributed 1 point. MI non-adherent items, were each attributed 1 point but only for the lowest category of responses (“Never”). Continuous variables in the third area – self-confidence in using MI-skills in clinical practice – were coded according to the number of points scored in the 10-point Likert-scale items. The totals for each area were then calculated by summing each item contained in the respective area. Raw totals were converted to 0–100 scales using simple linear transformation, accounting for items with missing values. Data from sections of the questionnaire with over 20% unanswered items were excluded from analyses. For the open-response question, the total score was calculated by summing the total points attributed for counts of each of the four MI skills and the score of the MI-Adherent item.

Calculated scores are presented using mean and standard deviation (SD). All statistical analyses were performed using SPSS version 24.

Internal consistency

In order to verify whether items measure the same general construct, analysis of internal consistency of domains was performed using the Kuder-Richardson coefficient (KR20) and Cronbach’s α for dichotomous and continuous variables, respectively.⁴⁰

Reliability

Test-retest reliability was measured for the MI-knowledge area by determining the concordance between mean scores at post-training day 1 and then two months later using the ICC. Mean scores were also compared using paired Student’s t-test for detecting significant difference between both scores.

The ICC was also used to determine intra- and inter-coder reliability of scores on the open-response question.⁴¹

Effect size

Finally, questionnaire responsiveness was analyzed by comparing data measured prior to the first day of training and 3-months later following the half-day training workshop. Mean scores were compared using paired Student’s t-test and effect size (Cohen’s d) was calculated for each area.⁴²

Funding Statement

This study was funded by the Eastern Townships Public Health Department. The authors have no financial relationship relevant to this article to disclose. The funding source has no involvement in the study design, collection, analysis and interpretation of data, writing, or the decision to submit this manuscript for publication.

Abbreviations

BECCI

Behaviour Change counseling Index

HRQ

Helpful Response Questionnaire

ICC

intra-class correlation coefficient

MI

Motivational interviewing

MIKAT

Motivational interviewing knowledge and attitude test

MINT

Motivational Interviewing Network of Trainers

MISC

Motivational Interviewing Skills Coding System

MISI

Motivational Interviewing Skills in Immunization

MISTS

Motivational Interviewing Supervision and Training Scale

MITI

Motivational Interviewing Treatment Integrity Code

SD

Standard deviation

Acknowledgments

We thank all of the nursing health professionals who participated in this study as well as all of the experts who provided their generous help in developing the questionnaire, including the MINT experts who reviewed the questionnaire, Danielle Pinsonneault, Karine Gaudreault, Jacques Dumont and Patrick Berthiaume. We also wish to thank Marie-Claude Battista and Monique Sullivan for critical revision of this manuscript.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed on the publisher’s website.