Abstract
Introduction
A key contributor to underimmunisation is parental refusal or delay of vaccines due to vaccine concerns. Many clinicians lack confidence in communicating with vaccine-hesitant parents (VHP) and perceive that their discussions will do little to change parents’ minds. Improving clinician communication with VHPs is critical to increasing childhood vaccine uptake.
Methods and analysis
We describe the protocol for a cluster randomised controlled trial to test the impact of a novel, multifaceted clinician vaccine communication strategy on child immunisation status. The trial will be conducted in 24 primary care practices in two US states (Washington and Colorado). The strategy is called Presumptively Initiating Vaccines and Optimizing Talk with Motivational Interviewing (PIVOT with MI), and involves clinicians initiating the vaccine conversation with all parents of young children using the presumptive format, and among those parents who resist vaccines, pivoting to using MI. Our primary outcome is the immunisation status of children of VHPs at 19 months, 0 day of age expressed as the percentage of days underimmunised from birth to 19 months for 22 doses of eight vaccines recommended during this interval. Secondary outcomes include clinician experience communicating with VHPs, parent visit experience and clinician adherence to the PIVOT with MI communication strategy.
Ethics and dissemination
This study is approved by the following institutional review boards: Colorado Multiple Institutional Review Board, Washington State Institutional Review Board and Swedish Health Services Institutional Review Board. Results will be disseminated through peer-reviewed manuscripts and conference presentations.
Trial registration number
Keywords: paediatrics, community child health, public health
Strengths and limitations of this study.
This study uses a robust design to minimise contamination as well as selection, ascertainment and participant biases.
The trial setting of 24 primary care practices across two US states facilitates generalisability and will provide a knowledge base for how an intervention can be integrated within a real-world practice setting.
In measuring fidelity, it is possible to assess how our study results are attributable to clinician communication behaviour.
We will not be able to assess the impact of the intervention beyond 19 months of age.
Introduction
The reduction of vaccine-preventable diseases (VPD) is a top public health goal.1 Vaccine hesitancy, defined as a state of indecision and reluctance that results in a desire to defer or omit any routinely recommended vaccines, is a major barrier to achieving this goal and was recently declared as one of the top 10 threats to global health by the WHO.2 Parental refusal or delay of childhood vaccines has been associated with increased odds of VPD3–5 as well as higher inpatient admission and emergency department utilisation rates.6 Therefore, there is renewed emphasis on sustaining and improving childhood vaccine coverage.7 8
A key influence on vaccine decision-making among vaccine-hesitant parents (VHP) is their child’s clinician.9 VHPs consider their child’s clinician to be a key information source in their decision-making about vaccines and their child’s health.10–14 Indeed, initially hesitant parents reported changing their mind about delaying or refusing a vaccine after their child’s clinician addressed their concerns, provided them with additional information or gave them reassurance.13 14 Consequently, how clinicians discuss and recommend vaccines is important. For instance, the quality and presence of a clinician’s recommendation has been associated with increased uptake of childhood and adolescent vaccines.15–25 Moreover, the communication format used to initiate the vaccine recommendation is influential: a presumptive (eg, ‘We have to do some shots’) rather than a participatory (eg, ‘How do you feel about vaccines today?’) format has been associated with increased parental acceptance of childhood and adolescent vaccines.26–28 Even among VHPs, significantly fewer verbally resisted vaccine recommendations when clinicians used a presumptive (vs participatory) format.26
A presumptive initiation format, however, is not always sufficient as some parents still voice resistance to vaccines even if the vaccine recommendation is initiated presumptively. In addition, many clinicians lack confidence in communicating with parents who voice substantial vaccine concerns.29 To address these barriers, the feasibility and efficacy of using Motivational Interviewing (MI) with VHPs has been explored. MI is a well-established, evidence-based, patient-centred framework for behaviour change30–47 that is effective even when delivered in a single session.33 41 MI’s three essential elements—having a conversation, leveraging inherent motivation for behaviours and making the conversation person centred—make it well adapted for use with VHPs given their known communication preferences regarding vaccines.10 14 In a large randomised controlled trial (RCT), clinician use of MI in discussions with parents who verbally resisted the HPV vaccine recommendation resulted in increased HPV vaccine acceptance and improved clinician perceptions of their ability to influence parental vaccine decision-making.48
This manuscript describes the protocol for a cluster RCT (cRCT) to evaluate the effect of a novel clinician communication strategy that combines use of the presumptive initiation format and MI—the Presumptively Initiating Vaccines and Optimizing Talk with Motivational Interviewing (PIVOT with MI) intervention—on child immunisation status as well as on parent and clinician experience.
Conceptual model
The PIVOT with MI intervention is grounded in a conceptual framework by Resnicow et al called the ‘Difficulty by Motivation’ matrix (figure 1).49 This matrix, which can be applied to both health behaviours and health behaviour interventions, parses behaviours or interventions into four quadrants. Quadrants represent properties of the health behaviour or intervention being considered (x-axis) and individual-level factors, including motivation and/or competence to comply with the recommended behaviour or intervention (y-axis). Vaccination can be categorised as a simple health behaviour given it is an intervention with a high certainty of low risk and high benefit. Vaccine-accepting parents reside in the upper left quadrant given their high motivation to comply with vaccination. Interventions to promote compliance with vaccination simply need to match this motivation, such as using a presumptive format for initiating the vaccine recommendation that leverages vaccination as a normative behaviour. In contrast, VHPs reside in the lower left quadrant, having high resistance and low motivation to comply. As such, interventions to promote vaccination among VHPs need to be more robust. Following a presumptive vaccine recommendation, VHPs who verbally resist vaccination signal to the clinician that the parent is in need of a more intense intervention. This recognition can serve as a ‘pivot point’ that triggers the clinician to shift from the presumptive recommendation to MI in order to leverage parents' intrinsic motivations for doing what they perceive is best for their children.
Aim and hypothesis
The main objective of this study is to evaluate the effect of the PIVOT with MI intervention on the immunisation status of children of VHPs using a pragmatic cRCT study design. We hypothesise that children of VHPs at intervention practices will be less underimmunised than those at control practices.
Methods and analysis
A summary of the trial’s specifications is presented in table 1.
Table 1.
Data category | Information |
Registry and trial number | ClinicalTrials.gov: NCT03885232 |
Date of registration | 21 March 2019 |
Secondary identifying numbers | 17–1274 |
Financial support | Eunice Kennedy Shriver National Institute of Child Health and Development at the US National Institutes of Health; PO Box 3006, Rockville, MD 20847 |
Contact for queries | douglas.opel@seattlechildrens.org |
Title | Evaluation of the Presumptively Initiating Vaccines and Optimizing Talk with Motivational Interviewing (PIVOT with MI) Intervention |
Countries of recruitment | USA |
Health condition studied | Infant vaccination |
Intervention(s) | Active comparator: clinician vaccine communication strategy Passive comparator: usual care |
Key inclusion and exclusion criteria | Inclusion: ≥18-year-old parent with child ≤2 months old who receives health supervision at participating practice. Exclusion: parent who is <18 years old or has child >2 months old who receives health supervision at a participating practice |
Study type | Cluster randomised controlled trial |
Date of first enrolment | 27 September 2019 |
Target sample size | 600 vaccine-hesitant parents |
Trial status | Ongoing data collection |
Primary outcomes | Percentage of days undervaccinated of child at 19 months of age |
Key secondary outcomes | Parent visit experience; clinician self-efficacy; clinician adherence to PIVOT with MI communication strategies |
Study design and registration
This study is a two-arm cRCT with longitudinal follow-up. Study arms include a control arm, in which clinicians practise usual care, and an intervention arm, in which clinicians use the PIVOT with MI communication strategy with parents. This study is registered with ClinicalTrials.gov (table 1).
Study overview and setting
We will randomise 24 primary care paediatric practices in two US states (Washington and Colorado) to control and intervention arms. Practices will be initially identified through two regional practice-based research networks in each state. Both states are ideal settings to conduct a study of an intervention designed to improve immunisation rates among VHPs, with each ranking in the highest quintile among US states in 2016 with respect to the proportion of parents claiming non-medical exemptions for their child from required school entry vaccines.50
Eligible parents will be identified through a screening survey administered at a visit prior to their children’s 2-month health supervision visit. Clinicians at intervention practices will receive training on the PIVOT with MI communication strategy. Clinicians at control clinics will give usual care, denoting that they will not receive any communication training and will continue to communicate with parents about childhood vaccines as they are accustomed. A subset of enrolled parents at intervention and control clinics will have their children’s health supervision visits with participating clinicians videotaped to assess clinician–parent vaccine communication practices, including adherence to the PIVOT with MI communication strategy among intervention clinicians (ie, intervention fidelity). Our main outcome is child immunisation status at 19 months expressed as the percentage of days underimmunised from birth to 19 months.
Study population and inclusion/exclusion criteria
All English and Spanish-speaking parents ≥18 years old with an infant ≤2 months old receiving health supervision at a participating practice during the enrolment period of August 2019 through March 2021 are eligible. VHPs will constitute our primary study population because changing the vaccination behaviour of these parents is of utmost interest.51 52 VHPs are defined as those with a positive score on the short form of the Parent Attitudes about Childhood Vaccine (PACV-SF) survey, a validated survey to identify VHPs.53–55
Consent and recruitment
The PACV-SF will be placed in participating practices’ standard intake paperwork and administered on check-in an infant’s health supervision visit between birth and age 2 months. The survey will be embedded in a larger questionnaire to minimise ascertainment bias and will include instructions that contain information to ensure that parents are fully informed, including that survey completion is voluntary, for research purposes, and that participation in the study will involve access of children’s immunisation record data. Parents who complete the survey will be considered enrolled and remain active in the study until their child turns 19 months of age.
We will obtain a waiver of documentation of written consent because parent participation consists primarily of information collected in a survey, and therefore completion of the survey is considered consent to participate. However, in participating practices in Washington State, a study information sheet that includes all elements of consent will be attached to the PACV-SF and parents’ signatures will be required to access immunisation records of children in the Washington State Immunization Information System (WAIIS).
For the fidelity substudy, study staff will approach enrolled English-speaking VHPs who answered an item on the PACV-SF affirmatively that asks for their permission to contact them via phone or email to determine their interest in participating in the substudy. We will describe the study to parents in general terms as a study in which we videotape their child’s visit to assess how doctors and parents communicate at child check-ups in order to minimise the chance that parents alter their immunisation-related behaviour during the videotaped visit to meet observer expectations. Interested VHPs will meet study staff at an upcoming health supervision visit that their children have with participating clinicians to obtain parents’ written informed consent. Participating clinicians will also provide written informed consent to be videotaped.
Assignment of intervention
Practices are the unit of randomisation. A key issue in cRCTs is the possibility of covariate imbalance in practices assigned to different treatment arms.56–60 Thus, practices will be randomly assigned by one analyst (MD) based on specific covariates that are potential confounders to assure balanced comparison groups using optimised probability sampling to the intervention or control arm using covariate-constrained randomisation. From the 24 practices that agree to participate in the study, prestudy information will be collected on several variables that may influence study outcomes: per cent of VHPs, per cent of Vaccines for Children-eligible patients, number of paediatric clinicians and proportion of participating clinicians who currently use the presumptive initiation format. Each variable will be used to develop and evaluate a balance criterion, defined as the sum of the squared differences between standardised practice means on these variables.57 All possible combinations of eligible practices in intervention and control arms will be generated using a SAS macro program (SAS version 9.4; SAS Institute Inc., Cary, NC, USA).59 The distribution of the balance criterion for the two study arms will be used to define an acceptable set of study arms that are reasonably balanced in terms of the selected variables (minimum 10% of balance criterion).57 From this set, one set will be chosen at random and used to randomly assign each practice to intervention or control arm. The process of randomisation will be undertaken separately among the 12 Colorado and 12 Washington practices and will occur prior to parent enrolment.
Blinding
Given our intervention, it is not possible to blind practices or investigators to study arm allocation; however, our analysts will be blinded. We will minimise selection bias by approaching all parents whose newborns receive healthcare at participating practices and by including those who complete the PACV-SF survey in the study and analysis. We will minimise participant bias by blinding clinicians at participating practices to PACV-SF scores of parents and parent ascertainment bias by embedding the PACV-SF in a larger survey.
Sample size calculation
Primary outcome
Based on preliminary data, children of VHPs have a mean percentage of days underimmunised of 26.2% (SD 29.8) from birth to 19 months of age for six vaccines combined (hepatitis B, diphtheria, tetanus and acellular pertussis (DTaP), Haemophilus influenzae type b (Hib), inactivated polio virus (IPV), measles, mumps and rubella (MMR) and varicella).55 In order to have adequate power (≥90%) to detect a decrease of seven percentage points in days underimmunised (26% to 19%), we plan to enrol 600 VHP/newborn pairs total with 300 per arm (assuming an α of 0.05, an SD of 20 and an intraclass correlation coefficient for within-clinic correlation of 0.02).29 This effect size of seven percentage points in days underimmunised corresponds to a clinically meaningful decrease in days late per vaccine dose to within the 30-day window in which most vaccine doses are recommended.61 Assuming a 10% prevalence of VHPs,29 we will need to approach 6000 parents to reach our sample size goal (ie, 5400 of parents will be non-VHPs). This is feasible in our planned 18-month enrolment period given our design to integrate PACV-SF screening of parents of newborns into standard workflow and our estimate of ≥10 newborns per clinic per week in participating practices. If we vary our assumptions regarding VHP prevalence (7.5%) or SD (25), we would need to approach 8000 and 12 000 parents, respectively, for adequate (≥90%) power to detect the same effect size, both of which remain feasible within the 18-month enrolment period assuming ≥10 newborns per clinic per week.
Secondary outcomes
Since our fidelity outcome is based on a qualitative study design, power is not appropriate. However, based on our past work,26 27 we believe we will achieve sufficient behavioural variation across clinicians to develop a coding scheme with three videotaped visits per clinician. We will therefore videotape 360 encounters (180 in each state), which averages to three videotaped encounters per practice clinician.
For the parent visit experience outcome, we will have 80% power to detect a 13 percentage point difference in the proportion of parents who rate their visit experience highly between study arms if we enrol 312 VHPs (156 per arm) assuming a baseline proportion of 72% of parents who rate their visit experience highly27 and equal distribution of participants in control and intervention practices. For clinician outcomes, we will have 80% power to detect a 12 percentage point pre-post difference between control (one percentage point pre-post difference) and intervention clinicians (13 percentage point pre-post difference), assuming 60 clinicians in each arm. Overall, this number of clinicians per arm seems feasible since we anticipate the study will involve >120 total clinicians (five to seven clinicians per practice).
Intervention
The PIVOT with MI intervention involves the combination of two evidence-based vaccine communication strategies in a tiered approach: use of the presumptive format to initiate the childhood vaccine recommendation with all parents followed by use of MI if a parent verbally resists the recommendation. To train clinicians to use the PIVOT with MI communication strategy, we developed a multifaceted curriculum62–64 that includes several approaches based on adult learning theory65 previously found to be effective in changing clinician behaviour.66 67 These approaches include interactive and tailored educational outreach,68 clinician rehearsal and coaching,69 audit and feedback,70 booster learning sessions69 and change agents.71 Given our intent to also have PIVOT with MI implemented with Spanish-speaking parents, the PIVOT with MI intervention draws on evidence for culturally adapting behavioural interventions in community settings.72 73 We obtained iterative input from clinicians at primary care paediatric practices not participating in the trial on the format, length, and content of the PIVOT with MI intervention components (table 2).
Table 2.
Intervention component | Description |
Online video module | Introduces the PIVOT with MI communication strategy and its rationale. |
One 60 min in-person interactive clinician training session | Includes (A) a brief didactic session on vaccine hesitancy, how the PIVOT with MI strategy addresses vaccine hesitancy, and practice data on vaccination coverage and vaccine hesitancy prevalence, (B) baseline assessments of clinician skills using the presumptive format and MI, and (C) modelling of elements of the PIVOT with MI intervention followed by clinician rehearsal through role-playing and coaching by the study team. An online version of this session is available when clinicians are unable to attend the in-person session. |
Reference sheets | Provides brief and accessible summaries of the communication behaviours that comprise PIVOT with MI, along with example statements for key steps in the PIVOT with MI communication strategy. |
Two 30–60 min in-person refresher trainings at 3–6 and 9–12 months after the start of the intervention | Includes a question and answer session regarding barriers to implementing the PIVOT with MI intervention followed by role-playing and coaching, with the 9–12 months of refresher training also including a review of videotaped encounters of intervention clinicians with VHPs to provide feedback for how to improve incorporation of PIVOT with MI into the vaccine discussion. Online versions of these sessions are available when clinicians are unable to attend the in-person sessions. |
Practice study champion | Will routinely solicit feedback from intervention clinicians regarding the PIVOT with MI intervention and liaise with the study team at regular intervals to communicate and help address implementation issues. |
MI, Motivational Interviewing; PIVOT with MI, Presumptively Initiating Vaccines and Optimizing Talk with Motivational Interviewing; VHP, vaccine-hesitant parent.
Outcomes
Our primary outcome will be the immunisation status of children of VHPs at 19 months, 0 day of age expressed as the percentage of days underimmunised from birth to 19 months for 22 doses of eight vaccines recommended during this interval (3 hepatitis B, 3 rotavirus, 4 DTaP, 3 Hib, 4 pneumococcal conjugate, 3 IPV, 1 MMR and 1 varicella). To calculate the percentage of days underimmunised, we will sum the days late for each dose and divide this by the maximum number of days a child could be late if they had received none of the total 22 doses for the eight vaccines by 19 months. We have chosen to use the percentage of days underimmunised because it is a sensitive measure of underimmunisation by accounting for missed vaccine doses and delay in receipt of vaccines.74 Our secondary outcomes include (A) adherence to PIVOT with MI communication techniques assessed via videotaping of health supervision visits, (B) parent visit experience assessed via a 15-item paper survey after a health supervision visit, and (C) clinician experience communicating with parents about vaccines (including time spent discussing childhood vaccines with typical parents and with parents who have substantial vaccine concerns) assessed via a paper or web-based survey before, during and after the parent enrolment period.
Data collection methods
Parents will complete the PACV-SF embedded within a survey on child health topics at enrolment. This survey will also include demographic items that have been associated with underimmunisation (parent age, childbirth order, household income, marital status, parent self-designated race/ethnicity, gender and number of children in their household) as well as permission for study staff to contact them for future studies if interested. For enrolled VHPs who provide permission, study staff will contact them via phone and/or email to assess their willingness to participate in the fidelity substudy. Parents and clinicians will be given the right to review the videotape after the visit and delete any or all portions. We will administer a 15-item visit experience paper survey to all VHPs immediately after the videotaped visit.
Intervention and control clinicians will complete a paper or web-based survey regarding their experience communicating with parents about vaccines. Clinicians will complete a survey at baseline prior to randomisation and parent enrolment, at an interim time point during the study period and at a final time point after parent enrolment is completed.
We will obtain immunisation data from WAIIS and the Colorado Immunization Information System (CIIS). Enrolled children’s medical records will serve as a secondary immunisation data source. WAIIS and CIIS operate in accordance with nationally recommended standards for immunisation registries, and all enrolled practices participate in CIIS or WAIIS. Both CIIS and WAIIS use birth certificate data to identify children and cover ≥95% of children <6 years old in their respective states.75 Both CIIS and WAIIS76 combine the immunisation information from multiple sources into a consolidated, complete and valid record of immunisations. Data quality assessments are done frequently. Offices are required to have <5% error rate in the registry, making these registries a highly accurate data source for assessing vaccine utilisation.
Participant retention
Retention of parent participants is facilitated by our study design. Specifically, parent participation in the overall study is limited to completing a survey at enrolment. Parent participation in the fidelity substudy is limited to a single videotaped encounter and completion of a short visit experience survey. To assist with parent retention in the fidelity substudy, parents will receive a $25 gift card after completing the visit experience survey.
Retention of participating clinicians in the practices is aided by receipt of Maintenance of Certification (MOC) Part 4 credit from the American Board of Pediatrics. To receive MOC, clinicians will need to complete the entire PIVOT with MI curriculum. MOC will be provided to clinicians at practices randomised to the intervention arm during the study and to clinicians at practices randomised to the control arm who complete the PIVOT with MI curriculum at the conclusion of the study. Participating clinicians who complete all surveys at baseline, interim and poststudy will also receive a $25 gift card.
Data security and storage
To ensure confidentiality and protection of the data, we will use several data security measures (box 1).
Box 1. Data security and storage practices.
Health Insurance Portability and Accountability Act (HIPAA) requirements will be adhered to as required by the law.
Each parent/child participant in the study will have a unique study ID and all linkages between study IDs and individual-level data will be destroyed on completion of the study.
Deidentified data will be placed on a secure password-protected file transfer protocol server.
Data will be stored at Seattle Children’s Research Institute (SCRI) and University of Colorado Denver (UCD) where access is limited to SCRI and UCD study staff, with data backed up automatically at least nightly.
We will maintain each data set separately and index the records using unique encrypted identifiers to facilitate linkages between files while maintaining confidentiality of personal health information.
All videotaped data will be kept in a locked storage area at SCRI and UCD; no one outside the research team will have access to the videotaped data.
Videotaped data will be sent to coinvestigators at outside institutions using a HIPAA compliant online data storage server.
The analysis of this videotaped data done at these outside institutions will be done on password-protected computers with access restricted to research team members.
Statistical methods
We will examine baseline characteristics of parents and clinicians by study arm using Pearson’s χ2 tests (or Fisher’s exact tests) for categorical variables and t-tests for continuous variables to assess for any unbalanced confounders. Any unbalanced confounders will be controlled for as covariates in all subsequent regression analyses. Given the nested structure of the data, we will apply mixed effects regression models to examine the effects of the intervention on our primary outcome while controlling for covariates and accounting for correlations due to clustering.
To analyse the videotaped data for our fidelity substudy, we will use conversation analysis (CA) with a subset of the videotaped encounters to develop a coding scheme to assess adherence to PIVOT with MI communication techniques. CA is an analytical technique that searches for patterns in the clinician–parent interaction that are systematically used to accomplish a social action either vocally or non-vocally.77 Multiple coders from the study team will be trained on the coding scheme using 10% of the videotaped data. We will measure inter-rater reliability between the coders using up to an additional 20% of the data until κ scores reach a minimum of 70. Coders will then independently code all remaining data (and recode the initial 10% of training data).
Given the skewed distribution parent visit experience data, we will use two dichotomisation methods to summarise parent visit experience: the top box method,78–81 consistent with Consumer Assessment of Healthcare Clinicians and Systems survey scoring,82 and an alternative method83–85 in which an average score of ≥6 (out of 7) on each of the 15 items is indicative of a highly rated visit experience.27 We will also summarise parent experience as a continuous variable. We will use Pearson’s χ2 test to compare parent (binary) ratings of visit experience between control and intervention arms. We will apply mixed effects logistic regression model to account for within-clinic correlation and any unbalanced confounding factors. When summarising parent experience as a continuous variable, we will apply Box-Cox transformation if it is highly skewed and linear mixed effects models to examine differences between control and intervention arms.
To summarise clinician experience, we will compare the baseline, interim and postintervention proportions of control and intervention clinicians on (1) time spent discussing vaccines at a typical visit and during visits with VHPs; (2) use of presumptive and/or MI techniques in vaccine discussions with families; and (3) reported ability to influence parents’ vaccine decisions. We will use Pearson’s χ2 test in unadjusted analyses of the pre-post differences between control and intervention arms and multivariable logistic regression for adjusted analyses that control for potential confounders such as clinician demographic and practice characteristics. Models will account for clustering of outcomes by practices where needed.
Analytical framework
For our primary outcome, we will conduct an intention-to-treat analysis with the patient as the unit of analysis. This analytical cohort will include all enrolled VHPs/child dyads with child immunisation data at 19 months of age. For clinician and parent experience outcomes, we will conduct a modified intention-to-treat analysis that includes all clinicians and VHPs with a completed survey.
Missing data
We will assess the amount of missing data and missing data mechanism. We will apply Little’s test to check if data are missing completely at random (MCAR).86 We will assess baseline characteristics by missing data as well as missing outcome data by study arm to evaluate MCAR assumptions. We will apply sensitivity analyses and multiple imputation techniques to address missing data.
Subgroup analyses
Planned subgroup analyses include examining clinician experience outcomes by practice type, %VHPs in the practice and practice size.
Monitoring
The principal investigators (PI) of this study (SOL and DJO) will have overall responsibility for participant safety monitoring. However, oversight for data safety and monitoring of the study will be conducted by a faculty member at University of Colorado Denver (UCD) who is not involved in the project. In this capacity, this individual will provide independent observation and verification of protocol compliance, recruitment and study progress, and data completeness. This will be done through correspondence with the PIs and by reviewing draft annual reports on these parameters provided by the study team. This individual will also monitor the study for adverse events, and the study team’s response to these events, should any occur. A letter summarising findings will be included in the finalised annual project reports for National Institutes of Health. Though adverse events are not anticipated, they will be reported to all involved institutional review boards at the time of the event, should any occur.
Assessment of harms and adverse events
The risks for this behavioural intervention are minimal. All participating practices will be regularly reminded via interactions with study champions to promptly report all adverse events to the PIs or designated study representative. Study participants will also be encouraged to contact the research team with any concerns and will be given the team’s contact information at enrolment.
Ethics and dissemination
This study is approved by the Colorado Multiple Institutional Review Board, the Washington State Institutional Review Board and the Swedish Health Services Institutional Review Board.
Informed consent
Parents who complete the enrolment survey will be considered to have consented to participate. This enrolment survey will contain information about the study and its risks and benefits (online supplementary material). Written informed consent will be obtained from VHPs and clinicians who participate in the fidelity substudy.
bmjopen-2020-039299supp001.pdf (784.7KB, pdf)
Access to data
Access to data will be limited to the Seattle Children’s Research Institute and UCD research teams. Access to a deidentified, aggregated version of the data set and analysis code will be available on request with approval by the research team.
Dissemination plans
Study materials will be developed so that they may be easily adapted to other settings, with particular focus on having the online video training module available for use by others immediately. In addition, should this intervention prove effective, we intend to collaborate with two US practice-based research networks to test the intervention on a broader scale. Results of the study will be presented at national and international research conferences and through peer-reviewed publications.
Patient and public involvement
Patients involved in this study are children under age 2 and are only involved in this study as research participants. Parents of child participants will not be involved in recruitment, data analysis or dissemination. Clinicians will be involved in the refinement of the PIVOT with MI intervention.
Supplementary Material
Footnotes
Contributors: DJO and SOL conceived the study and intervention and wrote the study protocol. DJO wrote the first draft of the manuscript. JDR, KG, HS, CS, AD, CP, MD, CZ, BP and JT provided input into the study design, intervention development and study protocol, and edited and/or reviewed the manuscript.
Funding: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Development at the US National Institutes of Health (grant number R01HD093628).
Competing interests: AD and BP serve on advisory boards for Merck, Pfizer and Sanofi Pasteur, and have provided consulting services to Pfizer. BP’s institution receives research funding from GSK and Pfizer.
Patient consent for publication: Not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Centers for Disease Control and Prevention A CDC framework for preventing infectious diseases: sustaining the essentials and Innovating for the future, 2011. [Google Scholar]
- 2.World Health Organization Ten threats to global health in 2019. Available: https://www.who.int/emergencies/ten-threats-to-global-health-in-2019 [Accessed 9 Oct 2019].
- 3.Glanz JM, McClure DL, O'Leary ST, et al. . Parental decline of pneumococcal vaccination and risk of pneumococcal related disease in children. Vaccine 2011;29:994–9. 10.1016/j.vaccine.2010.11.085 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Glanz JM, McClure DL, Magid DJ, et al. . Parental refusal of varicella vaccination and the associated risk of varicella infection in children. Arch Pediatr Adolesc Med 2010;164:66–70. 10.1001/archpediatrics.2009.244 [DOI] [PubMed] [Google Scholar]
- 5.Glanz JM, McClure DL, Magid DJ, et al. . Parental refusal of pertussis vaccination is associated with an increased risk of pertussis infection in children. Pediatrics 2009;123:1446–51. 10.1542/peds.2008-2150 [DOI] [PubMed] [Google Scholar]
- 6.Glanz JM, Newcomer SR, Narwaney KJ, et al. . A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA Pediatr 2013;167:274–81. 10.1001/jamapediatrics.2013.502 [DOI] [PubMed] [Google Scholar]
- 7.Assessing the state of vaccine confidence in the United States: recommendations from the National vaccine Advisory Committee: Approved by the National vaccine Advisory Committee on June 9, 2015 [corrected]. Public Health Rep 2015;130:573–95. 10.1177/003335491513000606 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Hammer LD, Curry ES, Harlor AD, et al. . Increasing immunization coverage. Pediatrics 2010;125:1295–304. 10.1542/peds.2010-0743 [DOI] [PubMed] [Google Scholar]
- 9.Omer SB, Salmon DA, Orenstein WA, et al. . Vaccine refusal, mandatory immunization, and the risks of vaccine-preventable diseases. N Engl J Med 2009;360:1981–8. 10.1056/NEJMsa0806477 [DOI] [PubMed] [Google Scholar]
- 10.Benin AL, Wisler-Scher DJ, Colson E, et al. . Qualitative analysis of mothers' decision-making about vaccines for infants: the importance of trust. Pediatrics 2006;117:1532–41. 10.1542/peds.2005-1728 [DOI] [PubMed] [Google Scholar]
- 11.Gust D, Brown C, Sheedy K, et al. . Immunization attitudes and beliefs among parents: beyond a dichotomous perspective. Am J Health Behav 2005;29:81–92. 10.5993/AJHB.29.1.7 [DOI] [PubMed] [Google Scholar]
- 12.Smith PJ, Kennedy AM, Wooten K, et al. . Association between health care providers' influence on parents who have concerns about vaccine safety and vaccination coverage. Pediatrics 2006;118:e1287–92. 10.1542/peds.2006-0923 [DOI] [PubMed] [Google Scholar]
- 13.Gust DA, Darling N, Kennedy A, et al. . Parents with doubts about vaccines: which vaccines and reasons why. Pediatrics 2008;122:718–25. 10.1542/peds.2007-0538 [DOI] [PubMed] [Google Scholar]
- 14.Fredrickson DD, Davis TC, Arnould CL, et al. . Childhood immunization refusal: provider and parent perceptions. Fam Med 2004;36:431–9. [PubMed] [Google Scholar]
- 15.McCauley MM, Kennedy A, Basket M, et al. . Exploring the choice to refuse or delay vaccines: a national survey of parents of 6- through 23-month-olds. Acad Pediatr 2012;12:375–83. 10.1016/j.acap.2012.06.007 [DOI] [PubMed] [Google Scholar]
- 16.Daley MF, Crane LA, Chandramouli V, et al. . Influenza among healthy young children: changes in parental attitudes and predictors of immunization during the 2003 to 2004 influenza season. Pediatrics 2006;117:e268–77. 10.1542/peds.2005-1752 [DOI] [PubMed] [Google Scholar]
- 17.Flood EM, Rousculp MD, Ryan KJ, et al. . Parents' decision-making regarding vaccinating their children against influenza: a web-based survey. Clin Ther 2010;32:1448–67. 10.1016/j.clinthera.2010.06.020 [DOI] [PubMed] [Google Scholar]
- 18.Gnanasekaran SK, Finkelstein JA, Hohman K, et al. . Parental perspectives on influenza vaccination among children with asthma. Public Health Rep 2006;121:181–8. 10.1177/003335490612100213 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Nowalk MP, Lin CJ, Zimmerman RK, et al. . Changes in parents' perceptions of infant influenza vaccination over two years. J Natl Med Assoc 2007;99:636–41. [PMC free article] [PubMed] [Google Scholar]
- 20.Brewer NT, Gottlieb SL, Reiter PL, et al. . Longitudinal predictors of human papillomavirus vaccine initiation among adolescent girls in a high-risk geographic area. Sex Transm Dis 2011;38:197–204. 10.1097/OLQ.0b013e3181f12dbf [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Guerry SL, De Rosa CJ, Markowitz LE, et al. . Human papillomavirus vaccine initiation among adolescent girls in high-risk communities. Vaccine 2011;29:2235–41. 10.1016/j.vaccine.2011.01.052 [DOI] [PubMed] [Google Scholar]
- 22.Smith PJ, Stokley S, Bednarczyk RA, et al. . HPV vaccination coverage of teen girls: the influence of health care providers. Vaccine 2016;34:1604–10. 10.1016/j.vaccine.2016.01.061 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Freeman VA, Freed GL. Parental knowledge, attitudes, and demand regarding a vaccine to prevent varicella. Am J Prev Med 1999;17:153–5. 10.1016/S0749-3797(99)00063-X [DOI] [PubMed] [Google Scholar]
- 24.Rosenthal SL, Kottenhahn RK, Biro FM, et al. . Hepatitis B vaccine acceptance among adolescents and their parents. J Adolesc Health 1995;17:248–54. 10.1016/1054-139X(95)00164-N [DOI] [PubMed] [Google Scholar]
- 25.Ylitalo KR, Lee H, Mehta NK. Health care provider recommendation, human papillomavirus vaccination, and race/ethnicity in the US national immunization survey. Am J Public Health 2013;103:164–9. 10.2105/AJPH.2011.300600 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Opel DJ, Heritage J, Taylor JA, et al. . The architecture of provider-parent vaccine discussions at health supervision visits. Pediatrics 2013;132:1037–46. 10.1542/peds.2013-2037 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Opel DJ, Mangione-Smith R, Robinson JD, et al. . The influence of provider communication behaviors on parental vaccine acceptance and visit experience. Am J Public Health 2015;105:1998–2004. 10.2105/AJPH.2014.302425 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Brewer NT, Hall ME, Malo TL, et al. . Announcements versus conversations to improve HPV vaccination coverage: a randomized trial. Pediatrics 2017;139. 10.1542/peds.2016-1764. [Epub ahead of print: 05 Dec 2016]. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Henrikson NB, Opel DJ, Grothaus L, et al. . Physician communication training and parental vaccine Hesitancy: a randomized trial. Pediatrics 2015;136:70–9. 10.1542/peds.2014-3199 [DOI] [PubMed] [Google Scholar]
- 30.Miller WR. Motivational interviewing: research, practice, and puzzles. Addict Behav 1996;21:835–42. 10.1016/0306-4603(96)00044-5 [DOI] [PubMed] [Google Scholar]
- 31.Motivational Interview Motivational interviewing. Available: http://www.motivationalinterview.org/index.html [Accessed 01 Feb 2013].
- 32.Berg CJ, Thomas JL, An LC, et al. . Change in smoking, diet, and walking for exercise in blacks. Health Educ Behav 2012;39:191–7. 10.1177/1090198111432252 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Brand VS, Bray KK, MacNeill S, et al. . Impact of single-session motivational interviewing on clinical outcomes following periodontal maintenance therapy. Int J Dent Hyg 2013;11:134–41. 10.1111/idh.12012 [DOI] [PubMed] [Google Scholar]
- 34.Cooper L. Combined motivational interviewing and cognitive-behavioral therapy with older adult drug and alcohol abusers. Health Soc Work 2012;37:173–9. 10.1093/hsw/hls023 [DOI] [PubMed] [Google Scholar]
- 35.Leask J, Kinnersley P, Jackson C, et al. . Communicating with parents about vaccination: a framework for health professionals. BMC Pediatr 2012;12:154. 10.1186/1471-2431-12-154 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.McCain J. To heal the body, get into the patient's head: motivational interviewing: to improve adherence. Biotechnol Healthc 2012;9:10–12. [PMC free article] [PubMed] [Google Scholar]
- 37.Thomas ML, Elliott JE, Rao SM, et al. . A randomized, clinical trial of education or motivational-interviewing-based coaching compared to usual care to improve cancer pain management. Oncol Nurs Forum 2012;39:39–49. 10.1188/12.ONF.39-49 [DOI] [PubMed] [Google Scholar]
- 38.Boutin-Foster C, Scott E, Rodriguez A, et al. . The trial using motivational interviewing and positive affect and Self-Affirmation in African-Americans with hypertension (triumph): from theory to clinical trial implementation. Contemp Clin Trials 2013;35:8–14. 10.1016/j.cct.2013.02.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Gabbay RA, Añel-Tiangco RM, Dellasega C, et al. . Diabetes nurse case management and motivational interviewing for change (dynamic): results of a 2-year randomized controlled pragmatic trial. J Diabetes 2013;5:349–57. 10.1111/1753-0407.12030 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Gance-Cleveland B. Motivational interviewing for adolescent obesity. Am J Nurs 2013;113:11. 10.1097/01.NAJ.0000425726.07495.ff [DOI] [PubMed] [Google Scholar]
- 41.Hides L, Carroll S, Scott R, et al. . Quik fix: a randomized controlled trial of an enhanced brief motivational interviewing intervention for alcohol/cannabis and psychological distress in young people. Psychother Psychosom 2013;82:122–4. 10.1159/000341921 [DOI] [PubMed] [Google Scholar]
- 42.Klimas J, Field C-A, Cullen W, et al. . Psychosocial interventions to reduce alcohol consumption in concurrent problem alcohol and illicit drug users: cochrane review. Syst Rev 2013;2:3. 10.1186/2046-4053-2-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Nirenberg T, Baird J, Longabaugh R, et al. . Motivational counseling reduces future police charges in court referred youth. Accid Anal Prev 2013;53:89–99. 10.1016/j.aap.2013.01.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Rongkavilit C, Naar-King S, Wang B, et al. . Motivational interviewing targeting risk behaviors for youth living with HIV in Thailand. AIDS Behav 2013;17:2063–74. 10.1007/s10461-013-0407-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Schoonheim-Klein M, Gresnigt C, van der Velden U. Influence of dental education in motivational interviewing on the efficacy of interventions for smoking cessation. Eur J Dent Educ 2013;17:e28–33. 10.1111/j.1600-0579.2012.00755.x [DOI] [PubMed] [Google Scholar]
- 46.Ski CF, Thompson DR. Motivational interviewing as a brief intervention to improve cardiovascular health. Eur J Cardiovasc Nurs 2013;12:226–9. 10.1177/1474515112472271 [DOI] [PubMed] [Google Scholar]
- 47.Tse MMY, Vong SKS, Tang SK. Motivational interviewing and exercise programme for community-dwelling older persons with chronic pain: a randomised controlled study. J Clin Nurs 2013;22:1843–56. 10.1111/j.1365-2702.2012.04317.x [DOI] [PubMed] [Google Scholar]
- 48.Dempsey AF, Pyrznawoski J, Lockhart S, et al. . Effect of a health care professional communication training intervention on adolescent human papillomavirus vaccination: a cluster randomized clinical trial. JAMA Pediatr 2018;172:e180016. 10.1001/jamapediatrics.2018.0016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Resnicow K, Teixeira PJ, Williams GC. Efficient allocation of public health and behavior change resources: the “difficulty by motivation” matrix. Am J Public Health 2017;107:55–7. 10.2105/AJPH.2016.303526 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Seither R, Calhoun K, Mellerson J, et al. . Vaccination coverage among children in kindergarten - United States, 2015-16 School Year. MMWR Morb Mortal Wkly Rep 2016;65:1057–64. 10.15585/mmwr.mm6539a3 [DOI] [PubMed] [Google Scholar]
- 51.Sadaf A, Richards JL, Glanz J, et al. . A systematic review of interventions for reducing parental vaccine refusal and vaccine hesitancy. Vaccine 2013;31:4293–304. 10.1016/j.vaccine.2013.07.013 [DOI] [PubMed] [Google Scholar]
- 52.Leask J. Target the fence-sitters. Nature 2011;473:443–5. 10.1038/473443a [DOI] [PubMed] [Google Scholar]
- 53.Opel DJ, Mangione-Smith R, Taylor JA, et al. . Development of a survey to identify vaccine-hesitant parents: the parent attitudes about childhood vaccines survey. Hum Vaccin 2011;7:419–25. 10.4161/hv.7.4.14120 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Opel DJ, Taylor JA, Mangione-Smith R, et al. . Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine 2011;29:6598–605. 10.1016/j.vaccine.2011.06.115 [DOI] [PubMed] [Google Scholar]
- 55.Opel DJ, Taylor JA, Zhou C, et al. . The relationship between parent attitudes about childhood vaccines survey scores and future child immunization status: a validation study. JAMA Pediatr 2013;167:1065–71. 10.1001/jamapediatrics.2013.2483 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Glynn RJ, Brookhart MA, Stedman M, et al. . Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers. Med Care 2007;45:S38–43. 10.1097/MLR.0b013e318070c0a0 [DOI] [PubMed] [Google Scholar]
- 57.Raab GM, Butcher I. Balance in cluster randomized trials. Stat Med 2001;20:351–65. [DOI] [PubMed] [Google Scholar]
- 58.Kraschnewski JL, Keyserling TC, Bangdiwala SI, et al. . Optimized probability sampling of study sites to improve generalizability in a multisite intervention trial. Prev Chronic Dis 2010;7:A10. [PMC free article] [PubMed] [Google Scholar]
- 59.Chaudhary MA, Moulton LH. A SAS macro for constrained randomization of group-randomized designs. Comput Methods Programs Biomed 2006;83:205–10. 10.1016/j.cmpb.2006.04.011 [DOI] [PubMed] [Google Scholar]
- 60.Dickinson LM, Beaty B, Fox C, et al. . Pragmatic cluster randomized trials using covariate constrained randomization: a method for practice-based research networks (PBRNs). J Am Board Fam Med 2015;28:663–72. 10.3122/jabfm.2015.05.150001 [DOI] [PubMed] [Google Scholar]
- 61.Committee on Infectious Diseases, American Academy of Pediatrics Recommended childhood and adolescent immunization schedule--United States, 2014. Pediatrics 2014;133:357–63. 10.1542/peds.2013-3965 [DOI] [PubMed] [Google Scholar]
- 62.Davis DA, Thomson MA, Oxman AD, et al. . Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274:700–5. 10.1001/jama.274.9.700 [DOI] [PubMed] [Google Scholar]
- 63.Grimshaw JM, Shirran L, Thomas R, et al. . Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001;39:II2–45. [PubMed] [Google Scholar]
- 64.Bauchner H, Simpson L, Chessare J. Changing physician behaviour. Arch Dis Child 2001;84:459–62. 10.1136/adc.84.6.459 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Merriam SB, Cafrrarella RS. Learing in adulthood. San Francisco, CA: Jossey-Bass, 2008. [Google Scholar]
- 66.Bero LA, Grilli R, Grimshaw JM, et al. . Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane effective practice and organization of care review group. BMJ 1998;317:465–8. 10.1136/bmj.317.7156.465 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Grimshaw JM, Eccles MP, Walker AE, et al. . Changing physicians' behavior: what works and thoughts on getting more things to work. J Contin Educ Health Prof 2002;22:237–43. 10.1002/chp.1340220408 [DOI] [PubMed] [Google Scholar]
- 68.O'Brien MA, Rogers S, Jamtvedt G, et al. . Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2007;308:CD000409 10.1002/14651858.CD000409.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Rao JK, Anderson LA, Inui TS, et al. . Communication interventions make a difference in conversations between physicians and patients: a systematic review of the evidence. Med Care 2007;45:340–9. 10.1097/01.mlr.0000254516.04961.d5 [DOI] [PubMed] [Google Scholar]
- 70.Ivers N, Jamtvedt G, Flottorp S, et al. . Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;154:CD000259. 10.1002/14651858.CD000259.pub3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Dearing JW. Applying diffusion of innovation theory to intervention development. Res Soc Work Pract 2009;19:503–18. 10.1177/1049731509335569 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Davidson EM, Liu JJ, Bhopal R, et al. . Behavior change interventions to improve the health of racial and ethnic minority populations: a tool kit of adaptation approaches. Milbank Q 2013;91:811–51. 10.1111/1468-0009.12034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.Netto G, Bhopal R, Lederle N, et al. . How can health promotion interventions be adapted for minority ethnic communities? five principles for guiding the development of behavioural interventions. Health Promot Int 2010;25:248–57. 10.1093/heapro/daq012 [DOI] [PubMed] [Google Scholar]
- 74.Luman ET, Barker LE, Shaw KM, et al. . Timeliness of childhood vaccinations in the United States: days undervaccinated and number of vaccines delayed. JAMA 2005;293:1204–11. 10.1001/jama.293.10.1204 [DOI] [PubMed] [Google Scholar]
- 75.Centers for Disease Control and Prevention (CDC) Progress in immunization information systems - United States, 2012. MMWR Morb Mortal Wkly Rep 2013;62:1005–8. [PMC free article] [PubMed] [Google Scholar]
- 76.Jackson ML, Henrikson NB, Grossman DC. Evaluating Washington state's immunization information system as a research tool. Acad Pediatr 2014;14:71–6. 10.1016/j.acap.2013.10.002 [DOI] [PubMed] [Google Scholar]
- 77.Heritage J, Maynard DW. Communication in medical care : interaction between primary care physicians and patients. 1 edn New York: Cambridge University Pres, 2006. [Google Scholar]
- 78.Rubin HR, Gandek B, Rogers WH, et al. . Patients' ratings of outpatient visits in different practice settings. results from the medical outcomes study. JAMA 1993;270:835–40. [PubMed] [Google Scholar]
- 79.Wissow LS, Roter D, Bauman LJ, et al. . Patient-provider communication during the emergency department care of children with asthma. The National cooperative inner-city asthma study, National Institute of allergy and infectious diseases, NIH, Bethesda, MD. Med Care 1998;36:1439–50. 10.1097/00005650-199810000-00002 [DOI] [PubMed] [Google Scholar]
- 80.Bean-Mayberry BA, Chang C-CH, McNeil MA, et al. . Patient satisfaction in women's clinics versus traditional primary care clinics in the Veterans administration. J Gen Intern Med 2003;18:175–81. 10.1046/j.1525-1497.2003.20512.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Tom JO, Mangione-Smith R, Solomon C, et al. . Integrated personal health record use: association with parent-reported care experiences. Pediatrics 2012;130:e183–90. 10.1542/peds.2011-1786 [DOI] [PubMed] [Google Scholar]
- 82.American Institutes of Research on behalf of the Robert Wood Johnson Foundation How to report results of the CAHPS clinician & group survey, 2010. Available: http://www.rwjf.org/en/research-publications/find-rwjf-research/2010/09/how-to-report-results-of-the-cahps-clinician-group-survey.html [Accessed 7 Nov 2013].
- 83.Hays RD. The outpatient satisfaction questionnaire (OSQ-37): executive summary. Santa Monica, CA: RAND, 1995. [Google Scholar]
- 84.Stewart MA. What is a successful doctor-patient interview? A study of interactions and outcomes. Soc Sci Med 1984;19:167–75. 10.1016/0277-9536(84)90284-3 [DOI] [PubMed] [Google Scholar]
- 85.Yancy WS, Macpherson DS, Hanusa BH, et al. . Patient satisfaction in resident and attending ambulatory care clinics. J Gen Intern Med 2001;16:755–62. 10.1111/j.1525-1497.2001.91005.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Little RJA. A test of missing completely at random for multivariate data with missing values. J Am Stat Assoc 1988;83:1198–202. 10.1080/01621459.1988.10478722 [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2020-039299supp001.pdf (784.7KB, pdf)