Methodology for Knowledge Synthesis of the Management of Vaccination Pain and Needle Fear

Supplemental Digital Content is available in the text.

Abstract

Background:

A knowledge synthesis was undertaken to inform the development of a revised and expanded clinical practice guideline about managing vaccination pain in children to include the management of pain across the lifespan and the management of fear in individuals with high levels of needle fear. This manuscript describes the methodological details of the knowledge synthesis and presents the list of included clinical questions, critical and important outcomes, search strategy, and search strategy results.

Methods:

The Grading of Assessments, Recommendations, Development and Evaluation (GRADE) and Cochrane methodologies provided the general framework. The project team voted on clinical questions for inclusion and critically important and important outcomes. A broad search strategy was used to identify relevant randomized-controlled trials and quasi-randomized-controlled trials. Quality of research evidence was assessed using the Cochrane risk of bias tool and quality across studies was assessed using GRADE. Multiple measures of the same construct within studies (eg, observer-rated and parent-rated infant distress) were combined before pooling. The standardized mean difference and 95% confidence intervals (CI) or relative risk and 95% CI was used to express the effects of an intervention.

Results:

Altogether, 55 clinical questions were selected for inclusion in the knowledge synthesis; 49 pertained to pain management during vaccine injections and 6 pertained to fear management in individuals with high levels of needle fear. Pain, fear, and distress were typically prioritized as critically important outcomes across clinical questions. The search strategy identified 136 relevant studies.

Conclusions:

This manuscript describes the methodological details of a knowledge synthesis about pain management during vaccination and fear management in individuals with high levels of needle fear. Subsequent manuscripts in this series will present the results for the included questions.

A series of systematic reviews was planned to update the 2010 Help ELiminate Pain in KIDS Team (HELPinKIDS) clinical practice guideline for the management of acute pain during childhood vaccine injections.1 The decision for a new knowledge synthesis on this topic was informed by 2 factors: (1) a large number of new published trials, warranting reexamination of original recommendations; and (2) interest by stakeholder organizations in expanding the scope of the original guideline to attain a more comprehensive approach to the topic. These additional domains included the inclusion of adults and the management of fear in individuals with high levels of needle fear. As a result of the changes in scope, the team name was changed to HELPinKids&Adults.

This manuscript describes the methodological details used to carry out the systematic reviews, including: selection of the clinical question domains and outcomes, search strategy, data extraction, quality assessment, and data synthesis approach. It also includes the results for included clinical questions, critical and important outcomes, and the number of studies identified from the search strategy. The results of the knowledge synthesis of each included question is presented separately, in other manuscripts in this series, according to type of intervention examined.

METHODS

The Grading of Assessments, Recommendations, Development and Evaluation (GRADE)2 and Cochrane Handbook methodologies3 provided the general framework for the systematic reviews. A working group including 6 individuals (ie, evidence leads: A.T., C.M.M., V.S., R.P.R., C.T.C., M.N.) led by the first author (A.T.) was convened to oversee the knowledge synthesis.

Protocol and Registration

This systematic review was registered on the Prospero register (registration number: CRD42014013527). The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline4 was used to guide reporting.

Eligibility Criteria

Using Appraisal of Guidelines for Research and Evaluation-II principles (http://www.agreetrust.org) and GRADE methodology as guidance, HELPinKids&Adults, an interdisciplinary panel of clinicians, researchers, policy makers, and consumer stakeholders involved in aspects of guideline development and implementation, vaccination, and pain from across Canada identified clinical questions for inclusion.

Forty-seven candidate clinical question domains (including population, intervention, comparison) were initially proposed for inclusion. Questions were identified from the prior guideline, clinical practice, and existing research. An independent electronic vote was carried out to determine which candidate clinical question domains would be considered further. A cut-off of >2/3 majority in favor of including a clinical question domain was used as the threshold for preliminary inclusion. Using this method, 37 question domains were retained as preliminary questions.

Outcomes for each preliminary question domain were then selected by having team members independently vote on the importance of 13 candidate outcomes identified by them (delineated below) using a scoring system of 1 to 9. Voting was carried out electronically. Consistent with the GRADE framework, outcomes with a mean score of ≥7 were defined as critically important for decision making; those with a mean score of 4 to 6 were defined as important and included as outcomes of interest to the review; the remainder (mean score <4) were not considered further.5 In selecting outcomes, consideration was given to the perspectives of individuals undergoing vaccination, parents of children undergoing vaccination, and clinicians administering vaccinations; however, the perspective of the individual undergoing vaccination was prioritized to guide selection.

Modifications to clinical question domains and outcomes were made after a preliminary review and discussion of the research evidence at an in-person meeting of the project team. Several questions were removed due to a lack of confidence regarding the applicability of the evidence base to the vaccination context, and others were added to examine additive effects of combined interventions of interest and/or alterations in the timing or delivery of the interventions.

Composition of Clinical Questions—Participants, Interventions, Comparisons, Outcomes, Study Designs (PICOS)

Participants included individuals of all ages undergoing vaccine injections in inpatient and outpatient settings, including schools, and individuals with high levels of needle fear. If no data existed for vaccine injections, then the closest related procedure or context was included (eg, venipuncture in outpatient clinic). Interventions included single and combination interventions used for vaccine injection pain management (or related procedures/contexts if there were no data for vaccine injections) including: procedural strategies, physical strategies, pharmacological strategies, psychological (and information provision) strategies, and process (education/implementation) strategies. Interventions for the management of fear in individuals with high levels of needle fear were also included. Comparators included: no treatment control (no documented intervention above usual/routine care) or other comparators, as specified by the clinical question. Cointerventions were allowed depending on the clinical question. The additive benefit of an intervention over another was also examined, as specified by the clinical question. Potential outcomes considered included: pain, fear, distress, preferences (for individuals undergoing vaccination, parents of children undergoing vaccination, clinicians administering vaccinations), satisfaction (individuals, parents, clinicians), fainting, procedure outcomes (duration, success), parent fear, knowledge about pain interventions (individuals, parents, clinicians), pain intervention utilization (individuals, parents, clinicians), safety outcomes, vaccine compliance, and/or memory of pain and/or fear. Study designs considered included randomized-controlled trials (RCTs) and quasi-RCTs with between-groups (parallel) and cross-over designs. Cluster trials were also included.

Information Sources and Search Strategy

The OvidSP platform was used to run the search strategy in MEDLINE, EMBASE, and PsycINFO databases; EBSCOHost was used for CINAHL and ProQuest was used for ProQuest Dissertations & Theses Global. The databases were searched from their date of inception; the last update was February 26, 2015. No language restrictions were applied. Search terms used to identify studies for inclusion were determined by the authors based on their content expertise in this area in consultation with an academic librarian (E.U.), who conducted the searches. Additional studies were identified from reference lists of included studies and by consulting experts working in this topic area. The titles and abstracts of retrieved citations were imported into an EndNote library and scanned by 2 reviewers (A.T., V.S.). The reviewers identified citations to be retrieved as full-text articles, and these were assessed for eligibility by 2 reviewers (A.T., C.M.M.). Reviewers were not blinded to the authors or settings of the studies in the scanned articles.

Inclusion and Exclusion Criteria

The review included original research articles involving: (1) individuals of all ages; (2) interventions included in the clinical questions; (3) vaccine injections and/or the closest related procedure or context to vaccine injections; and (4) highest level of evidence available (ie, RCTs and quasi-RCTs). Studies that were published as full reports or short reports were included, as well as published academic theses. We excluded published abstracts, letters, commentaries, and editorials.

Data Extraction

Data from eligible studies were extracted and checked by at least 2 reviewers in customized data extraction forms. Before extraction, all evidence leads provided feedback regarding the usability and comprehensiveness of the extraction forms. Data forms used an outcome-based approach, as specified by the GRADE methodology.6 Reviewers resolved any disagreements through discussion or, if required, consultation with a third individual (ie, the project lead and first author, A.T.).

Data extracted from each study included: author; country; year of publication; age of participants; sample size; design details; procedure and intervention details; comparison; and critical outcomes. Summary statistics (eg, means, SDs) and sample sizes were extracted for critically important and important outcomes for each clinical question by at least 2 reviewers using the data extraction sheet. Studies including multiple treatment arms could contribute to several analyses (ie, the same study could provide data for several clinical questions). Only data from the relevant treatment arms were included in any particular analysis. If a study provided multiple arms for 1 analysis, the sample size was divided by the appropriate number so as not to double-count individuals within the analysis.

If not provided, summary statistics were estimated from graphs and/or calculated from medians and ranges or other parameters (eg, SEs, interquartile ranges, 95% confidence intervals [CIs]) using established formulae7 and statistical programs (RevMan version 5.2; the Cochrane Collaboration, Copenhagen, Denmark). If not provided, sample size was estimated by dividing the total sample size by the number of groups. When data could not be obtained, a descriptive summary of the findings, as reported by the authors, was included in the review. Data were abstracted using an intent-to-treat (ITT) approach; however, if ITT results were not available, a per-protocol approach was used. Attempts were made to contact study authors by email in situations whereby additional information was needed to clarify methods and/or summary statistics.

Steps were undertaken to provide unique identifiers for included studies in the software programs used to carry out the review (ie, RevMan, GRADEprofiler). Studies were identified using the following notation: “First Author” “Year of Publication” [eg, Taddio 2014]. If studies contributed to multiple analyses, then “(#)” was added to enable their discernment [eg, Taddio 2014 (1)]. If the same author published more than 1 study in the same year, then a lower case letter was added for subsequent articles [eg, Taddio 2014 a (1)].

Quality of Research Evidence in Individual Studies

The included trials were not masked to reviewers. Methodological quality of included studies was assessed by at least 2 reviewers at the outcome level using the Cochrane risk of bias tool (https://bmg.cochrane.org/assessing-risk-bias-included-studies). Domains evaluated included: sequence generation, allocation concealment, blinding of study participants and personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting, and other sources of bias. When available, published studies were compared with trial registration information to evaluate selective outcome reporting. Ratings incorporated information from both the published paper and any supplemental data provided by the authors. Discrepancies were resolved by consensus and with the assistance of a third reviewer, if necessary. The results were used to rate the quality of the evidence and to evaluate heterogeneity in meta-analyses.

Delineation of Outcomes

Pain, fear, and distress were typically prioritized as critical outcomes; working definitions for these constructs are given in Table 1. In a separate manuscript in this supplement, these constructs are further delineated and explored in the needle context using a developmental perspective.8 Consistent with GRADE methodology, critical outcome selection was influenced by the available evidence base.9 In the absence of data for pain due to the inclusion of participants in studies whereby their self-report is not possible (eg, infants), distress was the critically important outcome.

TABLE 1.

Definitions Used for Knowledge Synthesis

The primary assessment method for subjective outcomes (eg, pain, fear) was self-report. Self-reported pain and fear were typically measured using a Visual Analog Scale (VAS), Numerical Rating Scale (NRS), or faces scale. If self-report was not possible (eg, infant unable to provide self-report), observational measures were used. Observational measures could also be considered if self-report was potentially unreliable (eg, child younger than 7 y). As observational methods typically cannot distinguish between pain and fear, the term used to describe these measures was distress. Behavioral scales (typically including facial actions, body movements, and/or cry) or global rating scales were used for observer-reported distress. Physiological measures reflect overall nonspecific arousal and were not considered. If onlookers (eg, parents) provided ratings of their own fear, distress, or anxiety while another individual was undergoing vaccination, this was reported as fear to maintain consistency in terminology.

Outcomes that were evaluated at multiple timepoints were analyzed according to the following phases to more precisely describe the intervention effects: (1) the preprocedure phase, which occurred postintervention but before vaccine injection(s); (2) the acute phase (within the first minute of needle puncture and vaccine injection); and (3) the recovery phase (1 to 5 min after vaccine injection(s)). Outcomes were also assessed over combinations of these procedure phases (eg, distress during acute and recovery phases).

Summary Measures and Data Synthesis

Qualitative (descriptive) and quantitative (meta-analytic) data synthesis methods were used. Quantitative syntheses were conducted using RevMan version 5.2 (Cochrane Collaboration). As per the GRADE approach,10 continuous outcome data were combined using the standardized mean difference (SMD) with 95% CI. SMD allowed standardization of results to a uniform scale. The SMD expresses the size of the intervention effect in each study relative to the variability observed in that study. For cross-over trials, continuous data were combined using the statistical approach described in the Cochrane review by Pillai Riddell.11 For cluster trials, the numbers provided were used, without consideration of the intraclass correlation coefficient.

For the purposes of this synthesis, an SMD as low as 0.2, representing a small effect,12 was considered important as pain from vaccine injections is by nature an iatrogenic harm of these procedures for which the team agreed that interventions should be given even if there is limited benefit. Dichotomous data (eg, presence/absence of pain based on a predetermined cut-off value), were combined using relative risk and 95% CI. A random-effects model was selected for pooling data.

Within specific clinical question domains, separate analyses were planned a priori for different age groupings to account for differences in the developmental level of recipients, and/or the interventions and their implementation (eg, timing or delivery method). For example, the effect of positioning on pain during vaccine injections was examined separately for neonates who were undergoing skin-to-skin care during injection (vs. lying supine) and children who were sitting upright during injection (vs. lying supine). Age categorizations also considered the existing evidence base (ie, the age groups for which there was evidence) and typically included; early childhood (0 to 3 y), childhood (>3 to 12 y), adolescence (>12 to 17 y), and adulthood (≥18 y). Early childhood was further subdivided into the first month of life (neonate), first year of life (infant), the first 2 years of life, and the first 3 years of life, as appropriate.

If a study included assessments of the same outcome measure at multiple timepoints within the same phase of the procedure (eg, acute distress measured at 15, 30, 45, and 60 s), or multiple outcome measures were used for the same construct in the same phase (eg, acute distress measured using VAS and cry duration), including measurement by proxy (eg, clinician and parent-rated), the data were combined into a single point estimate and associated variance using established statistical methods13 and an estimated correlation of 0.25.14 This comprehensive approach to data synthesis resulted in the ability to include all the data pertaining to critical and important outcomes in the meta-analysis and minimized potential bias from “cherry picking” outcomes from individual studies. The sample size used for the meta-analysis included the maximum number for any single assessment. The discrepancy in the sample sizes between assessments usually ranged between 1 and 2 participants; hence, this approach was deemed acceptable.

If multiple methods of presentation of the same outcome were included (eg, total score and difference score from baseline), only the difference score from baseline was used for the meta-analysis. If a study included both nonblinded and blinded assessments, outcomes were combined for blinded assessments only. This reduced the potential for a biased estimate in the meta-analysis. If blinded assessments were not available for the outcome in individual studies, then nonblinded assessments were used in the meta-analysis. Study quality ratings reflected blinding status. Scores were standardized to a 0 to 10 scale before pooling; however, in a minority of instances, outcome data could not be standardized due to missing information regarding the range of the measure.

Clinical heterogeneity was assessed by noting the differences among studies in the following variables: age group (participants), country, intervention, comparison, type of vaccine, injection method, cointerventions (eg, simultaneous use of other pain-reducing strategies), outcome assessment methods, and other study-specific design features.

Statistical heterogeneity was assessed using the I² index (percentage of total variability due to heterogeneity between studies) and χ² tests. For I², the following template was used to judge the results regarding heterogeneity: 0% to 40%, may not be important; 30% to 60%, may be moderate; 50% to 90%, may be substantial; and 75% to 100%, may be considerable. For I² values of >95%, the magnitude and accompanying P value from the χ² test were considered in the overall interpretation. Funnel plots were performed to assess for the possibility of publication bias if there were sufficient numbers of trials (>10).3

Quality of Research Evidence Across Studies

As per the GRADE approach, the quality of evidence from outcomes across studies was assessed. The quality assessment considered 5 factors: risk of bias (study limitations), inconsistency (heterogeneity of results), indirectness (evidence does not come from direct comparisons of interest), imprecision (sample size and CI), and publication bias (trials with positive findings more likely to be published). The quality of evidence rating for specific outcomes was assigned to 4 categories: high, moderate, low, and very low evidence, all reflecting the degree of confidence in the quantitative measure of benefit or harm suggested by the systematic review.15 Evidence profiles and summary of findings tables were created for each clinical question through the GRADEprofiler software (version 3.6.1) in which judgments pertaining to the evaluation of the quality of evidence were recorded with an extensive array of explanatory footnotes.

Additional Analyses

Additional analyses were carried out according to quality and/or study methodology (eg, removal of data from a study with serious methodological flaw(s) and/or major difference in study methodology).

RESULTS

Altogether, 55 clinical questions were included in the review; they are displayed in Table 2 along with relevant critically important and important outcomes. Individual clinical questions were organized into 6 categories: (1) procedural interventions; (2) physical interventions; (3) pharmacological interventions; (4) psychological interventions; (5) process interventions; and (6) interventions for individuals with high levels of needle fear. Pain, fear, and distress were typically prioritized as critically important outcomes across clinical questions.

TABLE 2.

Clinical Questions and Outcomes

The flow of studies and search strategy used to identify the relevant literature are displayed in Figure 1 and Supplemental Digital Content, Table 1, http://links.lww.com/CJP/A182, respectively. Altogether, 82,234 unique citations were screened and 136 were included in the knowledge synthesis. Of included studies, 25% were from low-income and middle-income countries.

FIGURE 1.

Guideline flow chart. **The 12 studies in this group contained data that were either superseded or reanalyzed in the group of 136 included studies. They were noted as containing duplicate data.

DISCUSSION

This manuscript describes the methodological details of a comprehensive knowledge synthesis undertaken to inform an update to the 2010 HELPinKIDS clinical practice guideline for vaccination pain management.1 The scope includes the management of vaccination pain and the management of fear in individuals with high levels of needle fear. The process for selecting clinical questions and relevant outcomes are described, as well as methods of data extraction, quality assessment, and data synthesis. The included questions and selected critically important and important outcomes are presented along with the search strategy and number of included studies.

There are several limitations to our data synthesis approach that require discussion. Firstly, we combined data from multiple measures of the same outcome before pooling results across studies and used a standardized measure of effect (ie, SMD). An alternative (and more typical) approach is to select results for a single and “best” measure from each study (ie, the measure with the most robust validity testing) and to pool data only if the outcome is measured using the same tool across studies (eg, pain measured using VAS). We elected to use the former approach for several reasons, including: (1) it allowed for inclusion of all the data from a study in the analysis improving generalizability of the results; (2) resulted in greater precision in estimates of treatment effect; and (3) there is currently no rationale for selecting any particular measure as the “gold standard” method of assessment across the outcomes included in the knowledge synthesis. Secondly, we defined the time intervals that bound the different phases of the vaccination procedure (eg, acute procedural distress was conceptualized as the first minute after vaccine injection). Although this decision was informed by prior research in the field, at present, the time intervals which optimally define the different procedural phases are not known. Moreover, the relative importance of these phases on the experience of pain is not known. It is possible that slightly different results would be obtained if only the data from specific tools were included or the boundaries for these time epochs were altered.

Strengths of our knowledge synthesis approach include the use of state-of-the-art methodologies (ie, GRADE and Cochrane) and a user-centric approach. Importantly, identification of clinical questions and associated outcomes to be included were informed by a broad group of stakeholders involved in vaccination across the lifespan; this approach resulted in data that are highly relevant to users. In particular, the inclusion of outcomes beyond “pain” allows for a more comprehensive synthesis than previously undertaken in this field. Outcomes that were identified as important to stakeholders can be used to inform the selection of outcomes that are included in future clinical trials. Moreover, aspects of data analysis described above (ie, combining different measures of the same construct, delineation of the effects of an intervention over time) are unique among knowledge syntheses in similar domains, and may serve as a template for others under taking knowledge syntheses. Finally, the inclusion of quasi-RCTs allowed for inclusion of more trials, particularly from low-income and middle-income countries, which improves generalizability of the findings.

In subsequent manuscripts in this series, the findings for the effects of each intervention included in the individual clinical questions are presented, including: details regarding study characteristics of included studies, GRADE evidence profiles and summary of findings tables, and interpretation of the results.16–22 Finally, we include a manuscript that outlines overarching limitations in the included evidence base and provides recommendations about areas worthy of additional investigation.23 Separately, we present the 2015 HELPinKids&Adults clinical practice guideline developed from this knowledge synthesis.24