Table 1.

Methods used to evaluate effectiveness of behavior change techniques (BCTs)

Evaluation method	What it involves	Strengths and limitations using PASS criteria	Total number of references included in the review (Supplementary File)
Experiments (including RCTs); For example, providing feedback on expired air carbon monoxide concentrations to aid smoking cessation (Shahab et al., 2011).	Adding or removing one or more BCTs under experimenter control and looking for differences in effectiveness.	P: Only feasible for evaluating small numbers of BCTs at any one time; resources required for adequately powered studies can be prohibitive; ethical and pragmatic barriers are often insuperable; timescales tend to be long (usually 3 or more years for experiments involving important outcomes). A: Generalization beyond the study population and setting is often problematic, particularly where informed consent is required and/or recruitment is low. Se: Where effects are found, can provide confidence in attributing these to BCTs, but this is still problematic when there is a loss to follow-up, differential uptake of the intervention, or potential bias in the measurement of outcomes. Sp: Inability to detect effects may be due to a wide range of factors other than ineffective BCT(s), including low power, inadequate delivery of the BCT(s), and low measurement accuracy.	73
Comparative observational studies; For example, identification of BCTs associated with higher success rates of stop smoking services in England (West et al., 2010).	Using naturally occurring variation in clinical or public health practice in inclusion of BCTs and outcomes to identify associations between BCT inclusion and intervention effectiveness.	P: Can be very cost effective if data are already available or can be recorded as part of routine care; completeness and accuracy of data collection are often low; rely on naturally occurring variation in use of BCTs; fidelity of delivery of BCTs may be low or unknown. A: Can involve “real-world” settings and populations making generalization less problematic. Se: Can make use of very large data sets increasing potential sensitivity; number of permutations of BCTs may undermine detection of interactions; susceptible to high bias and error of outcome measurement. Sp: Causality has to be inferred (usually by statistical adjustment for potential confounding variables such as mode of delivery, setting, population, and other BCTs); can result in high false positive rate where multiple BCTs are being considered.	4
Meta-analyses of experimental studies; For example, implementation intentions as actions plans to promote behavior change (Gollwitzer & Sheeran, 2006).	Statistically pooling the results or two or more experiments evaluating one or more BCTs as above.	P: Can be conducted within a few months at relatively low cost compared with empirical studies; often there are too few studies that are sufficiently similar in terms of interventions and methodology; studies mostly involve testing packages of BCTs. A: Can provide confidence about generalizability across specific contexts; generalization of findings is still constrained by contextual factors of the studies included. Se: When there are enough high-quality studies, they can provide a high level of confidence in effectiveness of BCTs and provide robust effect size estimates. Sp: Can be biased by failure of researchers to report negative findings; fidelity of delivery of BCTs may be low or unknown or variable.	16
Meta-regressions; For example, identification of self-monitoring, goal setting, and actions plan as effective BCTs in promoting physical activity and healthy eating (Michie et al., 2009).	Identifying inclusion versus exclusion of BCTs or their combinations as moderators of effect sizes in meta-analyses of multi-component interventions.	P: Can be conducted within a few months at relatively low cost compared with empirical studies; often there are too few studies that are sufficiently similar in terms of interventions and methodology; studies mostly involve testing packages of BCTs; interventions and controls are often not described well enough to be able to identify BCTs and important contextual factors. A: Can provide confidence about generalizability across specific contexts; generalization of findings is still constrained by contextual factors of the studies included. Se: May detect effects that are too small to be picked up in individual studies; rely on large number of studies that vary in use of BCTs in intervention and control condition; apparent BCT effects may be due to other study features. Sp: Failure to detect effects may be due to a large number of factors in the contributing studies too little variation in BCT use across studies.	9
Meta-CART (Classification and Regression Trees) (Dusseldorp et al., 2014).	A set of computational learning methods that produce either “classification” or “regression” trees, depending on whether the dependent variable is categorical or numeric, respectively. Starting with a “root” node, the sample is partitioned successively to create a branching tree of nodes with each branch terminating in a “leaf”, which is the subsample that differs maximally from other subsamples on the dependent variable.	P: Can be conducted within a few months at relatively low cost compared with empirical studies; often there are too few studies that are sufficiently similar in terms of interventions and methodology; studies mostly involve testing packages of BCTs; interventions and controls are often not described well enough to be able to identify BCTs and important contextual factors. A: Can provide confidence about generalizability across specific contexts; generalization of findings is still constrained by contextual factors of the studies included. Se: Well suited for testing BCT interactions; susceptible to false positives; apparent BCT effects may be due to other study features. Sp: Only able to detect a small proportion of BCT interactions that might be effective without extremely large numbers of studies.	1
Characterizing effective interventions; For example, identifying BCTs included in effective behavioral support interventions for smoking cessation (Michie, Churchill & West, 2011).	Identifying BCTs included in interventions found to be effective in RCTs May vary in implementation from inclusion of BCTs that are present in at least one effective intervention to those that have been present in all effective interventions.	P: Relatively inexpensive and can be undertaken in a few months; relies on accurate characterization of BCTs in intervention conditions. A: Can provide confidence about generalizability across specific contexts; generalization of findings is still constrained by contextual factors of the studies included. Se: High probability of picking up BCTs that are effective among those that are frequently tested but unable to differentiate ones that are less effective from those that are tested less often. Sp: Likely to include BCTs that are ineffective but included as part of intervention packages.	32

BCTs behavior change techniques; RCTs randomized controlled trials; P Practicability; A Applicability; Se Sensitivity; Sp Specificity.