Skip to main content
NCBI home page
Pilot and Feasibility Studies logoLink to Pilot and Feasibility Studies
. 2026 May 13;12:96. doi: 10.1186/s40814-026-01836-4

Examining predictors of scale-up penalty in preliminary studies of health behavior interventions and subsequent larger trials: a meta-epidemiological review of diverse fields

Lauren von Klinggraeff 1,✉, Christopher D Pfledderer 2, Sarah Burkart 3, Kaitlyn Ramey 3, Michal Smith 3, Alexander C McLain 4, Bridget Armstrong 3, R Glenn Weaver 3, Anthony Okely 5,6, David Lubans 7,8,9, John P A Ioannidis 10, Russell Jago 11, Gabrielle Turner-McGrievy 12, James Thrasher 12, Xiaoming Li 12, Michael W Beets 3
PMCID: PMC13340402  PMID: 42129912

Abstract

Background

Scale-up penalty, a common phenomenon in which the promising effects found in early preliminary studies are substantially reduced when evaluated in a subsequent larger trial, can stall the advancement of health behavior interventions. In obesity-related behavioral interventions, changes to key features between a preliminary study and subsequent larger trials inflate scale-up penalty. The purpose of this study is to examine whether changes in intervention features occur in other behavioral disciplines that utilize a similar developmental continuum wherein smaller-scale preliminary studies inform larger-scale trials, and whether changes in key features inflate scale-up penalty.

Methods

We conducted a systematic review identifying preliminary studies followed by a larger trial conducted by the same author(s) (i.e., a study pair) in four areas—tobacco/smoking cessation, alcohol use, interpersonal violence, and sexually transmitted diseases. We coded intervention features in the preliminary study and larger trial to capture changes in key study features (e.g., who delivered the intervention). Multi-level meta-regressions estimated the association between the changes to key study features and change in standardized mean difference for health outcomes and calculated scale-up penalty.

Results

We identified 222 effects across 69 study pairs of preliminary studies with subsequent larger trials. Fifty-eight study pairs (84%) contained at least one change to a key study feature. Forty-nine study pairs were included in the meta-regressions (k = 154 effects). Changing the delivery agent, outcome measure, intervention duration, or implementation support in subsequent larger trials followed the hypothesized pattern with larger scale-up penalty (i.e., less effective at-scale) relative to study pairs that did not change those features.

Conclusions

Multiple disciplines outside the obesity field make changes to key study features between preliminary studies and subsequent larger trials. Across disciplines, changes to the intervention delivery agent, intervention duration, outcome measurement, and reducing implementation support in subsequent larger trials are associated with inflated scale-up penalty. Recognition of the importance of key study features in preliminary studies and iterative intervention development could improve intervention scale-up.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-026-01836-4.

Background

Health behavior interventions with promising effects in early preliminary studies show reduced or null effects when evaluated with a larger number of participants. This phenomenon, referred to as scale-up penalty, voltage drop, and failure to scale [1, 2] may stall the development of effective, population-level health interventions and waste valuable resources [3, 4]. Though it is not uncommon, nor unexpected [5–7], there may be factors that influence the magnitude of scale-up penalty, presenting an opportunity to preemptively identify and mitigate some of the factors that inhibit successful scale-up [1, 8].

Preliminary studies (e.g., pilot/feasibility studies) are one of the first steps in developing and testing behavioral interventions for population-level impact and provide valuable information about feasibility, acceptability, appropriateness, and potential efficacy [9–13]. Though a substantial body of work exists which demonstrates that early-stage, often small sample size, studies do not provide reliable estimates of outcomes and should not be used as evidence of efficacy, they are frequently used to garner funding and make decisions about which studies should be pursued further. Therefore, preliminary studies play a critical role in the development of health promotion interventions [14–16].

The evidence produced from preliminary studies is intertwined with key features of the study, such as who delivers the intervention, to what population, for how long, and at what intensity. Changes to these features can produce changes in an intervention’s feasibility, acceptability, appropriateness, and potential efficacy [17, 18]. This is problematic when preliminary studies are used as evidence of an intervention’s viability for larger-scale testing, but the features of the preliminary study cannot be maintained, or are not maintained, in the larger trial [19].

In health behavior interventions targeting obesity-related behaviors (e.g., diet, physical activity), our research team has demonstrated that when key aspects of the preliminary study’s intervention are altered in subsequent larger trials, this change is associated with a larger-than-expected scale-up penalty [19, 20]. Put another way, the presence of certain features in preliminary studies inflates their estimates of effectiveness. We have called these features the risk of generalizability biases (RGBs) because their presence in preliminary study represents an increased risk that the findings of that preliminary study are biased (i.e., a systematic over or underestimation) and will not hold in a larger efficacy/effectiveness trial (i.e., generalize) [19].

The RGB framework identifies nine potential types of bias and can be conceptualized similarly to the to the Cochrane Risk of Bias (RoB) tool [21]. Within the RoB tool, a study that uses non-random subject allocation is considered at risk for selection bias as the presence of non-random allocation threatens the internal validity of the findings. Within the RGB framework, the delivery of a preliminary intervention by a lead researcher is at risk for delivery agent bias as the presence of delivery agent bias is a threat to the generalizability of the findings.

A hallmark example of an RGB is delivery agent bias, wherein the intervention is delivered by a highly trained individual (e.g., principal investigator, graduate student) during a preliminary study, followed by a larger-scale trial in which the intervention is delivered by people with less expertise and/or different professional responsibilities (e.g., community health worker) [22, 23]. Changes to other features in a study, such as a change in setting (e.g., academic medical center to federally qualified health center), implementation support (e.g., high support to no support), target audience (e.g., high-socioeconomic status [SES] to low SES), intervention intensity (8 sessions vs. 12), intervention duration (e.g., 12 weeks vs. 1 year), and study measure (e.g., self-report to objective measure), demonstrate similar patterns.

In this study, we aim to determine the presence and potential impact of RGBs in behavioral interventions outside of those targeting obesity-related behaviors. Given that multiple areas of health promotion research follow similar intervention development frameworks, where preliminary studies inform larger-scale trials, it is possible that there are similar patterns of introducing features in a preliminary study and then removing those features in subsequent larger-scale trials. We have chosen to focus on health promotion interventions targeting tobacco use, alcohol use, interpersonal violence, and behaviors related to sexually transmitted infections because their target behaviors and outcomes are distinct from those in obesity research, though they follow the same intervention development patterns with preliminary studies informing subsequent larger trials [24–26]. We hypothesize that (1) similar to interventions targeting obesogenic behaviors, intervention studies in health promotion interventions targeting tobacco use, alcohol use, interpersonal violence, and behaviors related to sexually transmitted infections will contain changes to key study features (i.e., RGB) between preliminary studies and subsequent larger trial; and (2) changes to key study features between preliminary studies and subsequent larger trials will be associated with a larger scale-up penalty relative to studies that do not contain changes to key study features (i.e., study pairs that contain RGBs will be less effective at-scale).

Method

Systematic review

Our methods were informed by the Cochrane Handbook for Systematic Review of Interventions [27] and are reported according to the Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) Scoping Review and Abstract Extension statements (Supplementary Material) [28, 29].

Inclusion/exclusion criteria

Consistent with our prior work, a preliminary health behavior intervention was defined as an initial evaluation of a behavior-focused health intervention with primary goals to test the feasibility, acceptability, preliminary efficacy, or other developmental features of the intervention [19, 30]. Health behavior interventions were defined as coordinated sets of activities that aim to promote a health behavior by targeting one or more levels of influence, including interpersonal, intrapersonal, policy, community, macro-environments, micro-environments, and institutions [13, 31–33]. For the purpose of this study, no distinction was made between pilot studies and feasibility studies because both are commonly used to inform subsequent larger trials [34, 35]. Similarly, no distinction was made between internal and external preliminary studies [36]. No participant age requirements or date boundaries were applied. Studies had to be published in indexed, refereed journals as verified by Ulrich’s Web (http://ulrichsweb.serialssolutions.com). Studies had to be available as a full-text article in English.

Data sources and search strategy

Our team used the following procedures to identify pairs of preliminary studies and their subsequent larger trial of the same or similar behavior intervention addressing tobacco use disorder, alcohol use disorder, interpersonal violence, or behaviors related to increased sexually transmitted infections (Fig. 1). In step 1, we used controlled vocabulary terms (e.g., MeSH, Emtree), free-text terms, and Boolean operators to identify systematic reviews and/or meta-analysis across OVID Medline/PubMed; Embase/Elsevier; EBSCOhost; and Web of Science databases [37]. The search strategy and syntax are provided in the supplementary material. In step 2, our team uploaded identified systematic reviews and/or meta-analysis into an EndNote Library (v. X9.2) where they were reviewed by at least one trained research assistant (LV, KR) prior to retrieving full-text articles of all included studies within each review (similar to an umbrella review procedures). In step 3, we retrieved the full-text articles included within each systematic review and/or meta-analysis uploaded them into NVivo (v.12, Doncaster, Australia). NVivo text search query was used to identify each study included within each systematic review and/or meta-analysis as either a (1) self-identified preliminary testing of an intervention (e.g., contained the words “pilot,” “feasibility,” “preliminary,” “proof-of-concept,” “vanguard,” “novel,” or “evidentiary” [13, 35, 38] or (2) a larger-scale trial referring to prior preliminary work to flag sections of text (e.g., “protocol,” “previously,” “rationale,” “elsewhere described,” “prior work,” “informed by”). In step 4, we used forward and backward citation searches to pair studies. Studies identified as large-scale trials were “followed back” using the references in the publication to identify preliminary testing and publication of an intervention within the body of the article. Studies identified as preliminary studies were “followed forward” using the Web of Science Reference Search interface (e.g., identify subsequent published studies referencing the identified preliminary study as informative preliminary work). Successfully paired preliminary studies and large-scale trials were catalogued in Excel (Microsoft) and referred to as a pair of studies.

Fig. 1.

Fig. 1

Open in a new tab

Diagram of search procedures of locating pilot studies and larger-scale trial pairs. Note: Figure originally appears in Beets et al. [30]

Pairing criteria

Publications were considered a pair if (1) one publication self-identified as a preliminary study and (2) the larger trial utilized the same or similar intervention as the preliminary study, and (3) both studies were conducted by the same research team and/or lead investigator as determined by publication authorship.

Extracting study outcomes

Outcomes reporting impact on health-related behaviors were extracted by the research team (LV, KR, MS, SB, CDP). This included tobacco cessation rates (e.g., 7-day point prevalence, exhaled carbon monoxide levels, self-reported quit rate), drinking rates (e.g., addiction severity index alcohol composite score, units of alcohol consumed over the previous week), measures of interpersonal functioning (e.g., nonviolent discipline, physical victimization, child abuse potential inventory score), and measures representative of constructs associated with reduced infection rates (e.g., transmission risk behaviors, medication adherence, psychosocial functioning).

Coding the presence of the risk of generalizability biases

At least two reviewers (LV, MB, SB) independently reviewed each study pair to identify the presence/absence of RGBs using definitions presented in prior work (Table 1) [19, 30]. Where discrepancies in coding RGBs occurred between reviewers, a third reviewer was consulted, and agreement was reached by discussion. Within each study pair, each RGB could be classified as not present, present in the preliminary study only, or present in preliminary study and larger trial (i.e., carried forward). Intervention duration bias, intervention intensity bias, and measurement bias are biases describing difference between preliminary study features and larger trials and, if present, were coded as present in both the preliminary study and larger trial. For studies where an RGB was present in the larger trial but not the preliminary study, for example, where implementation support was provided in the larger trial but was not mentioned in the preliminary study, it was assumed to have also been provided in the preliminary study, even if not explicitly mentioned.

Table 1.

Operational definitions of the risk of generalizability biases

Bias Operationalized definition
Delivery agent bias Difference(s) in the level of expertise of the individual(s) who deliver the intervention in the preliminary study compared to who will deliver the intervention in larger trial(s)
Implementation support bias Difference(s) in the amount of support provided to implement the intervention
Setting bias Difference(s) in the setting where the intervention is delivered in the preliminary study compared to who will deliver the intervention in larger trial(s)
Target audience bias Difference(s) in the demographics of those that received the intervention in the preliminary study compared to who will deliver the intervention in larger trial(s)
Intervention intensity bias Difference(s) in the number and length of contacts in the preliminary study compared to who will deliver the intervention in larger trial(s)
Intervention duration bias Difference(s) in the length of the intervention provided in the preliminary study compared to who will deliver the intervention in larger trial(s)
Measurement bias Difference(s) in the measures employed in the current study and the measures used in the preliminary study compared to who will deliver the intervention in larger trial(s)
Open in a new tab

Note: Table 1 based on definitions originally appearing in Beets et al. [19]

Analytic procedures

Consistent with previous studies [19, 20], our research team extracted outcomes reported across pairs and entered them into an Excel file (Microsoft). In Excel, effect sizes were corrected for differences in the direction of the scales so that positive effect sizes corresponded to improvements in health behaviors in the intervention group. This was done for the simplicity of interpretive purposes so that all effect sizes could be summarized and compared within and across studies. We performed all necessary data transformations in Excel (e.g., standard errors and confidence intervals transformed into standard deviations). Next, outcomes reported within pairs were transferred into Comprehensive Meta-Analysis software (Biostat Inc., v3.3.07) to calculate the standardized mean difference (SMD) for each effect. After effects were calculated, the complete data file was exported as a.CSV and uploaded into R statistical software (v4.3.2) for analysis.

Meta-regression

Given the hierarchical structure of the data, random-effects meta-regression model (metafor) [39] with a robust variance estimator (clubSandwhich) [40] were used to account for lack of independence between measures of SDM. Effects (SMD, level 1) were nested within studies (level 2), which are nested within pairs (level 3). These models were repeated for all levels of each RGB such that each row represents a single model.

Calculating scale-up penalty

The difference in the SMD from the preliminary and larger-scale trial were quantified according to previously defined formulas for scale-up penalty [2, 7, 41]. This was calculated as follows: the SMD of the larger-scale trial divided by the SMD of the preliminary study and multiplied by 100. A value of 100% indicated identical SMDs in both the preliminary and larger-scale trial. A value of 50% indicated that the larger-scale trial was half as effective as the preliminary study; a value above 100% indicated that the larger-scale trial was more effective than the preliminary, whereas a negative value indicated that the direction of the effect in the larger-scale trial was opposite of the preliminary. Scale-up penalty is used in compliment to change in SMD because it provides a relative measure of change in effect.

Results

A total of 222 effects were extracted from 69 study pairs (k = 138 studies; see Fig. 2) [28]. For 44 pairs, there were two effects reporting main outcomes; for 8 pairs, there were three or more effects; 17 pairs had four or more effects. Publications ranged in date from 1990 to 2022 with the majority published between 2000 and 2022 (95%). References to the 138 studies are provided in supplementary material.

Fig. 2.

Fig. 2

Open in a new tab

Modified PRISMA flow diagram of systematic literature search. Asterisks indicate that k represents a single systematic review or meta-analysis. Dagger indicates that k represents a single study

Outcomes by hypothesis

Hypothesis 1. Similar to interventions targeting obesogenic behaviors, intervention studies in health promotion interventions targeting tobacco use, alcohol use, interpersonal violence, and behaviors related to sexually transmitted infections will contain changes to key study features between preliminary studies and subsequent larger trials.

Across 69 study pairs, 16% (n = 11 pairs) were coded as no presence of bias, 46% (n = 32 pairs) contained at least 1 bias, and 36% (n = 25 pairs) contained two or more biases. Of the seven biases studied, all seven were present in interventions addressing tobacco use, six were present in STI, six were present in interventions addressing alcohol use, and four were present in interventions addressing interpersonal violence (Table 2). The most common biases were implementation support bias (38%, n = 26 pairs), delivery agent bias (20%, n = 14 pairs), followed by duration bias (13%, n = 9 pairs) and intensity bias (12%, n = 8 pairs). Setting bias and audience bias were carried forward in 12% (n = 8 pairs) and 7% (n = 5 pairs) respectively (i.e., a conducive setting or audience was utilized in both preliminary and larger trial) but were never coded as being present in the preliminary study only.

Table 2.

Prevalence of risk of generalizability biases by field

Tobacco use Sexually transmitted infections Alcohol use Interpersonal violence
Risk of generalizability biases n = 35 pairs
N (% of pairs)		 n = 14 pairs
N (% of pairs)		 n = 12 pairs
N (% of pairs)		 n = 8 pairs
N (% of pairs)
Delivery agent bias Not present 25 (71%) 12 (86%) 11 (92%) 7 (88%)
Present in pilot 6 (17%) 0 - 0 - 0 -
Present in both 4 (11%) 2 (14%) 1 (8%) 1 (13%)
Implementation support bias Not present 25 (71%) 9 (64%) 6 (50%) 2 (25%)
Present in pilot 1 (3%) 1 (7%) 0 - 6 (75%)
Present in both 9 (26%) 4 (29%) 6 (50%) 0 -
Setting bias Not present 32 (91%) 14 (100%) 8 (67%) 7 (88%)
Present in pilot 0 - 0 - 0 - 0 -
Present in both 3 (9%) 0 - 4 (33%) 1 (13%)
Target audience Not present 32 (91%) 13 (93%) 11 (92%) 8 (100%)
Bias Present in pilot 2 (6%) 0 - 0 - 0 -
Present 2 (6%) 1 (7%) 1 (8%) 0 -
Intensity bias Not present 31 (89%) 11 (79%) 12 (100%) 7 (88%)
Present in pilot 3 (9%) 3 (21%) 0 - 1 (13%)
Intervention duration bias Not present 33 (94%) 10 (71%) 9 (75%) 8 (100%)
Present in pilot 2 (6%) 4 (29%) 3 (25%) 0 -
Measurement bias Not present 28 (80%) 13 (93%) 12 (100%) 8 (100%)
Present in pilot 7 (20%) 1 (7%) 0 - 0 -
Open in a new tab

Note: This table is intended to provide information about the presence of RBGs across different disciplines of health behavior research. For those reasons, it reports all study pairs and effects, including pairs where studies contained no eligible effects or single-group, post-only data and therefor do not contribute data to models estimating ΔSMD. Percents are calculated as (n/N) indicating the proportion of pairs within a topic that contain a specified RGB

Hypothesis 2. Changes to key study features between preliminary studies and subsequent larger trials will be associated with a larger scale-up penalty relative to studies that do not contain changes to key study features (i.e., study pairs that contain RGBs will be less effective at-scale).

In total, 47 study pairs producing 154 unique effects were eligible for inclusion in the meta-regression analyses and are represented in Table 3. Twelve study pairs (29 effects) were excluded from analyses because the studies did not report usable data in the either the preliminary study or larger trial and could not form a pair of effects (e.g., reported only feasibility data with no outcome data, displayed results in graphs but did not provide numeric values). Ten study pairs (25 effects) provided single-group, post-only data and were not included in the analysis because proportions do not include a measure of variance and cannot be combined with a meta-regression utilizing SMD [42].

Table 3.

Scale-up penalty and change in standardized mean difference (SMD) of the presence, absence or carry forward of bias from preliminary study to larger trial

Bias Scale-up penalty Change in SMD 95 CI Effects (n) Studies (n) Study pairs (n)
Delivery agent* None 30%  − 0.29 (− 0.43, − 0.15) 124 70 35
Present in pilot 18%  − 0.45 (− 1.12, 0.22) 11 10 5
Present in both 92%  − 0.03 (− 0.43, 0.38) 19 12 6
Implementation support None 48%  − 0.17 (− 0.33, − 0.01) 72 50 25
Present in pilot 41%  − 0.32 (− 2.31, 1.67) 6 4 2
Present in both 25%  − 0.40 (− 0.61, − 0.19) 76 38 19
Setting None 35%  − 0.29 (− 0.43, − 0.15) 134 78 39
Present in pilot Not observed
Present in both 30%  − 0.24 (− 0.62, 0.15) 20 14 7
Audience bias None 33%  − 0.27 (− 0.40, − 0.15) 142 86 43
Present in pilot Not observed
Present 27%  − 0.63 (− 1.34, 0.08) 12 6 3
Intensity None 33%  − 0.28 (− 0.41, − 0.15) 140 84 42
Present 47%  − 0.27 (− 1.03, 0.49) 14 8 4
Duration* None 36%  − 0.30 (− 0.44, − 0.16) 138 80 40
Present 10%  − 0.24 (− 0.51, 0.03) 16 12 6
Measurement* None 38%  − 0.26 (− 0.39, − 0.14) 147 88 44
Present  − 24%  − 0.69 (− 1.77, 0.38) 7 4 2
Open in a new tab

Notes: Intervention duration, intensity, and measurement biases describe differences between smaller- and larger-scale studies, while the other biases describe the presence of conducive factors. SMD indicates standardized mean deviation and represents the difference in the SMD between the preliminary studies and subsequent larger trials

*Follows hypothesized pattern for scale-up penalty such that changes to key study features between preliminary studies and subsequent larger trials will reduce the proportion of the effect maintained at-scale (i.e., study pairs that contain bias will be less effective in the larger trial). Scale-up penalty is calculated as the larger trial SMD divided by the preliminary study SMD multiplied by 100%. The larger the percentage, the larger the proportion of the effect that is maintained at-scale. A scale-up penalty of 50% is read as “the larger trial was half as effective as the preliminary study”

Overall, for study pairs where no bias was present, the effect size decreased by an average − 0.27 SMD (range − 0.30 to − 0.17). For pairs where a study feature was present in the preliminary study but absent in the larger trial, the effect size decreased by an average − 0.39 (range − 0.69 to − 0.24). For study pairs where the feature was coded as present in the preliminary study and in the larger trial (i.e., carried forward), the effect size decreased by an average of − 0.32 SMD (range − 0.63 to − 0.03). Exploratory post hoc analyses also examined whether the scale-up penalty varied by publication year, but no evidence of a temporal trend was observed.

In an exploratory post hoc analysis including studies published 2000 or later, neither the publication year of the preliminary study nor an interaction between publication year and scaling were statistically significantly associated with scale-up penalty.

Delivery agent bias, measurement bias, and duration bias matched the hypothesized pattern of scale-up penalty observed in previous literature [19, 30]. Studies that were not coded as containing delivery agent bias, measurement bias, or duration bias were 30%, 38%, and 36% as effective (e.g., 62–70% reduction in effectiveness), respectively, in the larger trials as they were in preliminary studies. Conversely, studies containing agent delivery bias in the preliminary study but not in the larger trial were 18% as effective (i.e., 82% reduction in effectiveness); those containing measurement bias had a scale-up penalty of − 34% indicating that the results of the larger trial were opposite of the expected effect (i.e., the outcome worsened in the larger trial); and those containing duration bias were 10% as effective in the larger trial (i.e., 90% reduction in effectiveness). Studies that carried the delivery agent forward to the larger trial maintained 92% of their effect.

Implementation bias and intervention intensity bias did not match the hypothesized pattern. For implementation support bias, studies that were coded as not containing the bias (i.e., did not provide implementation support in either the preliminary study or larger trial) were 48% as effective in the larger trials and studies that contained implementation support in the preliminary study but not in the larger trial were 41% as effective. However, studies where implementation support was provided in both the preliminary study and the larger trial (i.e., bias carried forward) had the largest scale-up penalty, maintaining only 25% of their effect. For intervention intensity bias, study pairs with differing intervention intensity between their preliminary study and larger trial (i.e., contained intervention intensity bias) were 47% as effective in the larger trial, while those that maintained the same intervention intensity (i.e., did not contain intensity bias) had greater scale-up penalty, being 33% as effective in the larger trial.

For audience bias and setting bias, we were not able to test our hypothesis that the presence of audience bias inflated estimates of effectiveness because only one study pair containing audience bias reported usable data for the analysis and no pairs containing setting bias reported useable data. Studies with these biases were coded in the preliminary study only, and those that contained the bias in both preliminary and larger trials experienced similar amounts of scale-up penalty.

Discussion

In this review, we confirmed our first hypothesis that disciplines outside of obesity-related research change key features between their preliminary studies and larger trials. For our second hypothesis, three biases reflect the expected pattern (delivery agent, measurement, duration), two biases did not reflect the expected pattern (implementation support, intervention intensity), and two biases did not provide sufficient data for testing (audience, setting). Taken together, these findings provide initial evidence that these biases are not discipline-specific and changes to several key intervention features between the preliminary studies and larger trials are associated with increased scale-up penalty (i.e., reduced effectiveness of the larger-scale intervention).

Comparison of changes across fields

Changes to the delivery agent, outcome measurement, and duration have demonstrated the most consistent detrimental patterns of increased scale-up penalty across all fields studied (i.e., obesity-related interventions, substance use disorder, tobacco use disorder, and interpersonal violence interventions). Preliminary studies, initially delivered by a highly trained members of the research team, are less effective when delivered by individuals with less training, expertise, and/or different professional responsibilities in later, larger trials (i.e., community health worker). Similarly, when studies utilize less precise outcome measures in their preliminary study, such as self-reported nicotine use or physical activity, followed by more objective measures in the larger trial, such as cotinine testing or accelerometry, they also experience a drop in effectiveness. Studies that utilize a shorter duration in their preliminary study also follow this pattern, wherein later studies run for a longer duration and show reduced effectiveness.

Across all five fields, studies that provided implementation support in the preliminary study and then removed or reduced it in the subsequent larger trial experienced increased scale-up penalty relative to studies that did not contain implementation support at either stage. However, when implementation support is provided in the preliminary study and carried forward into the larger trial, the impact is mixed. In interventions targeting adult and childhood obesity, the scale-up penalty was similar for studies that did not contain implementation support in either the preliminary or larger trial, and studies that maintained the same implementation support across both study iterations. However, for the four fields in the present review, studies that provided implementation support in both trials were less effective at-scale relative to studies that never provided implementation support.

Utilizing a more conducive setting in the preliminary study but a less conducive setting in the larger trial (i.e., setting bias) was not observed in the four fields reviewed here nor in the childhood obesity literature. It was also infrequently observed in the adult obesity literature though it had a notable, negative impact, with studies that utilized conducive settings in the preliminary study but less conducive setting in the subsequent trial experience a scale-up penalty of − 7%. Similarly, utilizing a conducive audience in the preliminary study followed by a different audience in the larger trial (i.e., target audience bias) was observed infrequently in the disciplines reviewed here (2 study pairs) and the childhood obesity literature (3 study pairs), but was present in the adult obesity literature with a 26% scale-up penalty when removed from larger trials.

Across all fields, intensity bias was observed in one study pair in the childhood obesity literature, three study pairs in the adult obesity field with results supporting the hypothesized direction, and seven study pairs in the present review (four providing data) with results not supporting the hypothesized direction.

Possible rationales for changes to key study features

Given that early-stage research typically requires adaptability, cost-effectiveness, and speed, it is logical that we commonly see these features in preliminary studies. The lead investigator of a study may serve as the delivery agent in the first iteration of an intervention because they have the expertise, experience, and long-range vision to understand and implement real-time changes as needed to the intervention, protocol, curriculum, and/or data collection processes. Preliminary studies may be shorter by a function of the budget, the funding source, or academic schedule (i.e., internal grant, dissertation). If an intervention has never been tested before, a duration of 12 to 16 weeks could be sufficient to tell if intervention warrants further investment, therein accomplishing the preliminary study’s initial objective. Similarly, self-reported measures are often logistically easier, less expensive, are less invasive, and therefore less regulated (i.e., do not require biosafety approval or may have less stringent IRB review) than their objective counter parts which may require participants to wear devices or provide biospecimens. Therefore, it makes sense to use less precise measures when the budget is smaller, the timeline is shorter, and the viability of the intervention is unknown. Nonetheless, though the presence of these features in a preliminary study result from sound and practical reasoning, this body of work demonstrates that transitioning from a preliminary study directly into a larger trial, without additional developmental testing, leads to increased scale-up penalty and poorer outcomes.

These findings highlight that the data produced by the initial preliminary study is not sufficient to demonstrate scalability and does not provide sufficient evidence to evaluate whether the intervention warrants testing in a larger trial. If, after the initial test of a intervention, modifications to key features of the preliminary study are proposed (e.g., who delivers the intervention, what measures are used, how much implementation support is provided), this should warrant additional small-scale testing prior to more substantial investment (e.g., R01-level funding). Researchers designing intervention studies and those reviewing grant proposals for larger-scale trials should be mindful of, account for, and thoughtfully addresses proposed changes to maximize effectiveness at-scale [43, 44].

It should be noted that aspects of the research enterprise may make it challenging to iterative testing and refinement before scaling interventions. Translational paradigms for behavioral interventions promote strong internal validity and efficacy in earlier development stages, suggesting the testing of interventions under optimal conditions [13, 25] and leaving questions regarding external validity, generalizability, and effectiveness to be addressed in later stage development. In alignment with these frameworks, it is normal, acceptable, and possibly encouraged for researchers to deliver preliminary studies under more tightly controlled conditions—delivering the intervention themselves (i.e., delivery agent bias) and engage with participants more regularly to ensure intervention fidelity (i.e., implementation support bias). As studies transition to the later stages of translational frameworks, the emphasis shifts to external validity and focuses on testing the intervention in the general population or real-world setting.

Funding for preliminary studies often comes from internal sources (e.g., a university) or career development award (e.g., NIH K awards) with limited direct research funds [45, 46]. Principal investigators or graduate students may serve as the delivery agents during a preliminary study because of insufficient funds to hire others to deliver the intervention, therein introducing delivery agent bias. Budgets can also dictate what outcome measures are affordable (e.g., self-report), how participants are recruited (e.g., limited or no incentives), and where an intervention is delivered (e.g., on university campuses). However, while tight budgets may seem like a clear reason for the difference in key features between preliminary studies and larger trials, in interventions targeting youth obesity, preliminary studies with both small (e.g., doctoral student award) and large (e.g., NIH funded R21 or R01) budgets contained similar rates of bias [19], indicating that funding alone is not responsible for changes to key features.

Timelines may also contribute to the inclusion of bias in preliminary studies. Across disciplines, myriad factors work together to generate circumstances where project teams have limited opportunities to retest and refine an intervention before using a single preliminary study to support a grant application for a larger trial. For example, an investigator may need to procure sizeable (e.g., NIH R01) external funding to secure tenure or support their salary [46]. It may take 2–3 years to plan, conduct, and analyze a preliminary study with a further 2–3 years to apply and receive funding for a larger trial [47]. This timeline leaves little room for additional smaller-scale testing of an intervention to refine issues identified in prior tests, which would then allow for intervention refinement [13].

Limitations

Our primary limitation was that despite a comprehensive literature search, we had a limited number of study pairs and effects that could be included in the analyses. In obesity-related interventions targeting children, a review of 1381 full-text articles produced 38 study pairs (2.7% return), and in obesity interventions targeting adults, a review of 1710 full-text articles produced 114 study pairs (6.6% return) while in the four fields reviewed in this study 3963 full-text articles produced 69 study pairs (1.7% return). These low rates of return are likely due to a combination of several factors; not all preliminary studies or larger-scale trials are published and therefore cannot be identified as linked pairs; some preliminary studies are abandoned prior to further testing, and some large trials may be conducted without preceding preliminary studies. Although we cannot determine which of these is most common, the subset of studies that appear as linked pairs likely represents a distinct group of investigations. Prior work from our group examining publication bias in preliminary behavioral interventions suggests that approximately half of preliminary studies remain unpublished, but those that are published in peer-reviewed journals do not differ systematically from those that remain unpublished [48]. These findings suggest that publication of preliminary studies may be driven less by study characteristics and more by investigator decisions to submit manuscripts, which may be influenced by available resources (e.g., time, competing priorities). Taken together, this implies that the studies included as linked preliminary–scale-up pairs may reflect characteristics of investigators or research environments rather than characteristics of the studies themselves.

Additionally, a limitation of this study is that the coding of study features and potential sources of bias relied on study reporting. Key intervention aspects, such as who delivers the interventions, are known to be inconsistently reported [49], and therefore, it is likely that both this review and our prior work underestimate the true prevalence of bias. Lastly, study pairs often had multiple biases though the limited sample size precluded us from examining the potential impact of multiple biases or interactions between biases.

Conclusion

This cross-disciplinary review provides evidence that changes to key features of preliminary studies occur across many fields of health behavior interventions. Changes to who delivers the intervention, how the outcomes are measured, and the implementation support provided are associated with larger-than-expected (i.e., inflated) scale-up penalty across multiple disciplines. The differences in outcomes across the disciplines, but similarities in developmental processes, suggest that the changes may be due, in part, to features of the intervention development process and research enterprise. Efforts to proactively identify and avoid the introduction of non-scalable intervention features in preliminary studies and the promotion of multiple iterative preliminary studies has the potential to reduce scale-up penalty maximize the effectiveness of health behavior interventions and promote the judicious use of the resources that fund their development, more effectively promoting public health.

Supplementary Information

Supplementary Material 1. (30.2KB, docx)
Supplementary Material 2. (14.4KB, docx)
Supplementary Material 3. (239KB, docx)

Acknowledgements

Not applicable.

Authors’ contributions

Credit statement:

LV—conceptualization, validation, methodology, formal analysis, investigation, resources, data curation, writing original draft, writing—review and editing, visualization, project administration, funding acquisition.

CDP and SB—conceptualization, data curation, methodology, writing—review and editing.

KR and SM—investigation, resources, data curation.

AM, BA, RGW, AO, DL, JPAI, RJ, GTM, JT, and XL—conceptualization, writing—review and editing.

MWB—conceptualization, validation, formal analysis, investigation, writing original draft, writing—review and editing, visualization, supervision, project administration, funding acquisition.

Funding

Research reported in this manuscript was supported by The National Heart, Lung, and Blood Institute of the National Institutes of Health under award number R01HL149141 (Beets), F31HL158016 (von Klinggraeff), F32HL154530 (Burkart), and by the Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under award number P20GM130420 for the Research Center for Child Well-Being. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request. Preliminary data from this study were presented at the 43rd Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine in Baltimore, MD, April 6–9th, 2022 (only outcomes pertaining to tobacco and sexually transmitted diseases were presented), as well as the 45th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine in Philadelphia, PA, March 13–16th, 2024, including behaviors of alcohol use disorder and interpersonal violence. Preliminary data was analyzed in STATA using two-level meta-regression models.

Declarations

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  • 1.Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.McCrabb S, Lane C, Hall A, Milat A, Bauman A, Sutherland R, et al. Scaling-up evidence-based obesity interventions: A systematic review assessing intervention adaptations and effectiveness and quantifying the scale-up penalty. Obes Rev. 2019;20(7):964–82. [DOI] [PubMed] [Google Scholar]
  • 3.Green LW. Making research relevant: If it is an evidence-based practice, where’s the practice-based evidence? Fam Pract. 2008;25(Suppl 1):i20–4. [DOI] [PubMed] [Google Scholar]
  • 4.Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Milat AJ, King L, Bauman AE, Redman S. The concept of scalability: Increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. 2013;28(3):285–98. [DOI] [PubMed] [Google Scholar]
  • 6.McKay H, Nettlefold L, Bauman A, Hoy C, Gray SM, Lau E, et al. Implementation of a co-designed physical activity program for older adults: positive impact when delivered at scale. BMC Public Health. 2018;18(1):1289. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Lane C, McCrabb S, Nathan N, Naylor P-J, Bauman A, Milat A, et al. How effective are physical activity interventions when they are scaled-up: a systematic review. Int J Behav Nutr Phys Act. 2021;18(1):16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Zullig LL, Deschodt M, Liska J, Bosworth HB, De Geest S. Moving from the trial to the real world: Improving medication adherence using insights of implementation science. Annu Rev Pharmacol Toxicol. 2019;59:423–45. [DOI] [PubMed] [Google Scholar]
  • 9.Indig D, Lee K, Grunseit A, Milat A, Bauman A. Pathways for scaling up public health interventions. BMC Public Health. 2017;18(1):68. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Flay BR. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med. 1986;15(5):451–74. [DOI] [PubMed] [Google Scholar]
  • 11.Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: getting started and moving on from stage I. Clin Psychol Sci Pract. 2001;8(2):133–42. [Google Scholar]
  • 12.Fehlberg Z, Stark Z, Klaic M, Best S. Blurring the lines: an empirical examination of the interrelationships among acceptability, appropriateness, and feasibility. Implementation Science Communications. 2024;5(1):139. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, et al. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.NCCIH [Internet]. Pilot Studies: Common Uses and Misuses. Available from: https://www.nccih.nih.gov/grants/pilot-studies-common-uses-and-misuses.
  • 15.Kistin C, Silverstein M. Pilot studies: a critical but potentially misused component of interventional research. JAMA. 2015;314(15):1561–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sim J. Should treatment effects be estimated in pilot and feasibility studies? Pilot and Feasibility Studies. 2019;5(1):107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.McGinty B, Becker W, Alegria M, Chin-Purcell L, Glass J, Knudsen H, et al. Integrating implementation outcomes into effectiveness studies: Center for Dissemination and Implementation at Stanford (C-DIAS) and HEAL Data2Action Research Adoption Support Center; 2024 [Available from: www.hd2arasc.org/implementation-guides-and-measures.
  • 19.Beets MW, Weaver RG, Ioannidis JPA, Geraci M, Brazendale K, Decker L, et al. Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: a systematic review and meta-analysis. Int J Behav Nutr Phys Act. 2020;17(1):19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Beets MW, von Klinggraeff L, Burkart S, Jones A, Ioannidis JPA, Weaver RG, et al. Impact of risk of generalizability biases in adult obesity interventions: a meta-epidemiological review and meta-analysis. Obes Rev. 2022;23(2):e13369. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Fitzgibbon ML, Stolley MR, Schiffer L, Van Horn L, KauferChristoffel K, Dyer A. Two-year follow-up results for Hip-Hop to Health Jr.: a randomized controlled trial for overweight prevention in preschool minority children. J Pediatr. 2005;146(5):618–25. [DOI] [PubMed] [Google Scholar]
  • 23.Kong A, Buscemi J, Stolley MR, Schiffer LA, Kim Y, Braunschweig CL, et al. Hip-Hop to Health Jr. randomized effectiveness trial: 1-year follow-up results. Am J Prev Med. 2016;50(2):136–44. [DOI] [PMC free article] [PubMed]
  • 24.Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science: unifying the discipline to improve the public health. Clin Psychol Sci. 2014;2(1):22–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Nielsen L, Riddle M, King JW, Aklin WM, Chen W, Clark D, et al. The NIH Science of Behavior Change Program: transforming the science through a focus on mechanisms of change. Behav Res Ther. 2018;101:3–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Greenwald P, Cullen JW. The scientific approach to cancer control. CA Cancer J Clin. 1984;34(6):328–32. [DOI] [PubMed] [Google Scholar]
  • 27.Cochrane handbook for systematic reviews of interventions. 2nd Edition ed: John Wiley & Son; 2019. Ed000142 p.
  • 28.Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of Internal Medicine. 2018;169(7):467–73. [DOI] [PubMed]
  • 30.Beets MW, von Klinggraeff L, Burkart S, Jones A, Ioannidis JPA, Weaver RG, et al. Impact of risk of generalizability biases in adult obesity interventions: A meta‐epidemiological review and meta‐analysis. Obes Rev. 2022;23(2):e13369. 10.1111/obr.13369. PubMed PMID: 34779122; PubMed Central PMCID: PMC8755584. [DOI] [PMC free article] [PubMed]
  • 31.Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. [DOI] [PubMed] [Google Scholar]
  • 32.Araújo-Soares V, Hankonen N, Presseau J, Rodrigues A, Sniehotta FF. Developing behavior change interventions for self-management in chronic illness: an integrative overview. Eur Psychol. 2019;24(1):7–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Sallis JF, Owen N, Fisher E. Ecological models of health behavior. Health Behav. 2015;5:43–64. [Google Scholar]
  • 34.Pfledderer CD, von Klinggraeff L, Burkart S, da Silva Bandeira A, Lubans DR, Jago R, et al. Consolidated guidance for behavioral intervention pilot and feasibility studies. Pilot Feasibility Stud. 2024;10(1):57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS ONE. 2016;11(3):e0150205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Mbuagbaw L, Kosa SD, Lawson DO, Stalteri R, Olaiya OR, Alotaibi A, et al. The reporting of progression criteria in protocols of pilot trials designed to assess the feasibility of main trials is insufficient: a meta-epidemiological study. Pilot and Feasibility Studies. 2019;5(1):120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020;9(7):497–508. [DOI] [PubMed] [Google Scholar]
  • 38.Stevens J, Taber DR, Murray DM, Ward DS. Advances and controversies in the design of obesity prevention trials. Obesity (Silver Spring). 2007;15(9):2163–70. [DOI] [PubMed] [Google Scholar]
  • 39.Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;3(36):1–48. [Google Scholar]
  • 40.Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections. Available from: https://jepusto.github.io/clubSandwich/.
  • 41.Yohros A, Welsh BC. Understanding and quantifying the scale-up penalty: a systematic review of early developmental preventive interventions with criminological outcomes. J Dev Life-Course Criminol. 2019;5(4):481–97. [Google Scholar]
  • 42.Barendregt JJ, Doi SA, Lee YY, Norman RE, Vos T. Meta-analysis of prevalence. J Epidemiol Community Health. 2013;67(11):974–8. [DOI] [PubMed] [Google Scholar]
  • 43.Perez Jolles M, Lengnick-Hall R, Mittman BS. Core Functions and Forms of Complex Health Interventions: a Patient-Centered Medical Home Illustration. J Gen Intern Med. 2019;34(6):1032–8. 10.1007/s11606-018-4818-7. [DOI] [PMC free article] [PubMed]
  • 44.Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.von Klinggraeff L, Burkart S, Pfledderer CD, Saba Nishat MN, Armstrong B, Weaver RG, et al. Scientists’ perception of pilot study quality was influenced by statistical significance and study design. J Clin Epidemiol. 2023;159:70–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.von Klinggraeff L, Burkart S, Pfledderer CD, McLain A, Armstrong B, Weaver RG, Beets MW. Balancing best practice and reality in behavioral intervention development: A survey of principal investigators funded by the National Institutes of Health. Transl Behav Med. 2024;14(5):273–84. 10.1093/tbm/ibae009. PMID: 38493078; PMCID: PMC11056885. [DOI] [PMC free article] [PubMed]
  • 47.Chung KC, Shauver MJ. Fundamental principles of writing a successful grant proposal. J Hand Surg Am. 2008;33(4):566–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.von Klinggraeff L, Ramey K, Pfledderer CD, Burkart S, Armstrong B, Weaver RG, et al. The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences. Pilot Feasibility Stud. 2023;9(1):115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Rauh SL, Turner D, Jellison S, Allison DB, Fugate C, Foote G, et al. Completeness of intervention reporting of clinical trials published in highly ranked obesity journals. Obesity (Silver Spring). 2021;29(2):285–93. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Material 1. (30.2KB, docx)
Supplementary Material 2. (14.4KB, docx)
Supplementary Material 3. (239KB, docx)

Data Availability Statement

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request. Preliminary data from this study were presented at the 43rd Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine in Baltimore, MD, April 6–9th, 2022 (only outcomes pertaining to tobacco and sexually transmitted diseases were presented), as well as the 45th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine in Philadelphia, PA, March 13–16th, 2024, including behaviors of alcohol use disorder and interpersonal violence. Preliminary data was analyzed in STATA using two-level meta-regression models.

Articles from Pilot and Feasibility Studies are provided here courtesy of BMC

RESOURCES