Table 1.

Common researcher responses to finding null or negative effects in prevention science trials

Response	Effect	Legitimacy
1. Don’t publish: Researchers may not report null or negative findings, either because results papers are never submitted or because they are but scientific journals are not interested in publishing them (and the researcher gives up trying to get results into the public domain).	This contributes to a skewed impression of “what works” because the studies do not get picked up in systematic reviews and meta-analyses; specifically, evidence of effectiveness is likely to be exaggerated.	Failure to submit a results paper for publication is not necessarily a deliberate act, rather it can occur through inertia (although when an author is involved in intervention design or dissemination, this distinction becomes blurred). Journal editors and reviewers tend not to say that a lack of effect is the reason for rejection, but null effects rarely constitute the “ground-breaking” findings that journals invariably aspire to publish.
2. Embark on fishing trips: Researchers may embark on “fishing trips” to find evidence of impact. Usually this entails conducting spurious analyses in search of ad hoc subgroup effects.	The chances of finding false-positive results from a single dataset increase as more hypotheses are tested, so this practice can produce misleading results.	Moderator analyses specified a priori in the trial protocol or statistical analysis plan can be suitable, even if they are exploratory and acknowledged to be underpowered. However, there is widespread agreement that it is inappropriate to conduct ad hoc or theoretically uninformed moderator analyses in an attempt to find a positive effect for a subgroup.
3. “Cherry pick” positive results: In the context of predominantly null or negative results, researchers may single out any positive result, however small or practically insignificant, and accord it unwarranted prominence in the reporting of findings.	This creates the appearance of effectiveness, especially if the findings are “spun” in the write-up (e.g., by referring to “positive effects” in the abstract and relegating information about the lack of effect to the body text).	Given the difficulty of publishing null or negative findings, this response is unsurprising but problematic when it concerns any of the following: a secondary outcome or mediator; an interim data collection point; an outcome with marginal statistical significance (or the level of statistical acceptability is changed to make it “significant”); or a tiny effect that is unlikely to be of practical or clinical significance (even if it is statistically significant).
4. Focus on methodological limitations: Researchers may criticize measures or other aspects of trial design or conduct, implying that the test was unfair, or insufficiently rigorous, and that it therefore failed to uncover the “true” effectiveness of the intervention.	This casts doubt on the veracity of the findings (even when that is unfair), leading the reader to conclude that the intervention is potentially effective or of unknown effectiveness.	It is reasonable to identify limitations to trial methodology when reporting results, and for interested observers to critique the methods. Limitations in design or conduct might present a valid explanation for the lack of positive effects, with important implications for the interpretation of findings and conduct of future research. However, it is disingenuous to identify such problems only once results are known.
5. Focus on poor implementation: Researchers may attribute the null effect to a failure to implement the intervention with acceptable fidelity. To support the argument, extra analyses may be conducted to show that effects are observed when fidelity is stronger.	This suggests that the intervention would be effective if delivered as intended.	There is strong evidence for a positive association between fidelity and outcomes, so exploring this relationship is reasonable. However, care is needed not to use fidelity as an excuse once outcome results are known. Moreover, fidelity x outcome analyses should compare “compliers” in the intervention arm with a comparable group in the control arm (those who would have complied had they been offered the intervention) to avoid spurious positive associations.
6. Focus on unsuitable context: Researchers may contend that aspects of the context (e.g., organizational, cultural, political, economic) were unsuitable and help to explain why the intervention did not “work”.	This argument can be deployed to suggest that the intervention is effective but that it did not work here; put crudely, the problem is with the context not the intervention.	Contextual arguments may be legitimate, and can help with thinking about how to improve intervention development and implementation planning. However, they should not be used to cast doubt unfairly on null or negative effect findings, particularly if contextual issues were not considered before the findings were known.
7. Forecast delayed or “sleeper” effects: On failing to find effects at planned time points, researchers may argue that the study timeframe was too short and that positive effects will only become apparent in the future.	This argument can be used to imply that the intervention is effective but it was too soon to observe those positive effects.	Forecasting delayed effects may be reasonable if there are good theoretical or empirical grounds to justify it (e.g., observed effects on proposed mediators). When these are not present, it can cast doubt on null results unfairly, particularly in the absence of the means or intention of investigating longer-term effects.