TABLE 1.

Operational definition of risk of generalizability biases

Risk of generalizability bias		Example of bias
	What is the potential for difference(s) between …
Intervention intensity bias	… the number and length of contacts in the pilot study compared with the number and length of contacts in the larger scale trial of the intervention?	7 sessions in 7 weeks in pilot/feasibility study vs. 4 sessions over 12 weeks in larger scale trial. 24 contacts per week for 12 weeks in pilot/feasibility vs. 2 contacts per week for 12 weeks in larger scale trial
Implementation support bias	… the amount of support provided to implement the intervention in the pilot study compared with the amount of support provided in the larger scale trial?	Any adherence issues noted were immediately addressed in ongoing supervision. At the end of each session, the researcher debriefed with the interventionist to discuss reasons for the variation in approach and to maintain standardization, integrity of implementation, and reliability among interventionists.
Intervention delivery agent bias	… the level of expertise of the individual(s) who delivered the intervention in the pilot study compared with who delivered the intervention in the larger scale trial	All intervention sessions were led by the first author. The interventionists were highly trained doctoral students or a postdoc.
Target audience bias	… the demographics of those who received the intervention in the pilot study compared with those who received the intervention in the larger scale trial	Affluent and educated background, mostly White non‐Hispanic. Participants were predominately healthy and well‐educated.
Intervention duration bias	… the length of the intervention provided in the pilot study compared with the length of the intervention in larger scale trial?	8‐week intervention in pilot/feasibility study to 12‐month intervention in larger scale trial.
Setting bias	… the type of setting where the intervention was delivered in the pilot study compared with the setting in the larger scale trial	A convenience sample of physicians, in one primary care office practice agreed to participate. They were approached because of a personal relationship with one of the investigators who also was responsible for providing physician training in the counseling intervention. The study was conducted at a university health science center.
Measurement bias	… the measures employed in the pilot study compared with the measures used in the larger scale trial of the intervention for primary/secondary outcomes?	Use of objective measures in pilot to self‐report measures in larger scale trial.
Directional conclusions	Are the intervention effect(s) in the hypothesized direction for the pilot study compared with those in the larger scale trial?	Outcomes in the opposite direction (e.g., control group improved more so than treatment group).

Note: Based on definitions originally appearing in Beets et al. ¹¹