TABLE 2.
Odds for detecting a nominally statistically significant p value (p < 0.05 and p < 0.005) in a larger scale trial from the presence of a risk of generalizability bias
| Risk of generalizability bias | Odds ratio for p < 0.05 in larger scale trial | Odds ratio for p < 0.005 in larger scale trial | ||||||
|---|---|---|---|---|---|---|---|---|
| OR | (95% CI) | % Effects | OR | (95% CI) | % Effects | |||
| Delivery agent | Not present | Reference | 34% | Reference | 23% | |||
| Pilot only | 0.62 | (0.43, 0.90) | 24% | 0.60 | (0.39, 0.93) | 15% | ||
| Both | 1.39 | (0.97, 2.00) | 42% | 0.96 | (0.63, 1.47) | 22% | ||
| Implementation support | Not present | Reference | 34% | Reference | 22% | |||
| Pilot only | 0.81 | (0.42, 1.56) | 30% | 0.54 | (0.22, 1.30) | 13% | ||
| Both | 0.74 | (0.51, 1.08) | 30% | 1.03 | (0.67, 1.57) | 22% | ||
| Target audience | Not present | Reference | 32% | Reference | 20% | |||
| Pilot only | 0.72 | (0.47, 1.11) | 26% | 1.11 | (0.70, 1.75) | 21% | ||
| Both | 2.05 | (1.35, 3.12) | 49% | 2.51 | (1.58, 3.98) | 34% | ||
| Setting | Not present | Reference | 33% | Reference | 21% | |||
| Pilot only | 0.17 | (0.06, 0.47) | 8% | 0.22 | (0.07, 0.70) | 6% | ||
| Both | 2.01 | (1.29, 3.14) | 48% | 2.49 | (1.54, 4.03) | 35% | ||
| Intervention duration | Not present | Reference | 34% | Reference | 23% | |||
| Pilot only | 0.83 | (0.61, 1.12) | 32% | 0.60 | (0.42, | 0.84) | 17% | |
| Intervention intensity | Not present | Reference | 34% | Reference | 22% | |||
| Pilot only | 0.06 | (0.01, 0.46) | 3% | 0.12 | (0.02, 0.90) | 3% | ||
| Measurement | Not present | Reference | 35% | Reference | 22% | |||
| Pilot only | 0.86 | (0.52, 1.41) | 15% | 0.93 | (0.52, 1.65) | 20% | ||
| Directional conclusions | Not present | Reference | 34% | Reference | 23% | |||
| Pilot only | 0.27 | (0.13, 0.52) | 32% | 0.25 | (0.10, 0.60) | 8% | ||
Note: Bolded values 95% confidence intervals (CIs) do not cross 1.00. Reference group is pilot/feasibility study and larger scale trial pairs without a risk of generalizability bias present.