Table 1.
Reworded external validity items and extent of reporting by item (n = 39 studies) *
| Large extent | Some extent | Unclear | Not at all | ||
|---|---|---|---|---|---|
|
A Population: Representativeness of target population, setting & reach of intervention | |||||
| 1 |
Are data presented on variations in participation rate in improved housing interventions by a) setting b) delivery staff/organisations c) residents (for intervention among general target population not study area) |
0 |
0 |
0 |
39 |
| 2 |
Is the intended target audience for adoption clearly described |
11 |
18 |
8 |
2 |
| 3 |
Is the intended target setting for adoption clearly described? |
4 |
27 |
5 |
3 |
| 4 |
Is there analysis of the baseline socio-demographic and ‘condition tested’ (health status) of evaluation participants versus non-participants? (relating to evaluation population only) |
0 |
0 |
2 |
37 |
|
B Intervention: Implementation & adaptation | |||||
| 5 |
Are data presented on consistency of implementation of intervention & its different components? |
0 |
2 |
2 |
35 |
| 6 |
Are data presented on the level of training of experience required to deliver the programme or quality of implementation by different types of staff? |
0 |
1 |
1 |
37 |
| 7 |
Is information reported on whether/how the intervention is modified to individuals/households within the study? |
5 |
6 |
0 |
11 |
| 8 |
Are data presented on mediating factors or processes (mechanisms) through which the intervention had an impact? |
2 |
12 |
4 |
21 |
|
C Outcomes for decision making | |||||
| 9 |
Are the reported health (even if only one measure of health is comparable) outcomes comparable to wider policy/other studies? |
23 |
14 |
0 |
2 |
| 10 |
Have additional outcomes of potential adverse impacts been reported? e.g. socio-economic impacts |
4 |
21 |
1 |
13 |
| 11 |
Have authors demonstrated consideration of variation in reported health outcomes (key outcome of interest) by population sub-groups, or intervention setting/delivery staff? |
2 |
4 |
1 |
32 |
| 12 |
Is there sensitivity analysis of dose–response/threshold level required to observe health effect (effect on key outcome of interest not proxies)? |
3 |
4 |
1 |
31 |
| 13 |
Are data on costs presented? Are standard economic/accounting methods used? |
2 |
19 |
0 |
18 |
|
D Maintenance and institutionalisation of intervention | |||||
| 14 |
Are long term effects reported? (12 months or longer since exposure to the intervention) |
10 |
13 |
4 |
11 |
| 15 |
Are data reported on the sustainability (or reinvention or evolution) of programme implementation and intervention, at least 12 months after the formal evaluation? |
0 |
0 |
0 |
29 |
| 16 a |
Is the drop-out rate/attrition reported? |
19 (Yes) |
10 (N/A) |
||
| 16 b | Are data on attrition by baseline health status of dropouts reported and are analyses conducted of the representativeness of remaining sample at time of final follow-up (or main follow-up time point- as appropriate)? | 0 | 0 | 0 | 29 (10 N/A) |
(adapted from Green LW, Glasgow RE. Evaluating the Relevance, Generalization, and Applicability of Research: Issues in External Validation and Translation Methodology. Eval Health Prof 2006;29(1):126–153.)
* see Additional file 1 for full details of external validity assessment tool.