Assessment of generalisability in trials of health interventions: suggested framework and systematic review

Short abstract

Most evaluations of new treatments use highly selected populations, making it difficult to decide whether they would work elsewhere. Systematic evaluation and reporting of applicability is required

Randomised trials of health interventions generally describe outcomes among participants with little consideration of whether the effects can be generalised. However, generalisability cannot be assumed with either biomedical interventions or more complex social interventions.^w1 If their results are to be translatable into policy and practice decisions, trials must provide evidence about how relevant the interventions might be to other sites and populations.^{1 w2} Such information is particularly crucial for resource poor settings.²

Although CONSORT criteria for reporting randomised trials include assessment of generalisability,³ a framework for empirically assessing and reporting this is lacking. We consider the factors affecting generalisability using examples from HIV and sexual health, examine how a sample of trials looked at generalisability, and suggest how to improve evaluation.

Can the intervention be delivered elsewhere?

Several factors affect whether an intervention can be delivered and received in other sites. Firstly, an intervention must be feasible. Providers will vary in their capacity to implement an intervention,^w3 as will institutions in being suitable places for an intervention.^w4 The presence of local “champions” may influence feasibility in a particular site.⁴ Some interventions require the existence of other health services⁴—for example, services for treating sexually transmitted infections require microbiology laboratories to target the right patients. Interventions may also require adequacy in other sectors such as transport. Feasibility has a cost dimension: an unaffordable intervention lacks general feasibility.

Secondly, an intervention must achieve adequate coverage. This may depend on the overall comprehensiveness of health systems or on whether providers can reach people in other ways—for example, through outreach. Adequate coverage may be more difficult in some sites or sub-populations.

Finally, an intervention generally must be acceptable to be effective. Acceptability refers to participants' assessment of their experience of an intervention and will influence whether recipients adhere to treatment plans, act on health advice, or return for follow-up.⁴ For example, condom promotion has proved acceptable and subsequently effective in urban Tanzania but not in rural regions.^w5 Acceptability will vary between populations as it depends on cultural norms and can have economic dimensions. For example, HIV voluntary counselling and testing services that require clients to attend clinics twice (first for testing and then for results) may be acceptable in high income settings but not low income settings because transport or opportunity costs are too great.^w3

Factors relating to delivery of an intervention are best documented by embedding an evaluation of process in trials.⁵ The study collects quantitative and qualitative data on planning, delivery, and uptake and how context affects them.

Does the intervention meet recipients' needs?

To be effective an intervention must meet recipients' needs—that is, the recipients must have capacity to benefit from an intervention. Thus potential recipients of an intervention should have similar needs to those of the original study participants. Trial participants may be untypical of the general population even in the study site, let alone in other sites. Trials tend to under-represent certain groups, such as minority ethnic and low income groups, women, and older people, whose needs may differ from those of people included in trials.⁶ Trials should therefore describe the sociodemographic profile of participants and report the extent to which they are representative of the target population.

If the needs of future potential recipients differ from those of the study participants, interventions may not work in a new population or have to be adapted. For example, provision of antiretroviral drugs in low income countries, or to certain sub-populations may have to be accompanied by support to promote adherence in order to achieve similar outcomes to those achieved among trial participants.^w6

This is also true of public health interventions. The extent to which a factor contributes to the incidence of a particular disease, and therefore needs intervention, varies across populations. For example, treating ulcerative sexually transmitted infections may have a significant effect on HIV incidence in an HIV epidemic localised within high risk groups but not in a more generalised epidemic.^w1 Assessing whether an intervention has met recipients' needs, or will meet those of future recipients, requires investigators to be explicit about the causal pathways through which an intervention is expected to act and to measure relevant pathway variables.

Current assessment of generalisability

We reviewed whether trials of HIV prevention targeting homosexually active men explored generalisability or factors affecting this. We obtained and examined all available evaluation reports of eight interventions that a recent systematic review reported to have rigorously evaluated outcomes.⁷ Two reviewers independently assessed whether the studies had empirically examined local factors affecting feasibility, coverage, and acceptability; evaluated process; assessed needs; and assessed the potential generalisability of interventions.

Six of the eight trials had integral process evaluations,^8-14,¹⁶,¹⁸ but only three of these collected quantitative and qualitative data on the planning, delivery, and receipt of the intervention (table 1).⁹,¹²,¹⁸ Only one process evaluation stated that consideration of generalisability was an aim.¹⁰ Six trials gave some information about participants' ethnicity (usually the proportion described as white).⁸,⁹,^15-18 Seven trials provided data on educational level.⁸,⁹,¹²,^14-17 None commented on the extent to which study samples were representative of the populations being targeted.

Table 1.

Interventions and process evaluation in eight studies of HIV prevention

Outcome study	Additional process study	Intervention	Site	Any positive effects?	Methods of process evaluation	Stated rationale for process evaluation
Dilley et al (2002)8	None	Cognitive behavioural HIV prevention counselling	San Francisco HIV clinic	Yes	Adherence to treatment. Some sessions taped but data not reported	Assess consistency, completeness, and adherence to intervention guidelines
Elford et al (2001)9	Elford et al (2002),10	HIV peer education	London gyms	No	Survey of men's awareness, contact with and perceived usefulness of work.	Explore feasibility, practical constraints, transferability, and cultural adaptation
	Elford et al (2000)11				Interviews and group discussions with providers and stakeholders and documents about costs, planning, and delivery
Flowers et al (2002)12	Flowers et al (1999)13	HIV peer education, gay specific genitourinary medicine service, and telephone advice line	Gay venues and clinic, Glasgow	No (intention to treat); yes (treatment analysis)	Observation of intervention. Survey of men's awareness and contact with work and its acceptability. Provider diaries, interviews and group discussions, and documents about planning and delivery	None
Gold and Rosenthal (1998)14	None	Face to face HIV prevention discussion versus posters	Homes, Melbourne and Sydney	No	Survey of men's views on usefulness of interventions	None
Imrie et al (2001)15	None	Cognitive behavioural HIV prevention workshop	Sexual health clinic, London	No	None	None
Picciano et al (2001)16	None	HIV telephone counselling	US	Yes	Counsellor reports of contents and ratings of each session	None
Rosser et al (2002)17	None	HIV education seminars	US university	Yes	None	None
Shepherd et al (1997)18	Shepherd et al (1999)19	HIV peer education	UK homes and other informal sites	Yes	Group discussions and interviews with educators, including drop-outs, about intervention training, reach, acceptability, and delivery	To examine how peer education can be undertaken with gay men not yet involved in the gay community

Only those trials incorporating process evaluations identified contextual factors influencing the feasibility, coverage, and acceptability of their intervention (table 2). Elford et al, for example, reported that recruitment and retention of peer educators to provide HIV prevention in gyms was extremely difficult because of educators' low confidence.¹⁰

Table 2.

Discussion of contextual factors and generalisability in eight studies of HIV prevention

	Empirical examination of:
Outcome evaluation	Acceptability	Feasible delivery	Coverage	Local needs	Discussion of generalisability
Dilley et al (2002)8	No	No	No	No	Cost of intervention regarded as potential barrier to transfer
Elford et al (2001)9	Educators thought intervention period insufficient to develop rapport	Recruitment and retention of educators difficult due to time, interest, and confidence problems. Time needed for planning was more than expected	Educators found it difficult to make contact (and discuss sex) with participants. Possibly related to social norms in UK, big city, or gym context	No	Intervention informed by US work and authors question its transfer to UK because of different norms
Flowers et al (2002)12	Discussing sex in community sites went against local norms	Recruitment of popular peers difficult; educators found it difficult to discuss sex	No	No	Intervention informed by US work and authors question its transfer to UK because of different norms and needs
Gold and Rosenthal (1998)14	No	No	No	No	No
Imrie et al (2001)15	No	No	No	No	No
Picciano et al (2001)16	No	More feasible to address some issues with minority ethnic than white men	More drop-outs among young and less educated men	No	Argues intervention is generalisable to other sites and populations because it is client centred
Rosser et al (2002)17	No	No	No	No	Suggests intervention may be more effective in sites with higher rates of risk behaviour
Shepherd et al (1997)18	No	Prior links between educators and health promoters enabled recruitment. Educators reluctant to address factors other than knowledge because of norms about what constitutes education	No	Reported qualitative findings on sexual health needs from baseline interviews	Suggests prior rapport between health promoters and potential educators essential to recruitment. Also that educators need longer involvement to address factors other than knowledge

Only one study reported on needs (table 2).¹⁹ Although other studies reported baseline sexual behaviour⁸,⁹,¹²,^14-17 or sexual health related attitudes or knowledge⁸,¹²,¹⁶ of the target population or participants, the purpose was to check for baseline differences between intervention and comparison groups rather than to describe normative need.

Most of the studies speculated about the potential generalisability of their intervention to other sites but did not consider this empirically. Rosser et al, for example, wondered whether their intervention might prove more effective among populations with more risky sexual behaviour.¹⁷ The trials that examined contextual barriers and facilitators to delivering the intervention could make more considered assessments of generalisability. Two reports referred to sociological theory to hypothesise what contextual factors might have influenced the effect of the intervention in the study site compared with other sites.¹⁰,¹² However, these trials both reported on interventions previously reported as effective in other contexts^w7 that were largely ineffective in their own sites. Therefore, rather than consider the scope for transferring the interventions to new sites, they (reasonably) considered the contextual reasons for failure of transfer.

Systematic evaluation

To make informed decisions about whether they should implement interventions, providers require more information than simply whether interventions are effective in original study sites. They need information on context and needs. However, most of the studies we looked at did not empirically examine generalisability. Phase III trials should be judged not only in terms of the designs and methods they use to examine outcomes³ but also how they assess generalisability. To enable this trials should:

• Include process evaluations as integral elements⁵

• Develop evidence based theories about how intervention processes are influenced by context^w8 and how processes might differ if interventions are implemented in other sites^w9

• Report the extent to which their participants are representative of the population being targeted⁶

• Describe the prevalence of the needs being met by the intervention, informed by clear hypotheses about the intervention's mechanism.

We believe that these elements are essential to comply with the existing CONSORT requirement to report on “clinical characteristics” of participants if clinical is interpreted as meaning need for health intervention.

The most useful information on the potential for, as well as the barriers to, transfer of interventions comes from studies that compare an intervention in one site with similar interventions provided elsewhere, as in the study by Elford et al.¹⁰ Future phase III research might build on such work by setting out to examine interventions implemented across diverse contexts in multisite studies. These would examine differential effects by site and explore contextual determinants of success to generate hypotheses for future research and guidelines for the implementation of interventions outside trials.^w9 This approach is compatible with a phased approach to intervention trials. Assessing generalisability in phase III should inform choice of sites for phase IV replicability research.²⁰ However, such multi-site evaluations are unlikely unless funding for such work is increased.

Summary points

Few randomised trials assess the generalisability of their results

Such information is essential to decisions about adopting new interventions

Trials should include evaluations of the feasibility, coverage, and acceptability of interventions

They should also examine exactly for whom and what interventions are effective

Finally, systematic reviews should consider generalisability. Currently, many do not examine intervention process or context and do not comment on the potential for and limits to intervention effects being generalised to other settings and populations.^w10