Enhancing the Net Benefits of Disseminating Efficacious Prevention Programs: A Note on Target Efficiency with Illustrative Examples

Abstract

We consider the implementation, in a non-research setting, of a new prevention program that has previously been evaluated in a randomized trial. When the target population for the implementation is heterogeneous, the overall net benefits of the implementation may differ substantially from those reported in the economic evaluation of the randomized trial, and from those that would be realized if the program were implemented within a selected subgroup of the target population. This note illustrates a simple and practical approach to targeting that can combine risk-factor results from the literature with the overall cost-benefit results from the program’s randomized trial to maximize the expected net benefit of implementing the program in a heterogeneous population.

Introduction

The ongoing development of methods for outcome measurement and valuation in mental health (Hargreaves et al. 1998; Rosenheck et al. 1998) is expanding the possibilities for application of cost-benefit analysis (CBA) for mental health prevention and treatment programs. The application of CBA methods to prevention/early intervention programs for young adults and children is particularly challenging, because of the long time horizons required for assessing outcomes, but a number of recent examples have appeared in the literature (Karoly et al. 1998; Greenwood et al. 1996, Aos et al. 2001; Ludwig and Phillips 2007). The general approach followed in these studies is to assess program effects in preventing (promoting) undesirable (desirable) long-term outcomes such as adult criminal activity (or high-school graduation) and apply “shadow-price” measures of dollar benefit values (Boardman et al. 2006, Chap. 15; Zavala et al. 2005) associated with these long-term outcomes.

A parallel development in the literature is the recognition of heterogeneity among program participants (or treatment subjects) as a factor that influences the effectiveness, and consequently the realized net benefits, of a program. In most cases, program costs and/or effectiveness can be expected to vary across individuals in the target population (Kravitz et al. 2004; Foster et al. 2006). In the presence of heterogeneity, the overall net benefit of an intervention will also depend on the selection of program subjects from the target population (Karoly et al. 1998; Greenwood et al. 1996).

In this note, we focus on the situation of implementing, in a non-research setting, a new prevention program that has previously been evaluated in one or more randomized trials. We assume that the target population in this non-research setting is heterogeneous and we illustrate a simple approach to combining information about participants’ heterogeneity with CBA results from the research evaluations of the randomized trials for this new program. Our examples illustrate how targeting (i.e., selection of a subgroup of individuals for inclusion in the intervention group) can enhance the expected net benefit of implementing the program in non-research settings.

Methods

Selecting Program Subjects from the Implementation Target Population

We consider the situation of an administrator faced with deciding (1) whether or not to implement a new program based on prior results from a randomized trial, and (2) how to target the program if it is replicated. We assume the population of implementation subjects is heterogeneous and (in our initial exposition) that the program impact is simply to decrease the probability that the program subjects experience a single undesirable future outcome (e.g., dropout from high school, incarceration).

The heterogeneity that is relevant for the administrator’s decision arises because each potential program participant’s personal characteristics may influence either the marginal program cost of including that person in the implementation and/or the expected impact of the program on the probability of an undesirable future outcome for that person. In the examples presented in this study, we consider a particularly simple but illustrative pattern of heterogeneity of expected impact that parallels the distinction made in the literature on prevention programs between exposed and unexposed populations. In this pattern, potential subjects can be divided conceptually into two groups: (1) the subjects who would have experienced the undesirable outcome in the absence of the program and (2) those who would not have experienced this outcome in the absence of the program. The program has zero effectiveness for the latter group, but is assumed to have some beneficial treatment effect on the former group. The administrator can not tell in advance into which of the aforementioned two groups a potential program participant falls, but has information on each potential participant’s level of risk for the future undesirable outcome if they are not included in the program (which will usually be available from the published risk-factor literature).

The basic idea of targeting is to select for inclusion in the program those potential subjects whose level of risk exceeds some threshold. Our proposed targeting strategy is an extension of the idea of targeting an intervention to a single group of “high risk” subjects to enhance net benefits or cost-effectiveness (Karoly et al. 1998; Foster et al. 2006). However, unlike earlier applications of targeting, which defined arbitrarily a threshold “high risk” level, we explicitly define a criterion function for net benefits, and identify the threshold level of risk that will maximize net benefits. Targeting based on an arbitrarily chosen threshold level of risk will in general yield lower expected net benefits than our proposed approach

An estimate of each potential subject’s risk level, which we will denote by p*_i for the ith subject, could be obtained by applying risk factor models reported in the epidemiology, psychology, economics or sociology literatures. The models most appropriate for this purpose would be those that use predictors of risk that are also available for each potential subject in the implementation, and are estimated from populations and circumstances similar to those of the implementation’s target population.

In addition to information on each individual’s risk for one or more adverse outcomes, application of the proposed targeting strategy also requires information on the program’s impact, costs, and benefits. Estimates of program impact and cost per included subject would be obtained from the economic evaluation results of the randomized trials for the new intervention program. The necessary information on program impact includes the rate of occurrence of the undesirable outcome in the control population (denoted by RC) and the reduction in the rate of this occurrence attributable to the intervention (denoted by RRC). This information, combined with an appropriate “shadow price” (dollar value) for undesirable outcomes prevented by the program (V), can be used to form an expression for the program’s net benefits in the randomized trial:

$${\text{NB}}_{\mathrm{T}}={\mathrm{N}}_{\mathrm{T}}\times \text{RRC}\times \mathrm{V}-{\mathrm{N}}_{\mathrm{T}}\times \mathrm{C}$$ (1)

where N_T is the number of treatment group subjects in the randomized trial and C is the cost per treatment group subject. For simplicity, we assume that C is the same for all subjects and does not vary with their characteristics.

An equivalent expression of net benefits that allows us to illustrate the role of heterogeneity is

$${\text{NB}}_{\mathrm{T}}=\{\text{RC}\times {\mathrm{N}}_{\mathrm{T}}\times \text{RRC}\times \mathrm{V}/\text{RC}\}-{\mathrm{N}}_{\mathrm{T}}\times \mathrm{C}$$ (2)

where RC × N_T can be viewed as the randomized trial estimate of the number of treatment group subjects who would have had the undesirable outcome in the absence of the program, and where RRC × V/RC is the intervention benefit (i.e., gross benefits in dollars) divided by that number of treatment group subjects. This formulation makes explicit the notion that program benefits only arise from including those subjects who would otherwise have had the undesirable outcome while program costs are incurred for including any subject.

The administrator undertaking the implementation of the program obviously does not know which of the potential program participants would otherwise have the undesirable outcome in the distant future, but can obtain an estimate of the risk for each potential subject, p*_i, from the risk factor literature. The administrator also has estimates, from the randomized trial, of the per subject gross benefit for each treated subject who would otherwise have the undesirable outcome, RRC × V/RC, and the program cost per treated subject, C. Therefore, the administrator can compute the expected net benefit of including the ith potential subject in the implementation as

$$\mathrm{E}({\text{NB}}_{\mathrm{i}})=\mathrm{p}{\ast }_{\mathrm{i}}\times (\text{RRC}\times \mathrm{V}/\text{RC})-\mathrm{C}$$ (3)

In contrast to the case of constant expected treatment effects, the net benefits expression above indicates that expected treatment effects are heterogeneous and that the size of the effect is proportional to the level of risk. Assuming that the expected benefit or cost for including any one potential subject in the implementation is independent of the characteristics of the other subjects included in the implementation, the administrator can maximize the expected net benefit of the implementation by including all potential subjects for whom E(NB_i) > 0, that is, those subjects for whom p*_i > [C/(RRC × V/RC)]. This selection strategy can be viewed as achieving target efficiency. That is, this strategy achieves a larger expected net benefit than that achieved by any other selection strategy, unless the threshold risk level of that other strategy happens to equal [C/(RRC × V/RC)]. Apart from concerns about uncertainty, this notion of efficiency (i.e., maximizing net benefits) parallels the general usage of the term in CBA.

We also note that in some CBAs of randomized trials, RRC and/or V may not be reported separately but a gross benefit figure will be provided instead. In that case, the gross benefit figure can be substituted for RRC × V in Eq. 3.

Results: Illustrative Examples of Targeting

The Programs to be Implemented

To illustrate this method, we apply figures for per subject cost (C) and gross benefit per treated subject who would otherwise have an undesirable outcome (B* = RRC × V/RC) adapted from several recent interventions directed at high-risk children. (For brevity B* is henceforth referred to as “adjusted gross benefit”.) Typically, a major purpose of these interventions is to prevent the development of violent and antisocial behaviors as these children grow into adulthood. An important element of these benefit calculations for such programs is the prevention of future arrests and criminality. For our illustration, dollar benefit figures for future arrests and incarceration were drawn from analysis of the Seattle Social Development Project (O’Donnell et al. 1995) as analyzed by Aos et al. (2001), and from the results of the Elmira Prenatal/Early Infancy project (Olds 1996; Olds et al. 1997) as analyzed by Karoly et al. (1998).

The Target Population for the Implementation

For purposes of our examples, we use a target population for the implementation that consists of cohorts 1 and 2 of the Johns Hopkins Prevention Intervention Research Center’s (JHU PIRC) Baltimore intervention trials. The cohorts were recruited in 1985 and 1986 from 43 first-grade classrooms in 19 elementary schools located in 5 socio-demographically distinct areas in eastern Baltimore City. The numbers of children in the two successive cohorts were 1,196 and 1,115; data on the 812 PIRC subjects who were not exposed to an intervention are used here. Some subjects were excluded because of missing data. For information on the characteristics of the children, the interventions, and the data content and collection processes, see the project web-site (http://www.jhsph.edu/prevention/Data/Cohort3/C3%20Methods%20and%20Measures, accessed on February 5, 2008). Descriptive statistics on some of their characteristics are shown in the first two blocks of Table 1 below.

Table 1.

Variable definitions and descriptive statistics (n = 812)

Name	Definition	Mean	SD
Socio-demographic variables
WHITE	=1 if white; else = 0	0.294	0.456
MALE	=1 if male; else = 0	0.489	0.500
CGLTHS	=1 if individual’s caregiver education level is less than high school	0.318	0.466
CGHS	=1 if individual’s caregiver education level is high school	0.415	0.493
MCGEDUCN	=1 if individual’s caregiver education level is unknown	0.031	0.173
EMPLDCG	=1 if individual’s caregiver employed	0.548	0.498
MEMPLDCG	=1 if individual’s caregiver employment status unknown	0.050	0.219
Teacher ratings
SCTAGb,c	Teacher rated aggressive disruptive behavior, grade 3 spring	2.040	1.095
SCTCPb,c	Teacher rated attention concentration problems, grade 3 spring	2.922	1.410
TOCGBd	Teacher’s global rating of how individual is progressing as a student, grade 3 spring	2.837	1.307
Outcome variables
PRISON	=1 if individual ever incarcerated	0.089	0.284
ARRESTa	=1 if individual arrested before age 16	0.151	0.358

^an = 704

^b(1 = Almost never … 6 = Almost always)

^c= Grade 4 score if missing, grade 4 = interpolated value if missing

^d(1 = excellent, 2 = good, 3 = fair, 4 = poor, 5 = probably failing, 6 = definitely failing)

The Risk-factor Models

While the literature provides many examples of studies of children’s risk factors for future arrest and for incarceration, and a thorough review would presumably yield bivariate or multiple-risk factor models that could be applied to our target population, for reasons of convenience we have simply used logistic regression analysis to estimate these risk factor models from our target population data. (This would not, of course, be feasible for the administrator considering implementation of the program in the real world since observable future outcome data on the population would not be available to her.) We use as observable risk factors the socio-demographic characteristics and teacher rating scores shown in Table 1. We use as our observed outcomes a 0–1 indicator for incarceration as an adult, and a 0–1 indicator of having been arrested while under the age of 16. Incarceration data are from administrative records for the State of Maryland Department of Corrections; at the time these data were collected, study subjects were approximately 26 years old. Juvenile arrest data are self-reported from a follow-up interview of study subjects at age 20. These data were only available for 704 respondents, so some of our illustrative calculations are limited to these subjects. As shown in the third block of Table 1, rates of undesirable outcomes were 0.089 for incarceration in the adult criminal justice system and 0.151 for having been arrested prior to age 16.

Predicted probabilities of arrests and incarcerations are estimated from logistic regressions. Results of these regressions are reported in Table 2 (for incarceration as an adult) and Table 3 (for an arrest before age 16). In both regressions, demographic characteristics and the teacher ratings of aggressiveness are clearly the strongest predictors of our undesirable outcomes.

Table 2.

Logistic regressions of risk factors on PRISON

Variable	Brief definition	Odds ratio	P > |z|	Odds ratio	P > |z|
		1	2	3	4
WHITE	=1 if white; else = 0	0.312	0.001	0.340	0.011
MALE	=1 if male; else = 0	14.579	< 0.000	11.854	< 0.001
CGLTHS	=1 if caregiver educ. level < high school	5.840	0.002	5.074	0.003
CGHS	=1 if caregiver educ. level = high school	4.405	0.007	3.061	0.035
MCGEDUCN	=1 if caregiver educ. level unknown	0.255	0.220	1.932	0.609
EMPLDCG	=1 if caregiver employed	0.828	0.543	.7080	0.310
MEMPLDCG	=1 if caregiver employment status unknown	15.001	0.003	13.250	0.003
SCTAG	Teacher rated aggressive disruptive behavior	1.606	0.001	1.934	< 0.001
SCTCP	Teacher rated attention concentration problems	0.999	0.995	1.042	0.843
TOCGB	Teacher’s global rating of student progress	1.128	0.351	0.984	0.912
N		812		704

Note: Results in column 1 are used in Example 1 in the text. Results from column 3 are used in Example 2

Table 3.

Logistic regression of risk factors on ARREST (n = 704)

Variable	Brief definition	Odds ratio	P ≥ |z|
WHITE	=1 if white; else = 0	0.828	0.500
MALE	=1 if male; else = 0	4.300	< 0.001
CGLTHS	=1 if caregiver educ. level < high school	1.468	0.256
CGHS	= 1 if caregiver educ. level = high school	1.161	0.643
MCGEDUCN	=1 if caregiver educ. level unknown	0.868	0.915
EMPLDCG	=1 caregiver employed	0.707	0.178
MEMPLDCG	=1 if caregiver employment status unknown	1.535	0.539
SCTAG	Teacher rated aggressive disruptive behavior	1.632	< 0.001
SCTCP	Teacher rated attention concentration problems	1.048	0.765
TOCGB	Teacher’s global rating of student progress	1.268	0.036

Note: Results are used in Example 2 in the text

Results for Example 1

We compute expected net benefit from including the ith subject in the implementation as [(p*_i × B*) − C]. In our first illustration, we use figures for B* and C calculated from the evaluation of the Seattle Social Development Project (Aos et al. 2001). To compare results with benefit measures that differ in breadth, we use two figures for B*:B*1 ($20,845), which is based only on criminal justice system tax dollars saved by preventing a child from having any adult incarcerations, and B*2 ($37,320) which includes taxpayer savings plus the estimated value of savings for potential crime victims. The cost per participant, C, is $4,355. (See the Appendix for details of the calculations.)

Illustrative calculation results are reported in Table 4. Using the B*1 gross benefit figure, expected net benefits are greatest ($466,755) when target subjects with p*_i ≥ 0.21 are included in the program. When a more comprehensive gross benefit figure (B*2) is used, expected net benefits are maximized ($1,303,370) by including subjects with p*_i ≥ 0.13, and the percent of the target population selected for inclusion rises from 13.9 to 23.2. Also note that the expected net benefit values for a universal program (i.e., setting p*_i > 0) is negative. Thus, in this illustration, the program only becomes justified on economic grounds when targeting is used.

Table 4.

Expected net benefits for alternative selection criteria: Example 1

Select if predicted probability ≥ to	Expected net benefit with alternative adjusted gross benefit values		Percent of target population selected
	B* = $20,845	B* = $37,320
0.00	−$2,035,420	−$849,220	100
0.05	$17,010	$1,098,420	38.4
0.10	$216,070	$1,166,400	28.1
0.13a	NA	$1,303,370	23.2
0.20	$458,045	$1,211,755	14.2
0.21b	$466.755	NA	13.9
0.30	$316,145	$807,695	8.7
0.40	$181,095	$443,255	4.3
0.50	$134,415	$298,265	2.1

^aTarget efficiency for B* = $20,845

^bTarget efficiency for B* = $37,320

Results for Example 2

Contributors to the CBA and CEA literature in mental health have often noted that there are multiple dimensions of program impact (e.g., Rosenheck et al. 1998; Hargreaves et al. 1998, Chap. 7; Sindelar et al. 2004). In particular, early intervention programs seek to prevent multiple undesirable outcomes in adolescence and adulthood. For example, the recent CEA analysis of the Fast Track conduct problems prevention program (Foster et al. 2006) used three different outcomes for describing program effectiveness: adult criminal careers prevented, adolescent crimes prevented, and incidents of interpersonal violence by adolescents prevented. Thus, it is useful to consider how the illustration of targeting presented above could be extended to account for multiple outcomes.

Because expected net benefit is merely the sum of the expected benefit from preventing each of the multiple outcomes, we can extend the method just demonstrated by incorporating separate risk factor models and separate gross benefit figures for each of the multiple outcomes. We illustrate this by considering two undesirable outcomes, adult incarceration and juvenile arrests. Let subscripts A and J denote these outcomes. Our expected net benefit from including the ith subject in the implementation becomes [(p_Ai × B*_A) + (p_Ji × B*_J) − C]. Corresponding predicted probabilities, estimated from the risk factor models for each outcome, are p*_Ai and p*_Ji.

For this example, we use estimates for B*_A, B*_J and C derived from cost and benefit figures in Karoly et al. (1998) for the Elmira PEIP. (See the Appendix for details.) The predicted probabilities for each target group subject (p*_Ai and p*_Ji) are based on the logistic regressions reported in Tables 2 and 3. Selecting subjects to achieve target efficiency now involves taking account of both predicted probabilities. A direct way to do this is to compute [(p*_Ai × B*_A) + (p*_Ji × B*_J) − C] for each subject and to select only those subjects for whom this figure is > 0.

Our calculations are based on the following values (derived in the Appendix): B*_A = $4,600, B*_J = $910 and C = $3,561. Results are reported in Table 5. In this illustration expected net benefits are maximized by selecting subjects who meet the following two inclusion criteria: p*_Ai ≥ 0.5468 and p*_Ji ≥ 0.6292. As in our previous example, the increase in expected net benefits when target efficiency is achieved is quite substantial. In this example, however, since the gross benefit figures are not large relative to C, target efficiency requires selecting only the small fraction of subjects who are identified as particularly high risk. We also compare our target efficiency results with the results obtained with a simpler selection rule that both p*_Ai and p*_Ji are greater than or equal to the same cutoff risk value, and we test various values of this cutoff risk. In this case the maximum expected net benefit is larger than the expected net benefit from the best of the simpler selection rules shown, though the size of the differential is small.

Table 5.

Expected net benefits for alternative selection criteria: Example 2

Select if arrest probability ≥ to	Select if incarceration probability ≥	Expected net benefit	Percent of target population selected
0.30	0.30	−$46,445	8.52
0.40	0.40	−$10,918	5.11
0.50	0.50	$1,822	3.27
0.50468a	0.6292a	$5,711	1.42
0.60	0.60	$3,888	1.85
0.70	0.70	$2,009	0.28

^aTarget efficiency selection criterion

Discussion

The foregoing examples illustrate a simple approach to targeting that can combine published information from the literature on risk factors models, and information on the risk—related characteristics of potential subjects in the implementation population, with the basic CBA results from a randomized program trial. Our examples illustrate that the gains in expected net program benefits for the implementation could be substantial.

The simplicity of the proposed approach does, however, depend on several assumptions that may need modification in some real-world applications. First, our assumption that the cost per participant, C, is constant regardless of subject characteristics or the number of participants in the program could need modification. Our approach could allow for fixed costs and variable (incremental) costs per treated subject. In that case, only two straightforward modifications to our procedure are required. One is to use the variable cost per subject figure for C in the targeting process, and the other is to subtract out fixed cost at the end of the targeting calculations to obtain an expected net benefit figure that includes these fixed costs. While this modification requires additional information on program cost, it seems likely that such information is available either from the randomized trial CBA results or from the preliminary budget forecasting which an administrator will presumably undertake in planning for the possible implementation of a program.

A more complicated relationship of scale to cost could arise from indivisibilities such as fixed sizes of classrooms needed for the intervention. This could also be accommodated in the same manner except that the costs for the indivisible inputs would be treated as fixed over the relevant range of program size and would increase as larger size necessitated adding more classrooms to the program. In this situation, budget plan data from the implementation site may be the best source of relevant cost information for the targeting process. (More detailed discussions about costing procedures are provided in Chatterji et al. (2001, 2004).

The assumption that the incremental (variable) cost of including a subject is independent of the subject’s characteristics could also be modified if relevant information is available. For example, if part of the cost of the program includes home visits and there is data to show that the expected number of home visits varies with the subject’s characteristics, this information can be used to compute an expected incremental cost for each potential subject that varies among subjects. In this example, actual data on the relationship of subject characteristics to the number of home visits under the program would presumably only be available from the randomized trial evaluation of the program itself or from other implementations of the program that have already occurred. Projection in the absence of such data would of course involve greater uncertainty.

The assumption that program effectiveness can be described by impacts on a small number of binary indicators could also be relaxed with additional information from the trial CBA on the benefit values attached to additional outcome measures (e.g., the benefit per violent crime prevented), and on the effectiveness of the program in producing these additional benefits (e.g., numbers of violent crimes prevented). Risk factor models for predicting these additional non-binary dimensions of undesirable outcomes in the absence of the prevention program would also be required. These could be estimated from the control population for the randomized trial.

Our illustrations also assume that there are no constraints on the subject selection process and that any risk factors can be used to guide selection. This assumption may need to be modified for several reasons. For interventions that are implemented in a school setting, selection may be constrained by potential subjects’ classroom assignments and schedules for other school activities. Accommodating these constraints into a selection process is straightforward conceptually but may complicate the computer algorithm for maximizing expected net benefits. Political or legal concerns may limit the use of gender, racial, or economic class affiliation in subject selection. This implies excluding proscribed characteristics from the X vector in the risk factors models used for predicting undesirable outcomes.

Our illustrations also do not explicitly account for uncertainty about the magnitudes of B* and C, or uncertainty relating to the risk estimates (the p*_I’s). Intuitively, greater uncertainty about any of these magnitudes increases the probability that a targeting strategy that maximizes expected net benefits might yield actual net benefits that are not higher than those that would have been obtained in the absence of targeting. In administrative applications, the most straightforward way to assess the importance of this uncertainty would be to apply upper-bound and lower-bound subjective estimates to all the uncertain magnitudes and examine the implications of these estimates for the optimal targeting strategy. This sort of best-case versus worst-case sensitivity analysis is frequently applied in the CBA literature, though more sophisticated methods of sensitivity analysis, often involving simulation modeling, and bootstrap procedures, are also recommended (Manning et al. 1996; Briggs et al. 2002; Boardman et al. 2006, Chap. 7). As a practical matter, the sophistication used in incorporating uncertainty will presumably depend on the analytic resources available to the administrator and on the statistical evidence (from the randomized trial and from the risk-factor literature) on the degree of uncertainty in the estimates for B*, C and the p*_I’s.

It is also worth noting that the degree of uncertainty about the appropriate value for B* will tend to be much greater than uncertainty about the value of C. As noted in our illustrations, the value of B* will vary considerably with the range of beneficial effects captured by the “shadow price” estimates used to compute B*. Comparisons of alternative shadow price values in the literature also suggest wide variations for the same effect (e.g., the averted cost to victims of one adult crime prevented) (Boardman et al. 2006, Chap. 15). Such variations in B* may reflect differences in CBA methodologies (e.g., human-capital versus willingness-to-pay approaches) as well as variations in discount rates applied to effects occurring in the distant future. Discount rate choice may also have an important influence on the relative sizes of the B*’s corresponding to different effects that occur at different points in the future; higher discount rates, for example, will tend to reduce the relative B* for adult crime prevention in comparison to juvenile arrest prevention. Finally, given the multiplicity of beneficial consequences from many intervention programs, especially those programs that reduce the number of future serious crimes, it seems likely that the set of effects for which specific B* “shadow price” values are applied will not include all of these beneficial consequences. Thus it may be prudent to regard the net benefit estimates used in our proposed method as biased downward (and perhaps even lower-bound estimates). This problem can, of course, be addressed via the choice of “best case” and “worst case” values to be used in a sensitivity analysis of the targeting results.

We have also ignored the possibility that program benefits for one subject may depend on the characteristics of other included subjects. Such interaction effects could arise from peer influences or from effects of overall treatment group characteristics on the intervention process. For example, classroom interventions may have differing effects when all participating subjects are very high risk versus the case where the intervention subjects are more diverse in their risk levels. This example suggests that concern for interaction effects will be greater for programs that involve group activities (e.g., classroom instruction) than for those that rely mainly on small-group, family, or individual interventions. In the former case, empirical evidence on the impact of group characteristics on individual subjects’ benefits would clearly be useful in the analysis of targeting strategies. Developing this information from a randomized trial is difficult, however, unless the trial is implemented in many sites with subject groups of varying characteristics.

More generally, the breadth or limits of the information available from randomized trials of the program will have important consequences for our targeting methodology in subsequent implementations. If these trials are sufficiently powered to estimate interactions between intervention effects (RRC in our earlier notation) and subject characteristics, we can drop the assumption that the intervention effect for every child is proportional to their risk of the undesirable outcome; instead Equation (3) becomes:

$$\mathrm{E}({\text{NB}}_{\mathrm{i}})=\mathrm{p}{\ast }_{\mathrm{i}}\times ({\text{RRC}}_{\mathrm{i}}\times \mathrm{V}/\text{RC})-\mathrm{C}$$ (4)

where RRC_i is the reduction in the rate of the undesirable occurrence attributable to the intervention for subjects with the characteristics of the ith child. (Once these interaction effects are allowed for, it is even conceivable that for some interventions the intervention effect may actually decline with the level of risk because, for example, children with very severe behavioral problems may be much more resistant to behavior changes that the program seeks to induce. In this case, optimal targeting might even suggest excluding those children with the most severe behavioral problems, which implies that alternative programs need to be explored for these children.)

Another advantage of trials that are powered to estimate intervention effects for subgroups or as interactions is that these trials can directly test our assumption that there is indeed heterogeneity in the impact of the program on the treatment subjects. Finally, if the randomized trial data includes results from a variety of different settings or locations with target populations of varying characteristics, we can assess the degree to which results from trials on a particular study population can in fact be assumed to generalize to the population in a non-research setting, an assumption on which our method critically depends. (For example, one can use regression analysis to estimate identical risk factor models from the control groups across the variety of settings of the randomized trials, and then assess the stability of the parameter estimates for these risk factor models. A finding of fairly stable estimates would suggest the results could be extrapolated in disseminating the program to other populations.)

The foregoing suggests that while widespread use of the targeting method we propose may require the development of additional information that will allow some of our simplifying assumptions to be relaxed, efforts in this direction are a reasonable response to the problem of subject heterogeneity. It is likely that such heterogeneity can account for major differences between the “efficacy” results of randomized program trials and the “effectiveness” results observed by administrators who replicate these programs in non-research settings. While this problem could be attacked by substantial additional research funding of randomized implementations for new programs in a variety of settings and populations, it is unlikely that this will in fact occur. More detailed analyses of randomized trials to assess costs and benefits for various subgroups of subjects defined by risk level could also be helpful and seems feasible, though substantial increases in sample sizes (and the cost of the trials) may be required to allow subjects to be partitioned into multiple groups (instead of just “high risk” versus “low risk”) and maintain adequate statistical power. As we noted in our second example above, a more discriminating approach to risk classification may be critical in deciding whether or not to replicate programs that are only “cost-beneficial” when applied to a small group of very high-risk subjects.

Conclusion

Several recent studies have stressed the importance of targeting for preventive interventions aimed at reducing adolescent and adult violence and criminality. Use of CBA/CEA results separately for “high-risk” versus “low-risk” groups provides an initial approach to targeting but does not allow one to find the best targeting strategy for implementation of the intervention program in non-research settings. As recent literature suggests, and our own examples illustrate, targeting can make a major difference in concluding that a new prevention program is or is not worth replicating, since the expected net benefits from disseminating the program may be increased substantially by targeting, and may be changed from negative to positive.

Our examples illustrate a relatively simple method for achieving target efficiency that can be applied when results from CBAs of demonstration projects and relevant information on risk-factor models are available in the literature. Application of this method requires a modest amount of information and does not require the conduct of additional and sophisticated statistical analyses. Thus, we believe it can be easily applied by administrators with only a modicum of expert assistance (mainly on interpreting and applying the results from the relevant CBA and risk factor literature and assessing their applicability to the site for the implementation). We also discuss possible strategies for relaxing some of the simplifying assumptions made in our examples.

Wider use of target-efficiency methods in program implementation decisions can be expected to result in wider dissemination of new interventions that have been shown to be effective, by allowing administrators to identify the most cost-beneficial strategies for targeted implementation. This may be especially important for new interventions that are costly and not judged to be cost-effective or cost-beneficial when applied to all treatment-group subjects in the initial randomized trial of the program.

Appendix

Sources of the Illustrative Estimates for Adjusted per Person Gross Benefits (B*) and Costs (C)

The information needed to compute the two different figures for B* and the figure for C used in the text for the Seattle Social Development Project were all obtained from p. 135 of Aos et al. (2001). The reported rate of persons with a felony arrest in the demonstration control group was 18.7 per cent. The reported present value, per program participant, of averted future criminal justice system costs to the taxpayer was $3,898. Dividing this figure by 0.187 yielded our taxpayer benefit estimate of B* = $20,845. The reported present value, per program participant, of averted crime victim costs was $3,064. Adding this to the taxpayer savings and dividing the result by 0.187 yielded a taxpayer + crime victim benefit estimate of B* = $37,320. The reported program cost per participant was $4,355.

The benefit and cost figures from the Elmira PEIP were reported in Karoly et al. (1998), pp. 132–35 and pertain to a program targeted at “lower-risk” families. The benefit figure for reductions in adult criminal careers is computed as the present value of this benefit per program participant ($1,012) divided by the fraction of the control group who became adult criminals (0.22), yielding a value of B*_A = $4,600. The present value of reductions in child (under age 16) arrest costs per program participant was reported as $131. Since the fraction of the control group who had such arrests was not reported, we used the corresponding figure for the PIRC control sample that responded to the young adult interview (N = 993), which was 0.144. These figures yielded a B*_J = $910 per control group persons who had a child arrest. The cost figure was computed as $3,561. This was based on the direct program cost ($6,083) net of several benefits to taxpayers from beneficial outcomes excluded from our simple analyses (reduced ER visit costs for children, increased maternal tax payments and reduced maternal welfare payments). This netting out from costs assumes that the netted out benefits accrue evenly to all program participants regardless of their risk levels for the undesirable outcomes (adult incarceration and adolescent arrest). The assumption is made purely for expositional purposes.