Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Dec 1.
Published in final edited form as: Am J Community Psychol. 2017 Nov 20;60(3-4):309–315. doi: 10.1002/ajcp.12200

Valuing Our Communities: Ethical Considerations for Economic Evaluation of Community-Based Prevention

Max Crowley 1,2, Damon Jones 1,2
PMCID: PMC5729092  NIHMSID: NIHMS906817  PMID: 29154476

Abstract

Restricted public budgets and increasing efforts to link the impact of community interventions to public savings have increased the use of economic evaluation. While this type of evaluation can be important for program planning, it also raises important ethical issues about how we value the time of local stakeholders who support community interventions. In particular, researchers have to navigate issues of scientific accuracy, institutional inequality and research utility in their pursuit of even basic cost estimates. We provide an example of how we confronted these issues when estimating the costs of a large-scale community-based intervention. Principles for valuing community members’ time and conducting economic evaluations of community programs are discussed.

Keywords: Ethics, Economic Evaluation, Community-based Prevention, Cost Analysis

The field of community psychology operates in a persistently tight funding climate that increasingly seeks to tie funding to outcomes (Crowley, 2014; Haskins & Margolis, 2015). As a result, there are growing incentives to understand how community investments translate into not only improved physical and mental health, but also public savings (Crowley & Jones, 2015; National Academies of Medicine, 2015). Such economic evaluations can allow community researchers to estimate the cost of implementing community interventions (Beatty, 2009; Belfield & Levin, 2013; Crowley, Hill, Kuklinski, & Jones, 2013; Vining & Weimer, 2010). These estimates have the potential to be highly beneficial for program planning and facilitate efforts to achieve sustainability (Barnett & Masse, 2007; Chinman et al., 2005; Crowley, Jones, Greenberg, Feinberg, & Spoth, 2012; Wandersman, 2000). Yet, this methodological approach to community research can also raise key ethical issues that must be carefully considered (National Academies of Medicine, 2015). In this work, we identify such considerations and provide an example of how to navigate them as they arise.

Economic Evaluation in Community Psychology

Economic evaluation refers to analytic approaches and tools for estimating the costs, benefits and return-on-investment of an intervention (Brouwer, Koopmanschap, & Rutten, 1997; Cohen, Neumann, & Weinstein, 2008; Crowley et al., 2013; Foster, Dodge, & Jones, 2003). Increasingly this methodology is being incorporated within community research to inform practice decisions (Belfield & Levin, 2013; National Academies of Medicine, 2015). Such evaluations have been helpful in articulating the value of community-based interventions in a wide array of domains including substance abuse, violence, and obesity (Kuklinski, Briney, Hawkins, & Catalano, 2012; Crowley, Jones, Coffman, & Greenberg, 2014). For instance, the Communities that Care (CTC) program builds local coalitions to conduct community needs assessments and implement evidence-based behavioral interventions (Hawkins, 2009). An economic evaluation of CTC found that for every dollar spent on this approach it had a societal savings of $5.30 from reduced substance abuse and delinquent behavior (Kuklinski et al., 2012). Specifically, these savings were the result of increased community health and reduced crime that stemmed from behavior change. Such findings have justified increased use of CTC and been recognized as key to facilitating greater investment in community health infrastructure (Haskins & Margolis, 2015).

While the use of economic evaluation is a valuable complement to other evaluation methods, it can at times be limited by certain ethical considerations (Crowley et al., 2013; Foster, Porter, Ayers, Kaplan, & Sandler, 2007). In particular, it can be difficult to place a dollar value (i.e., monetize) on all of the inputs (i.e., costs) and outputs (i.e., benefits) of community interventions (Haddix, Teutsch, & Corso, 2003). A number of obstacles can limit an evaluator’s ability to estimate the costs or benefits of a program (Zerbe, 2005). One of the most common in community-based research is how to value a person’s time (Zerbe, Davis, Garland, & Scott, 2010). At first glance this may seem straightforward. The common approach to valuing a person’s time is to base it upon how much they are paid for doing a job. In this manner, the labor market determines the ‘worth’ of their time (Nas, 1996). Yet, as most community practitioners know, substantial amounts of work is done to make community efforts successful that is not formally compensated (Feinberg, Bontempo, & Greenberg, 2008; Gruen et al., 2008). Specifically, community-based interventions routinely rely on volunteer time to operate intervention activities and achieve their goals (Scheirer & Dearing, 2011; Spoth, Guyll, Redmond, Greenberg, & Feinberg, 2011). There are multiple reasons for this reliance on volunteer time. Most often it is the lack of core resources to operate robust community efforts (Savaya & Spiro, 2011). Others include a recognition of the need for local buy-in around a community intervention (e.g., demonstrated through a willingness to volunteer) or conflicts-of-interest among community leaders that can only be navigated if their time is donated (Feinberg et al., 2008; Perkins et al., 2011).

Valuing our Communities’ Time: Ethical Considerations

The ethical challenge surrounding how to value volunteer time stems from the need to decide what should be used to calculate the worth of that donated time (Drummond, 2005; Zerbe et al., 2010). Specifically, it is the need to determine what is the dollar amount that should be multiplied by the number of hours an individual volunteered. At issue is whether to use a rate commensurate with the hourly rate the volunteer makes at their main source of employment or to value their time in some other way. Using the rate the volunteer makes at their main source of employment is known as ‘targeted wage estimation’ and often offers the most straightforward approach for valuing their time (Drummond, 2005; Haddix et al., 2003). Yet, it also raises important ethical issues. Estimates based upon a targeted wage approach embeds labor market inequalities within a cost estimate. For instance, the lower wages experienced by women and minority populations will become part of the community program’s cost estimate (see US Census Bureau, 2014). The ethical question arises around whether an implementer should pay individuals based on the task being done or on the personnel’s ‘worth’ as dictated by the labor market. For instance, using cost analyses based on targeted wage estimates to budget for a new implementation can reinforce such inequalities. The issue exists regardless of whether personnel are available who can serve on a volunteer basis, given the value of all labor must be determined for an accurate cost assessment. As many community programs are, by design, focused on issues of social equity, this can pose a difficult ethical dilemma around reinforcing the very inequities that community psychology seeks to overcome.

While the process of valuing time is a core element of economic evaluation, the ethical ramifications of doing so become particularly pronounced within community-based interventions. Volunteer time is such a crucial component of many community programs that it is sometimes taken as a given that such time will be relied upon. Yet, when faced with the task of putting a monetary value on that time, the field may need to confront the ethics of relying on such time with such little consideration—particularly when the tenuous nature of volunteering can jeopardize the sustainability of the intervention and the health of those we serve. Undervaluing the resources needed to successfully implement these interventions ultimately compounds this ethical dilemma, which can extend beyond an effort to determine ‘what people are worth’, challenging the field to balance what is common practice with how to ensure stable service delivery for those in need.

Alternative valuation approaches can use different rates in an attempt to address these equity concerns in a community intervention’s cost estimate (Levin & McEwan, 2000). For instance, considering whether the volunteers derive a benefit from their volunteering may indicate that a different rate should be used. Research on the economic value of leisure time has been extensively studied (Feather & Shaw, 1999; Larson, 1993). Psychological and economic studies have both found evidence that leisure time is often worth more to an individual then their time at work (Jara-Díaz, Munizaga, Greeven, Guerra, & Axhausen, 2008). In such cases, volunteer time might be valued at a higher rate. Yet, valuing volunteer time at a higher rate would increase the documented cost of the community effort. Further, it would make it difficult to translate cost estimates to new contexts. Alternatively, some argue that volunteer time should not be included in the cost of a community intervention at all, as that time was not paid for by project budgets and would not reflect more practical budgetary estimates that reflect true spending (see National Academies of Medicine, 2015). Even further, volunteer opportunities can often provide substantial intrinsic value—and some see that as a form of compensation. These scenarios compound the ethical challenge. For instance, if a community wanted to implement another community’s successful program, but lacked an ample pool of volunteers, they would need to hire additional staff to ensure the intervention could be successfully replicated with fidelity (Crowley et al., 2013). If the cost estimate did not include the cost of volunteer time at all, the effort would likely be severely underfunded and unsustainable. Alternatively, if the cost estimate was based on a higher rate for volunteer time then a provider may have over budgeted (potentially tying up precious community resources).

Any ethical dilemma ultimately places two moral imperatives in conflict. While perhaps an oversimplification of the philosophical concerns at work here, the decision of how to value volunteer time ultimately pits the moral imperative of equity (wage inequality) against the moral imperative of truth (what actual costs were). Yet, the growing need to accurately and fairly estimate community intervention costs necessitate that researchers find a way to conduct these evaluations. In the next section, we provide an example of how we handled this challenge in a community-based evaluation of the PROSPER prevention delivery and support system.

The PROSPER Prevention Delivery and Support System

The PROSPER delivery and support system cultivates local prevention teams to support the delivery of universal school- and family-based prevention programs (see Spoth, Greenberg, Bierman, & Redmond, 2004). It employs a strategic planning and community outreach process that links university-based prevention researchers with two established program delivery systems—the Cooperative Extension System at Land Grant Universities and the local public school system. The Extension System offers knowledge of the community and experience in disseminating educational programs. The public school system offers access to youth in the community and to educators working with students. Research on the PROSPER system has demonstrated significant reductions in youth substance abuse and delinquency sustained for over an eight-year period (Crowley et al., 2014; Spoth, Randall, Trudeau, Shin, & Redmond, 2008).

PROSPER starts with these existing resources and then cultivates partnerships among other key youth and family service providers in the community to form small, strategic teams. These teams are comprised of eight to ten members and led by county-based Extension personnel and co-led by a school district staff member. Team members include social- and health-service providers, school administrators, parents and youth from the community, and representatives from other community institutions: faith-based, parent groups, businesses, law enforcement, and the juvenile justice system (Feinberg, Chilenski, Greenberg, Spoth, & Redmond, 2007; Perkins et al., 2011). The Community Extension Agent devotes the equivalent of 25% of their full-time responsibilities to leading the Community Team (Crowley et al., 2012). The rest of the team volunteers their time.

These partnerships deliver school and family-based universal substance abuse prevention programs. In this project the PROSPER Prevention Teams implement a family and a school evidence-based program that they select from a menu of preventive interventions (Spoth et al., 2004). All programs on the PROSPER menu are evidence-based – that is, they have been rigorously evaluated and shown to be effective. Offering a menu of family and school programs allows teams to tailor their program offerings to meet the needs of their own communities. The teams are supported throughout their programming effort by program area specialists and evaluation experts from their university's Cooperative Extension System, and by prevention scientists and PROSPER implementation coaches. These programs are designed to build youth competencies, prevent problem behaviors (e.g., substance abuse, conduct problems), support positive youth development, and improve family functioning. Past studies of this model have found significant reductions in substance abuse and delinquency among youth receiving community prevention team programming (Spoth et al., 2013; Spoth et al., 2008).

We evaluated the PROSPER system within a community-level randomized controlled trial. This trial had a sample of 28 rural and semi-rural communities located in Iowa and Pennsylvania (see Crowley et al., 2014; Spoth, Randall, Trudeau, Shin, & Redmond, 2008). The communities were grouped into matched pairs based on school district size and geographic location; one member of each pair was randomly assigned to participate in PROSPER and the other to participate in the control condition (community services as usual). The school districts (control and intervention) participating in the PROSPER trial contained a total of 5,500 students in each of two sixth grade cohorts. Approximately half of the sample was male, 85% were white, 77% were from a two-parent home, and the families’ average annual income was approximately $67,800 (in 2014 dollars). As part of this trial, we conducted a cost analysis of the PROSPER system. This cost analysis spanned the first five years of the community prevention team’s operations. Team members’ volunteer contributions were tracked and ultimately monetized in order to estimate the full cost of the PROSPER system (Crowley et al., 2012).

Valuing Community Prevention Team Members’ Volunteer Time

The cost analysis approach taken for the PROSPER evaluation involved first quantifying the number of hours team members spent volunteering and then valuing that time (Levin & McEwan, 2000). As previously described, the ethical issues surrounded what was the most appropriate valuing rate to use. Over the five year period, team members volunteered over 7,000 hours of time (Crowley et al., 2012). This is roughly equivalent to the total amount of paid hours to local personnel (i.e., the cooperative extension agent). It is easy to see how quickly volunteer time can accumulate in a large-scale community effort—especially one that stretches across time. Thus, the valuing decision we needed to make was crucial to calculating the cost estimate. Failing to include this time would underestimate the needed resources to implement PROSPER in new communities. However, assuming the value of the team members’ time was equal to that of their occupations—considering all were local leaders and highly specialized professionals—would overestimate the costs (to the point of the program potentially appearing cost prohibitive). Moreover, volunteer time is integral to how the PROSPER system functions, so assuming a version without involving volunteers would misrepresent the nature of the program as well as more practical budgetary expectations. Ultimately, we took a two-pronged approach to navigate this challenge.

First, we settled on a partial targeted wage estimating procedure, which involved using the state median wage for volunteers. This allowed us to use the same rate across volunteers (by state) reducing concerns that labor market inequities may have been embedded in the cost estimate. We used the state wage rate instead of the national wage rate, as it allowed for some tailoring to local economic conditions, but also allowed us to produce a number that was generalizable to other communities within those states. We did not use county-level wages as that would have reduced the generalizability of the estimates outside the study sites. The major reason we did not tailor wages to the individual was that many of the team members’ primary occupations were highly specialized and these skills were not being used as part of the PROSPER effort (e.g., medical professionals, law enforcement, lawyers). Their primary tasks were fundraising, logistic coordination, and community mobilization. We also did not assume this volunteer time to be leisure time (carrying a higher rate). Further, we did not assume that PROSPER team activities provided a degree of intrinsic reward commensurate with the time given up, as the activities were far removed from those generally seen as being intrinsically motivating (e.g., religious service, children’s assistance, etc.). In this manner, we estimated that over $166,000 in volunteer time was provided for this implementation of PROSPER.

Next, in order to understand and convey the magnitude of these costs, in our reporting of PROSPER’s cost estimates we included how volunteer time impacted the overall cost of the program. This is commonly known as a sensitivity analysis. Specifically, we included what the cost would be if no volunteer costs were included, versus the cost that reflected the rate we provided. Reporting these multiple estimates (within one table) enhanced the transparency of the cost analysis, which increased the utility of these numbers as the PROSPER system expanded into new communities and new states.

Lessons Learned for Valuing our Communities’ Time

How we as a field choose to value our community partners’ time is not only symbolic of how we view their contribution, but it also has major ethical and pragmaticsignificance. Dismissing volunteer time as having no bearing on the cost of a community program is not only inaccurate, but can undermine the success of the effort in the long run (August, Bloomquist, Lee, Realmuto, & Hektner, 2006; Crowley et al., 2012). Not using wage rates that are reflective of what communities will experience in the ‘real world’ obfuscates the realities of community work and leaves local policymakers without key information for budgetary planning (National Academies of Medicine, 2015). More nuance is required to best handle such aspects of a cost analysis. As with many ethical challenges, there are no simple answers that apply to all situations. From this example we can draw important lessons for community researchers and practitioners who may want to include economic evaluations in their work. Specifically, what can and should those on the front lines of community psychology do when faced with decisions concerning how to value the time of volunteers who are crucial to the success of intervention efforts? From this work emerges three clear lessons that can guide our field and help those seeking to tackle this ethically fraught issue. These include ensuring evaluation projects provide (1) transparency in methods, (2) rigorous handling of uncertainty, and (3) thoughtful communication.

Transparency in Methods

One of the best approaches for navigating ethical issues in economic evaluation of community programs, such as the valuing of volunteer time, is to actively seek to be transparent about the methods you use and results you find (Beatty, 2009; Crowley et al., 2013). Although transparency is good practice universally, the methods required to conduct an economic evaluation are often unfamiliar to many community psychologists. In particular, they are often complicated and at times jargon laden. The space needed to explain the methodology used to estimate the costs (as well as benefits) of complex community interventions is unlikely to fit within a standard AJCP or similar journal article. Researchers are well served to prepare detailed appendices and supplementary material that can be housed online to ensure that interested audiences can understand and replicate findings.

In the example above of PROSPER, a supplementary materials document was prepared and is available online. That document walks the readers through the entire cost analysis process, providing clear steps on how different estimates were calculated. Further, discussion of how different ethical challenges were navigated can be more fully explored in such documents—even when not the primary focus of the paper. In this manner, we can build a written narrative around how to handle different ethical issues and support the development of a robust ethical framework for our field.

Rigorous Handling of Uncertainty

As researchers attempt to navigate ethical issues, such as how to value community volunteers’ time, they make decisions that are likely to impact final cost and benefits estimates. In a traditional statistical analysis, we handle uncertainty from researcher decision making (e.g., measurement choices, study design, population included, etc.) by calculating the standard error around a coefficient. That error then factors into significance testing. Some decisions on the part of the researcher will often increase uncertainty surrounding the estimate and make it more difficult to obtain significant findings. In a similar fashion it is important to not just develop point estimates of the cost or benefit of a community program, but to model the uncertainty in that estimate and produce a confidence interval (Crowley et al., 2013; Yates, 1994). In the example of PROSPER, we conducted such sensitivity analyses to develop this confidence interval. For instance, the inclusion or exclusion of the valued volunteer time had a meaningful impact on the total cost of the program. By rigorously seeking to handle and understand the uncertainty in our estimates—particularly those introduced by efforts to navigate ethical issues—we will better serve the field and our communities. Sensitivity analyses alone cannot answer all questions regarding the uncertainty in a cost estimate. The evaluator themselves must determine the ranges of values to model. For instance, how will the time of nonworking volunteers be valued (e.g., retirees, unemployed). In this context, we point community researchers to both the economics literature (Feather & Shaw, 1999; Jara-Díaz et al., 2008; Weinstein et al., 1997) and suggest meaningful exploration of the issue with community partners.

Thoughtful Communication of Results

Our third recommendation is that researchers seeking to value communities’ time should follow, is the importance of carefully communicating findings to audiences (Kuklinski et al., 2012; National Academy of Medicine, 2014). Successful and accurate communication is largely influenced by the way findings are reported. In the case of PROSPER, reporting program costs with and without volunteer costs in a single table has been valuable in ensuring that as we work with new stakeholders interested in PROSPER we immediately have a conversation about the role volunteers play in the success of the effort. In particular, this allows us to highlight the need to budget for additional staff time if they do not believe local volunteers are going to be available at the time of implementation. Researchers should carefully consider how tables and figures display information about the findings of the project and consider community stakeholders’ perspectives. For instance, do these estimates apply to the local conditions of the communities you are working with? How might they be adjusted? How will stakeholders’ time be used? By empathizing with their audience, researchers are likely to better humanize the economic estimates produced and increase their value to local stakeholders.

The Ethics of Valuing Community Time: Resources and Moving Forward

As the field continues its efforts to build a robust ethical framework (Campbell, 2016), it will be important to consider the growing role of economic evaluation within community research. How we value the time of our community partners is one of many issues to be addressed. Recent efforts to build standards around economic evaluation by the National Academies, the Society for Prevention Research and the AHRQ 2nd Panel on Cost Effectiveness Analysis will be key resources for this work. Further, collaborations between community research organizations and multidisciplinary groups such as the NIH-supported Prevention Economics Planning and Research Network offer opportunities for fruitful partnerships.

In general, it is expected that the importance of economic evaluation within community psychology will continue to grow, as researchers and practitioners make the case for investing in evidence-based interventions. Yet, economic evaluation must be conducted in a way that recognizes the important contributions of the multitude of stakeholders that make community based programming possible. This is not only a matter of scientific accuracy, but a key ethical consideration for the field.

Acknowledgments

This work was supported in part by the Prevention Economics Planning and Research Network (R13 DA036339) and the National Institute on Drug Abuse (DA036339)

References

  1. August GJ, Bloomquist ML, Lee SS, Realmuto GM, Hektner JM. Can Evidence-Based Prevention Programs be Sustained in Community Practice Settings? The Early Risers’ Advanced-Stage Effectiveness Trial. Prevention Science. 2006;7:151–165. doi: 10.1007/s11121-005-0024-z. https://doi.org/10.1007/s11121-005-0024-z. [DOI] [PubMed] [Google Scholar]
  2. Barnett WS, Masse LN. Comparative benefit–cost analysis of the Abecedarian program and its policy implications. Economics of Education Review. 2007;26(1):113–125. https://doi.org/10.1016/j.econedurev.2005.10.007. [Google Scholar]
  3. Beatty AS. Strengthening benefit-cost analysis for early childhood interventions workshop summary. Washington, D.C.: National Academies Press; 2009. [Google Scholar]
  4. Belfield C, Levin H. Guiding the development and use of cost-effectiveness analysis in education. Center for Benefit-Cost Studies of Education: Columbia University; 2013. Retrieved from http://cbcse.org/wordpress/wp-content/uploads/2013/08/Guiding-the-Development-And-Use-of-Cost-effectiveness-Analysis-in-Education.pdf. [Google Scholar]
  5. Brouwer WBF, Koopmanschap MA, Rutten FFH. Productivity Costs Measurement Through Quality of Life? A Response to the Recommendation of the Washington Panel. Health Economics. 1997;6(3):253–259. doi: 10.1002/(sici)1099-1050(199705)6:3<253::aid-hec266>3.0.co;2-6. https://doi.org/10.1002/(SICI)1099-1050(199705)6:3<253::AID-HEC266>3.0.CO;2-6. [DOI] [PubMed] [Google Scholar]
  6. Campbell R. “It’s the Way That You Do It”: Developing an Ethical Framework for Community Psychology Research and Action. American Journal of Community Psychology. 2016 doi: 10.1002/ajcp.12037. https://doi.org/10.1002/ajcp.12037. [DOI] [PubMed]
  7. Chinman M, Hannah G, Wandersman A, Ebener P, Hunter SB, Imm P, Sheldon J. Developing a Community Science Research Agenda for Building Community Capacity for Effective Preventive Interventions. American Journal of Community Psychology. 2005;35:143–157. doi: 10.1007/s10464-005-3390-6. https://doi.org/10.1007/s10464-005-3390-6. [DOI] [PubMed] [Google Scholar]
  8. Cohen JT, Neumann PJ, Weinstein MC. Does Preventive Care Save Money? Health Economics and the Presidential Candidates. New England Journal of Medicine. 2008;358:661–663. doi: 10.1056/NEJMp0708558. https://doi.org/10.1056/NEJMp0708558. [DOI] [PubMed] [Google Scholar]
  9. Crowley DM. The Role of Social Impact Bonds in Pediatric Health Care. PEDIATRICS. 2014;134(2):e331–e333. doi: 10.1542/peds.2013-4056. https://doi.org/10.1542/peds.2013-4056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Crowley DM, Hill LG, Kuklinski MR, Jones D. Research Priorities for Economic Analyses of Prevention: Current Issues & Future Directions. Prevention Science. 2013 doi: 10.1007/s11121-013-0429-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Crowley DM, Jones DE, Greenberg MT, Feinberg ME, Spoth R. Resource Consumption of a Diffusion Model for Prevention Programs: The PROSPER Delivery System. Journal of Adolescent Health. 2012;50(3):256–263. doi: 10.1016/j.jadohealth.2011.07.001. https://doi.org/10.1016/j.jadohealth.2011.07.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Crowley M, Jones D. Financing prevention: opportunities for economic analysis across the translational research cycle. Transaltional Behavioral Medicine. 2015 doi: 10.1007/s13142-015-0354-8. https://doi.org/10.1007/s13142-015-0354-8. [DOI] [PMC free article] [PubMed]
  13. Drummond MF. Methods for the economic evaluation of health care programmes. Oxford; New York: Oxford University Press; 2005. [Google Scholar]
  14. Feather P, Shaw WD. Estimating the Cost of Leisure Time for Recreation Demand Models. Journal of Environmental Economics and Management. 1999;38(1):49–65. https://doi.org/10.1006/jeem.1999.1076. [Google Scholar]
  15. Feinberg ME, Bontempo DE, Greenberg MT. Predictors and Level of Sustainability of Community Prevention Coalitions. American Journal of Preventive Medicine. 2008;34(6):495–501. doi: 10.1016/j.amepre.2008.01.030. https://doi.org/10.1016/j.amepre.2008.01.030. [DOI] [PubMed] [Google Scholar]
  16. Feinberg ME, Chilenski SM, Greenberg MT, Spoth R, Redmond C. Community and Team Member Factors that Influence the Operations Phase of Local Prevention Teams: The PROSPER Project. Prevention Science. 2007;8:214–226. doi: 10.1007/s11121-007-0069-2. https://doi.org/10.1007/s11121-007-0069-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Foster EM, Dodge KA, Jones D. Issues in the Economic Evaluation of Prevention Programs. Applied Developmental Science. 2003;7:76–86. doi: 10.1207/S1532480XADS0702_4. https://doi.org/10.1207/S1532480XADS0702_4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Foster EM, Porter MM, Ayers TS, Kaplan DL, Sandler I. Estimating the Costs of Preventive Interventions. Evaluation Review. 2007;31(3):261–286. doi: 10.1177/0193841X07299247. https://doi.org/10.1177/0193841X07299247. [DOI] [PubMed] [Google Scholar]
  19. Gruen R, Elliott J, Nolan M, Lawton P, Parkhill A, Mclaren C, Lavis J. Sustainability science: an integrated approach for health-programme planning. The Lancet. 2008;372(9649):1579–1589. doi: 10.1016/S0140-6736(08)61659-1. https://doi.org/10.1016/S0140-6736(08)61659-1. [DOI] [PubMed] [Google Scholar]
  20. Haddix AC, Teutsch SM, Corso PS. Prevention effectiveness : a guide to decision analysis and economic evaluation. Oxford; New York: Oxford University Press; 2003. [Google Scholar]
  21. Haskins R, Margolis G. Show me the evidence: Obama’s fight for rigor and evidence in social policy. Washington, DC: Brookings Institution Press; 2015. [Google Scholar]
  22. Hawkins JD. Results of a Type 2 Translational Research Trial to Prevent Adolescent Drug Use and Delinquency<subtitle>A Test of Communities That Care</subtitle><alt-title>Adolescent Drug Use and Delinquency</alt-title>. Archives of Pediatrics & Adolescent Medicine. 2009;163(9):789. doi: 10.1001/archpediatrics.2009.141. https://doi.org/10.1001/archpediatrics.2009.141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Jara-Díaz SR, Munizaga MA, Greeven P, Guerra R, Axhausen K. Estimating the value of leisure from a time allocation model. Transportation Research Part B. Methodological. 2008;42(10):946–957. https://doi.org/10.1016/j.trb.2008.03.001. [Google Scholar]
  24. Kuklinski MR, Briney JS, Hawkins JD, Catalano RF. Cost-Benefit Analysis of Communities That Care Outcomes at Eighth Grade. Prevention Science. 2012 doi: 10.1007/s11121-011-0259-9. https://doi.org/10.1007/s11121-011-0259-9. [DOI] [PMC free article] [PubMed]
  25. Larson DM. Separability and the Shadow Value of Leisure Time. American Journal of Agricultural Economics. 1993;75(3):572. https://doi.org/10.2307/1243564. [Google Scholar]
  26. Levin, McEwan Cost-effectiveness analysis: methods and applications. [Retrieved March 15, 2011];2000 from http://books.google.com/books?hl=en&lr=&id=HniLG23vYDwC&oi=fnd&pg=PR15&dq=%22comparison+group*%22+in+cost-effectiveness+analysis&ots=cIosgGlSjI&sig=-H4khQJnQCjqPlzUAgAR-Mwsc5E#v=onepage&q=comparison%20group&f=false.
  27. Max Crowley D, Jones DE, Coffman DL, Greenberg MT. Can we build an efficient response to the prescription drug abuse epidemic? Assessing the cost effectiveness of universal prevention in the PROSPER trial. Preventive Medicine. 2014 doi: 10.1016/j.ypmed.2014.01.029. https://doi.org/10.1016/j.ypmed.2014.01.029. [DOI] [PMC free article] [PubMed]
  28. Nas TF. Cost-benefit analysis : theory and applications. Thousand Oaks, Ca: Sage; 1996. [Google Scholar]
  29. National Academies of Medicine. The Use of Economic Evidence to Inform Investments in Children, Youth, and Families. Washington D.C.: National Academies of Sciences, Engineering and Medicine; 2015. Retrieved from http://iom.nationalacademies.org/Activities/Children/EconomicEvidence.aspx. [PubMed] [Google Scholar]
  30. National Academy of Medicine. Considerations in Applying Benefit-Cost Analysis to Preventive Interventions for Children, Youth, and Families: Workshop Summary. Washington, D.C.: National Academies Press; 2014. Retrieved from http://www.nap.edu/catalog/18708. [PubMed] [Google Scholar]
  31. Perkins DF, Feinberg ME, Greenberg MT, Johnson LE, Chilenski SM, Mincemoyer CC, Spoth RL. Team factors that predict to sustainability indicators for community-based prevention teams. Evaluation and Program Planning. 2011;34(3):283–291. doi: 10.1016/j.evalprogplan.2010.10.003. https://doi.org/10.1016/j.evalprogplan.2010.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Savaya R, Spiro SE. Predictors of Sustainability of Social Programs. American Journal of Evaluation. 2011 https://doi.org/10.1177/1098214011408066.
  33. Scheirer MA, Dearing JW. An Agenda for Research on the Sustainability of Public Health Programs. American Journal of Public Health. 2011;101(11):2059–2067. doi: 10.2105/AJPH.2011.300193. https://doi.org/10.2105/AJPH.2011.300193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Spoth R, Greenberg M, Bierman K, Redmond C. PROSPER Community–University Partnership Model for Public Education Systems: Capacity-Building for Evidence-Based, Competence-Building Prevention. Prevention Science. 2004;5:31–39. doi: 10.1023/b:prev.0000013979.52796.8b. https://doi.org/10.1023/B:PREV.0000013979.52796.8b. [DOI] [PubMed] [Google Scholar]
  35. Spoth R, Guyll M, Redmond C, Greenberg M, Feinberg M. Six-Year Sustainability of Evidence-Based Intervention Implementation Quality by Community-University Partnerships: The PROSPER Study. American Journal of Community Psychology. 2011;48:412–425. doi: 10.1007/s10464-011-9430-5. https://doi.org/10.1007/s10464-011-9430-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Spoth RL, Randall GK, Trudeau L, Shin C, Redmond C. Substance use outcomes 5½ years past baseline for partnership-based, family-school preventive interventions. Drug and Alcohol Dependence. 2008;96(1–2):57–68. doi: 10.1016/j.drugalcdep.2008.01.023. https://doi.org/10.1016/j.drugalcdep.2008.01.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Spoth R, Redmond C, Shin C, Greenberg M, Feinberg M, Schainker L. PROSPER community–university partnership delivery system effects on substance misuse through 6 1/2 years past baseline from a cluster randomized controlled intervention trial. Preventive Medicine. 2013;56(3–4):190–196. doi: 10.1016/j.ypmed.2012.12.013. https://doi.org/10.1016/j.ypmed.2012.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. US Census Bureau. PINC-10. Wage and Salary Workers -- People 15 Years Old and Over, by Total Wage and Salary Income in 2008, Work Experience in 2008, Race, Hispanic Origin, and Sex. [Retrieved March 15, 2011]; (n.d.). from http://www.census.gov/hhes/www/cpstables/032009/perinc/new10_001.htm.
  39. Vining A, Weimer DL. An Assessment of Important Issues Concerning the Application of Benefit-Cost Analysis to Social Policy. Journal of Benefit-Cost Analysis. 2010;1(1) https://doi.org/10.2202/2152-2812.1013. [Google Scholar]
  40. Wandersman A. Getting to outcomes: a results-based approach to accountability. Evaluation and Program Planning. 2000;23(3):389–395. https://doi.org/10.1016/S0149-7189(00)00028-8. [Google Scholar]
  41. Weinstein MC, Siegel JE, Garber AM, Lipscomb J, Luce BR, Manning WG, Torrance GW. Productivity costs, time costs and health-related quality of life: a response to the Erasmus Group. Health Economics. 1997;6(5):505–510. doi: 10.1002/(sici)1099-1050(199709)6:5<505::aid-hec294>3.0.co;2-i. https://doi.org/10.1002/(SICI)1099-1050(199709)6:5<505::AID-HEC294>3.0.CO;2-I. [DOI] [PubMed] [Google Scholar]
  42. Yates BT. Toward the incorporation of costs, cost-effectiveness analysis, and cost-benefit analysis into clinical research. Journal of Consulting and Clinical Psychology. 1994;62(4):729–736. doi: 10.1037//0022-006x.62.4.729. https://doi.org/10.1037/0022-006X.62.4.729. [DOI] [PubMed] [Google Scholar]
  43. Zerbe RO. Should moral sentiments be incorporated into benefit-cost analysis? An example of long-term discounting. Policy Sciences. 2005;37(3–4):305–318. https://doi.org/10.1007/s11077-005-5750-3. [Google Scholar]
  44. Zerbe RO, Davis T, Garland N, Scott T. Toward principles and standards in the use of benefit-cost analysis. Seattle, WA: Benefit-Cost Analysis Center, Evans School of Public Affairs, University of Washington; 2010. Retrieved from http://evans.washington.edu/files/Final-Principles-and%20Standards-Report-6_23_2011.pdf. [Google Scholar]

RESOURCES