Abstract
Rational, aims and objectives:
Individual Placement and Support (IPS) is an evidence-based strategy for facilitating employment among adults with severe mental illness (SMI), and relies on staff to facilitate mock job interviews, which limits its scalability and cost effectiveness. A virtual reality job interview training program (VR-JIT)—delivered via the internet—has demonstrated efficacy for increasing employment among adults with SMI. Now, VR-JIT is being implemented with a community mental health agency (CMHA) and evaluated for its effectiveness within IPS. This study is a budget impact analysis, evaluating the costs of preparing a CMHA to implement VR-JIT.
Method:
Implementation preparation occurred over 7 months from October 1, 2016 to April 30, 2017. Members of the CMHA (n=15) and the external research partner (n=3) tracked their hours completing implementation preparation activities. Salaries plus a 28% fringe benefit rate were used to derive a per-hour salary amount for each individual and applied to each activity. Non-labor equipment costs were obtained from purchase receipts. A budget impact analysis evaluated the expenditures associated with preparing the CMHA to implement VR-JIT.
Results:
The total implementation preparation costs equaled $25,482. Labor costs equaled $22,882 and non-labor costs equaled $2,600. In total, 655 person-hours were spent preparing for VR-JIT implementation. Preparation activities included time spent in meetings ($11,475; 269 hours), preparing materials ($2,100; 67 hours), in trainings ($3,349; 215 hours), obtaining technical support ($1,508; 33 hours), supervising operations ($3,545; 60 hours), and preparing the lab space ($905; 23 hours).
Conclusions:
This study presents an initial evaluation of the budget impact of preparing to implement VR-JIT in a CMHA. Cost considerations for future implementation preparation will be discussed. Given that the cost to prepare to implement an intervention can hinder its adoption, results provide an important analysis for decision-makers that may enhance uptake. Future work will determine the cost-effectiveness of VR-JIT implemented within IPS. This study is registered at http://clinicaltrials.gov, NCT = 03049813, “Virtual Reality Job Interview Training: An Enhancement to Supported Employment in Severe Mental Illness.”
Keywords: budget impact analysis, implementation cost, virtual reality job interview training, health economics, community mental health
Introduction
Individual Placement and Support (IPS) is the standard version of supported employment recommended by the Substance Abuse and Mental Health Services Administration. There are 523 IPS-supported employment programs in the United States (U.S.)1. Across 25 randomized controlled trials, 56% of IPS clients were employed compared with 23% of controls, and IPS clients obtained employment in fewer days, held their jobs for more days, and worked more hours per week compared with controls2,3. Although IPS has positive vocational outcomes and is cost-effective4, the developers of IPS suggest that technology may be able to enhance IPS services5. However, few technology-based enhancements have been evaluated, so there is little information available to make informed decisions about how much it will cost to integrate technology-based enhancements into IPS services.
Virtual reality job-interview training (VR-JIT) is a technology-based intervention delivered via the Internet with demonstrated efficacy at improving interview skills and producing a twofold increase in access to jobs across five randomized controlled trials of adults with varying mental health needs6–9. Although the results of a VR-JIT community effectiveness study are pending10, VR-JIT is being implemented in 46 organizations serving over 674 trainees across 20 U.S. states and in Canada and New Zealand. Hence, there is a large-scale effort to disseminate this intervention prior to establishing its community-based effectiveness. Thus, given the emerging rollout of VR-JIT in the community, service providers would greatly benefit from learning more about the budgetary impact of preparing to implement VR-JIT within their agencies, especially because cost is the leading barrier to adoption and sustainability of technology-based interventions11. Specifically, understanding the costs of preparing to implement an intervention is important because this is a calculation of costs that are incurred before any trainees are enrolled in the intervention, meaning prior to the service provider receiving any return on the investment of delivering the intervention.
Implementation science process models define the period between when an organization chooses to adopt a new innovation and when that innovation becomes available in the organization as the “Preparation” phase12. Key activities of the preparation phase focus on training staff, identifying communication channels and champions, adapting program materials, specifying staff roles for delivery of intervention components, and identifying and preparing delivery methods (e.g., technology and space). Thus, this study reports on the costs of implementation preparation within a large IPS service provider that is currently evaluating VR-JIT effectiveness. We refer to “implementation preparation” to refer to the resources and activities that are required to make VR-JIT available in this setting (i.e., to prepare agencies to deliver VR-JIT), prior to actually engaging trainees in the intervention. The primary aim of this paper is to present the results of a budget impact analysis that we conducted to assess the expected short-term changes in expenditures for a community mental health agency (CMHA) delivering IPS after choosing to adopt VR-JIT. Given that this study was the first implementation evaluation of VR-JIT, time spent on some implementation-preparation activities was likely greater than what would be necessary for future implementation. Thus, as a secondary exploratory aim, we conducted sensitivity analyses to estimate the costs associated with replicating the activities associated with implementation preparation of VR-JIT, to inform future implementation of VR-JIT.
Methods
Our presentation of methods and results adheres to the reporting guidelines outlined by Sullivan and colleagues for conducting a budget impact analysis13.
Participants
Participants in this study were members of the implementation support team, which consisted of an external scientific partner (n = 3) and members of the CMHA (n = 15). The CMHA contained two divisions that were involved with preparing for implementation: an employment team (whose role was to conduct the job-interview trainings with clients presenting to the community agency) and an internal research team (whose role was to facilitate partnerships with external research groups).
Intervention Mix
VR-JIT is a computer-based job-interview simulation delivered via the Internet, in which trainees repeatedly practice interviewing with a virtual hiring manager named Molly Porter. The Molly character was created by filming an actress reciting approximately two thousand lines of dialogue with variations in mood and personality. Each interaction between trainees and Molly is facilitated by speech-recognition software and lasts approximately 25 minutes, with 10–15 hours of unique virtual job-interview experience gained overall. Our randomized, controlled efficacy evaluations suggested that VR-JIT improved interview skills in five cohorts of adults with various mental and behavioral health disorders (depression, schizophrenia, PTSD, autism, addiction)6,9,14–16 and increased their likelihood of getting job offers more than twofold within six months of completing VR-JIT (i.e., 14–25% of controls received job offers compared with 48–54% of VR-JIT trainees)6–9.
Time Horizon
Implementation preparation occurred over seven months: between October 1, 2016, and April 30, 2017.
Perspective
This analysis takes the perspective of the budget holders—that is, the decision-makers at CMHAs considering adoption of this intervention.
Analytic Framework
Our computing framework involved a cost-calculator approach, which has been described as the preferred approach for a budget impact analysis13.
Input Data
Costs of Preparing to Implement VR-JIT.
Table 1 presents the labor and nonlabor cost-input parameters associated with implementation preparation. All costs are presented in 2017 U.S. dollars, and salary estimates included a 28% standard fringe benefit rate. Labor costs were based on time spent preparing to implement the intervention and are presented in aggregate to keep these data unidentifiable. Nonlabor costs were attributed to equipment needed to carry out the intervention.
Table 1.
Budget Impact Analysis Labor and Nonlabor Input Parameters
| Variables | Input Parameter | Reference |
|---|---|---|
| Average Salaries | ||
| External Scientific Partner (n = 3) | $77,440 | Actual salaries |
| Community Agency (n = 15) | $70,252 | Average or median salaries |
| Hardware | ||
| Computers (n = 5) | $1,650 | Purchase order receipts |
| Headphones (n = 50) | $600 | |
| Microphones (n = 5) | $150 | |
| Sound-boxes (n = 5) | $200 |
Estimated Costs to Replicate the Implementation Preparation of VR-JIT.
Members of the implementation support team provided estimates of the time that would be necessary to replicate implementation preparation in another CMHA setting. Using those estimates, members also identified a reasonable range of values for each estimate for sensitivity testing. The values were derived based on team members’ experiences in the trial and confirmed by team consensus. Table 2 shows the replication estimate input parameters.
Table 2:
Assumptions of Estimates for Replication Input Parameters for Sensitivity Testing
| Labor Costs for Each Implementation-Preparation Activity | Estimate | Range |
|---|---|---|
| Weekly meetings of and correspondence among the implementation support team members | 15% | 10–20% |
| Meetings of and correspondence among CMHA staff | 20% | 15–25% |
| Meetings of and correspondence among external scientific partner staff | 15% | 10–20% |
| Developing, tailoring, and printing training materials | 20% | 15–30% |
| Reviewing VR-JIT training materials | 75% | 50–100% |
| Leading VR-JIT orientation didactic training | 20% | 15–30% |
| Participating in meetings or sending emails for orientation training | 25% | 20–35% |
| Supervising VR-JIT orientation role-play training | 10% | 5–15% |
| Participating in training to deliver VR-JIT or to supervise training to deliver VR-JIT | 50% | 40–70% |
| Communicating with the technology partner for technical support | 60% | 50–75% |
| Engaging CMHA technical support | 60% | 25–80% |
| Supervising CMHA operations and ongoing correspondence | 50% | 40–60% |
| Designing the lab space | 100% | 95–105% |
| Purchasing materials for the VR-JIT lab | 100% | 95–105% |
| Corresponding with information systems | 100% | 95–105% |
| Ordering materials | 100% | 95–105% |
| Reviewing resources and allocation with the finance department | 100% | 95–105% |
| Tracking materials for resource allocation | 100% | 95–105% |
| Nonlabor Costs | Needed for Replication? | |
| Computers | Required1 | |
| Headphones | Required1 | |
| Microphones | Required1 | |
| Sound-boxes | Required1 | |
Alternative equipment (e.g., headphones with attached microphones) could be substituted.
Data Sources
Actual salaries were obtained from each member of the external scientific partner. Salaries for CMHA staff were based on averages or midpoint estimates for each job role at the agency; actual salaries were not requested from the community agency to protect the anonymity of the participating employees. Nonlabor equipment costs were obtained from purchasing receipts, and therefore were based on actual amounts spent.
Data Collection
Individuals involved in this phase of the project tracked the time spent engaged in implementation-preparation activities. They recorded their time completing each activity via a cost-capture log17, delivered online. The cost-capture log was developed for this study, and items were tailored to different roles on the project for ease of completion. This evaluation received approval by the Northwestern University and University of Michigan’s Institutional Review Boards. All community partners provided informed consent for their participation in this study.
Analyses
Costs of Preparing to Implement VR-JIT.
Labor costs were derived from calculating a per-hour salary amount (including fringe) from each individual’s salary. We applied each individual’s per-hour rate to each implementation-preparation activity. We then aggregated the values across all individuals to derive estimates of the time and costs spent completing each implementation-preparation activity. Descriptive analyses were used to estimate the total number of labor hours, total labor costs, and total nonlabor costs associated with implementation preparation.
Estimated Costs to Replicate the Implementation Preparation of VR-JIT.
Descriptive analyses were used to explore the costs of replicating the implementation preparation of VR-JIT in a future setting. We tested the sensitivity of the replication estimates by varying each estimate across a range of reasonable values for each activity.
Results
Costs of Preparing to Implement VR-JIT
Table 3 shows the total number of labor hours and labor costs accrued during implementation preparation. The total number of labor hours was 566, and the total labor cost was $22,882. Just over half of the labor hours (54%) and labor costs (56%) were accrued by staff from the external scientific partner. Meetings and correspondence activities required the most hours (269 hours; 48%) and greatest costs ($11,476; 50%). Nonlabor costs, presented in Table 1, amounted to $2,600, with computers accounting for the majority of the expenses ($1,650; 63%). Combined, the total cost of implementation preparation (labor and nonlabor costs) was $25,482.
Table 3.
Labor Hours and Costs by VR-JIT Implementation-Preparation Activity
| Implementation-Preparation Activities | N1 | Total Hours | Proportion of Total Hours | Proportion of Hours Accrued by the External Scientific Partner | Total Labor Costs | Proportion of Costs Accrued by the External Scientific Partner |
|---|---|---|---|---|---|---|
| Meetings and Correspondence | ||||||
| Weekly meetings of and correspondence among implementation support team members2 | 7 | 92 | 16% | 60% | $3,958 | 55% |
| Meetings of and correspondence among CMHA staff3 | 5 | 77 | 14% | 6% | $3,480 | 9% |
| Meetings of and correspondence among external scientific partner staff | 3 | 100 | 18% | 100% | $4,037 | 100% |
| Materials | ||||||
| Developing, tailoring, and printing training materials | 5 | 62 | 11% | 97% | $1,813 | 96% |
| Reviewing VR-JIT training materials | 2 | 5 | 1% | 80% | $287 | 86% |
| Training | ||||||
| Leading VR-JIT orientation didactic training | 1 | 6 | 1% | 100% | $369 | 100% |
| Participating in meetings or sending emails for orientation training | 10 | 6 | 1% | 0% | $169 | 0% |
| Supervising VR-JIT orientation role-play training | 4 | 23 | 4% | 87% | $752 | 84% |
| Participating in training to deliver VR-JIT or to supervise training to deliver VR-JIT | 9 | 80 | 14% | 0% | $2,059 | 0% |
| Technical Support | ||||||
| Communicating with the technology partner for technical support | 4 | 11 | 2% | 57% | $569 | 65% |
| Engaging community agency technical support | 4 | 22 | 4% | 18% | $939 | 26% |
| Supervision | ||||||
| Supervising community agency operations and ongoing correspondence | 2 | 60 | 11% | 67% | $3,545 | 69% |
| Lab Space | ||||||
| Designing the lab space | 2 | 4 | 1% | 0% | $155 | 0% |
| Purchasing materials for the VR-JIT lab | 1 | 5 | 1% | 100% | $103 | 100% |
| Corresponding with information systems | 1 | 1 | <0.1% | 0% | $54 | 0% |
| Ordering materials | 2 | 3 | 1% | 0% | $139 | 0% |
| Reviewing resources and allocation with the finance department | 1 | 5 | 1% | 0% | $271 | 0% |
| Tracking materials for resource allocation | 1 | 5 | 1% | 0% | $183 | 0% |
| TOTAL | 18 | 566 | — | 54% | $22,882 | 56% |
Notes. CMHA = Community Mental Health Agency.
N refers to the number of participants, and individuals can be counted in multiple categories.
The implementation support team is composed of the community agency and the external scientific partner.
Includes occasional brief consultation with a member of the external scientific partner.
Estimated Costs to Replicate the Implementation Preparation of VR-JIT
Table 2 shows the estimates to replicate this effort for each implementation-preparation activity. Based on these values, the estimated total cost for replication is $7,276, which equates to 32% of the implementation-preparation costs incurred in this project.
Because these replication values were estimates, we conducted a sensitivity analysis to vary these estimates across a reasonable range for each implementation-preparation activity. To estimate the range of potential costs across each implementation-preparation activity, the cost of each activity (presented in Table 3) was multiplied by the minimum and maximum percent effort that were estimated to be needed to replicate it (from Table 2); these resulting values were then added to or subtracted from the total estimated replication cost to demonstrate how much more or less each implementation-preparation activity might be expected to cost. For example, the activity “Participating in training to deliver VR-JIT or to supervise training to deliver VR-JIT” cost $2,059 in the study (shown in Table 3). We estimated it would require 50% effort to replicate this activity ($1,029), with a possible range of 40–70% for that replication (presented in Table 2). This means that if replication required 70% effort, there could be an increase in costs of $412 (i.e., $1,029 + $412 = $1,441 for that activity), and if replication required only 40% effort, there could be a decrease in costs of $206 (i.e., $1,029 – 206 = $823 for that activity). Thus, the total estimated cost to replicate the entire study ($7,276) could be expected to change by +$412 or −$206, depending on the amount of effort expended on the specific activity “Participating in training to deliver VR-JIT or to supervise training to deliver VR-JIT.” Figure 1 presents the results of the sensitivity analysis and depicts the range of estimated costs for each implementation-preparation activity. For each row in the figure, the $0 mark in the center of the diagram refers to the estimated cost to replicate each implementation-preparation activity.
Figure 1.
Sensitivity testing to determine the change in costs for each replication estimate input parameter for each VR-JIT implementation-preparation labor activity
As shown in Figure 1, the activity that is expected to incur the greatest variation in a replication is supervising CMHA operations and ongoing correspondence (estimated to increase or decrease the total replication costs by $354). Results show that to save the most money in a replication, implementation would benefit from reducing the time spent supervising CMHA operations (e.g., a savings of $354 could be realized if replication required only 40% effort instead of 50% effort) and engaging CMHA technical support (e.g., a savings of $329 could be realized if replication required only 25% effort instead of 60% effort).
Table 2 also identifies which nonlabor costs would be required for a replication. As shown, all the equipment could be replaced with alternative models (e.g., headphones with attached microphones). Also, these items were purchased for use in a stationary lab. Alternative equipment also might be used for a nonstationary lab (e.g., tablets could be used instead of computers).
Discussion
Because cost is frequently cited as the primary barrier to adoption and sustainability of new interventions18,19, we conducted a prospective budget impact analysis of the costs to prepare to implement VR-JIT in a CMHA. Our study results show that the total cost of preparing to implement VR-JIT was $25,482. The vast majority of these costs were labor costs, and half of all labor costs arose from meetings and correspondence activities. In exploratory analyses, we estimated that the cost of replicating VR-JIT at another CMHA would be $7,276, or approximately one-third of the cost of implementation preparation in the initial setting. The dramatically lower implementation-preparation costs for a subsequent implementation of VR-JIT are primarily due to this study being the first to implement VR-JIT in a CMHA, which required notable development work by the CMHA employment team, the CMHA internal research team, and the external scientific partner; the results of this work could be adopted with minimal cost by future implementers.
Although the costs of preparing to implement VR-JIT may appear high, many interventions have notable startup costs that are separate from the costs of delivering the interventions themselves20,21. Research on the IPS model of supported employment has shown that the average cost per client to deliver the intervention during the first two years is higher than in subsequent years (approximately $3,500-$5,000 annual cost per client in 2005 dollars)22, which supports the idea that implementation start-up costs exceed ongoing operation costs. Further, because our study was the first implementation of VR-JIT in a CMHA, there were several activities that were required for successful implementation, such as development, review, and tailoring of training materials, as well as infrastructure development. Many of the training materials developed for this implementation can be reused by other CMHAs that wish to implement VR-JIT, which should lead to lower implementation costs, presumably making the endeavor more attractive to decision-makers. Additionally, as our analysis focused on the costs of preparing to implement VR-JIT, we anticipated that many of these up-front implementation activities will lesson over time with ongoing implementation. Evaluating the implementation costs associated with delivering VR-JIT is an important next step.
An advantage of the detailed, prospective, cost-capture-based approach of the budget impact analysis when estimating implementation-preparation costs is its specificity in identifying costs associated with particular implementation activities. For example, our analysis indicates that the time spent on email communication and in meetings by the implementation support team, the CMHA staff, and the external scientific partner staff represented about half of all labor costs. Although we estimated that VR-JIT-replication efforts at other CMHAs will require only 10%−25% of those costs, future adopters of VR-JIT may want to consider other, potentially more cost-effective ways to use technology to exchange information about implementation preparation (e.g., using a SharePoint site to help collaborative teams organize and share information).
Another strength of our study is the prospective estimation of costs specific to implementation preparation aside from the costs of the VR-JIT technology itself. Some readers may wonder how the estimates and totals resulting from this study differ from what could be ascertained from other sources (e.g., by contracting with VR-JIT developers for VR-JIT training and implementation support). In a contract for implementation support, implementation-preparation costs would be aggregated into a broader line item such as training, where total cost to the adopter is provided but the individual training component costs (e.g., time spent tailoring training materials, time spent by the trainer traveling to and then conducting the training) are not necessarily provided. Our results also captured costs that might not be included in a contract for implementation support, such as the cost of having CMHA staff attend meetings and training sessions. The costs associated with these activities are not trivial and also may vary by implementation site. Having disaggregated information about implementation-preparation activities and their associated costs helps CMHA decision-makers understand the true cost to their organizations of preparing to implement VR-JIT.
Limitations
Despite the noted strengths of this study, we acknowledge a few study limitations. Our study of VR-JIT was implemented in a large CMHA in a major metropolitan area. Our results may not generalize as well to smaller CMHAs or those in less densely populated areas, so future research should assess implementation-preparation costs of VR-JIT in CMHAs of different sizes and in different geographic areas. In addition, our study design focused on the preparation of delivering VR-JIT in a computer lab at the CMHA, and it has limited generalizability to other delivery strategies (e.g., delivery to clients at home or in public settings).
Conclusion
Our assessment of implementation-preparation activities associated with providing VR-JIT in a CMHA provides critical information about the expected costs of adopting and preparing to implement a new, technology-based IPS intervention. Because implementation-preparation cost data are scarce, it is difficult to contextualize our methods and the generalizability of our findings. IPS is a secondary prevention intervention, and prevention science experts have pointed out the need for more prospective cost analyses of prevention interventions across the entire implementation spectrum (from preparation to implementation to sustainment)23. For preventive interventions such as VR-JIT to be adopted and sustained, it is essential to have economic data like those reported herein for budgetary planning by CMHA decision-makers who may wish to implement VR-JIT.
Acknowledgements
This study was funded by the National Institute of Mental Health (R01 MH110524; PI: M.J. Smith), the Eunice Kennedy Shriver National Institute of Child Health and Human Development (F32 HD089586; PI: A. Graham), and the National Institute of Diabetes and Digestive and Kidney Diseases (K01 DK116925; PI: A Graham). We would like to acknowledge the Administration and Staff at Thresholds, Inc. for their help with this study and their continued support in our work together. We would like to acknowledge the members of Thresholds for participating in this study.
Footnotes
Conflict of Interest
Dr. Matthew Smith will receive royalties on sales of an adapted, unpublished (at the time of this submission) version of VR-JIT that will focus on meeting the needs of transition-age youth with autism spectrum disorders. Dr. Smith’s research on the adapted version of VR-JIT is independent of the data reported in this manuscript. No other authors report any conflicts of interest.
References
- 1.Johnson-Kwochka A, Bond G, Becker DR, Drake RE, Greene MA. Prevalence and Quality of Individual Placement and Support (IPS) Supported Employment in the United States. Administration and Policy in Mental Health and Mental Health Services Research. 2017;44(3):311–319. [DOI] [PubMed] [Google Scholar]
- 2.Becker D, Whitley R, Bailey EL, Drake RE. Long-term employment trajectories among participants with severe mental illness in supported employment. Psychiatric services. 2007;58(7):922–928. [DOI] [PubMed] [Google Scholar]
- 3.Bond GR, Drake RE, Campbell K. Effectiveness of individual placement and support supported employment for young adults. Early intervention in psychiatry. 2016;10(4):300–307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Bond GR, Drake RE. Making the case for IPS supported employment. Administration and policy in mental health. 2014;41(1):69–73. [DOI] [PubMed] [Google Scholar]
- 5.Lord SE, McGurk SR, Nicholson J, et al. The potential of technology for enhancing individual placement and support supported employment. Psychiatric rehabilitation journal. 2014;37(2):99–106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Smith MJ, Bell MD, Wright MA, Humm L, Olsen D, Fleming MF. Virtual reality job interview training and 6-month employment outcomes for individuals with substance use disorders seeking employment. Journal of Vocational Rehabilitation. 2016;44:323–332. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Smith MJ, Fleming MF, Wright MA, et al. Job Offers to Individuals With Severe Mental Illness After Participation in Virtual Reality Job Interview Training. Psychiatric services. 2015;66(11):1173–1179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Smith MJ, Fleming MF, Wright MA, et al. Brief report: vocational outcomes for young adults with autism spectrum disorders at six months after virtual reality job interview training. J Autism Dev Disord. 2015;45(10):3364–3369. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Smith MJ, Fleming MF, Wright MA, et al. Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment. Schizophr Res. 2015;166(1–3):86–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Smith MJ, Smith JD, Fleming MF, et al. Enhancing individual placement and support (IPS) - Supported employment: A Type 1 hybrid design randomized controlled trial to evaluate virtual reality job interview training among adults with severe mental illness. Contemp Clin Trials. 2019;77:86–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ramsey A, Lord S, Torrey J, Marsch L, Lardiere M. Paving the Way to Successful Implementation: Identifying Key Barriers to Use of Technology-Based Therapeutic Tools for Behavioral Health Care. The Journal of Behavioral Health Services & Research. 2016;43(1):54–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Sullivan SD, Mauskopf JA, Augustovski F, et al. Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2014;17(1):5–14. [DOI] [PubMed] [Google Scholar]
- 14.Smith MJ, Boteler Humm L, Fleming MF, et al. Virtual Reality Job Interview Training For Veterans with Posttraumatic Stress Disorder. Journal of Vocational Rehabilitation. 2015;42:271–279. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Smith MJ, Ginger EJ, Wright M, et al. Virtual reality job interview training for individuals with psychiatric disabilities. The Journal of nervous and mental disease. 2014;202(9):659–667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Smith MJ, Ginger EJ, Wright K, et al. Virtual reality job interview training in adults with autism spectrum disorder. J Autism Dev Disord. 2014;44(10):2450–2463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Annals of behavioral medicine : a publication of the Society of Behavioral Medicine. 2009;37(2):218–227. [DOI] [PubMed] [Google Scholar]
- 18.Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. . Annual Review of Public Health. 2007;28:413–433. [DOI] [PubMed] [Google Scholar]
- 19.Raghavan R The role of economic evaluation in dissemination and implementation research . : Brownson GC R, & Proctor EK, ed. Dissemination and implementation research in health: Translating science to practice 2nd ed New York, NY: Oxford University Press; 2017:89–106. [Google Scholar]
- 20.Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The cost of implementing new strategies (COINS): A method for mapping implementation resources using the stages of implementation completion. . Children and youth services review. 2014;39(177–182). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Jordan N, Graham AK, Berkel C, Smith JD. Budget impact analysis of preparing to implement the Family Check-Up 4 Health in primary care to reduce pediatric obesity. Prevention Science. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Latimer EA, Bush PW, Becker DR, Drake RE, Bond GR. The cost of high-fidelity supported employment programs for people with severe mental illness. Psychiatric services. 2004;55(4):401–406. [DOI] [PubMed] [Google Scholar]
- 23.Crowley DM, Hill LG, Kuklinski MR, Jones DE. Research priorities for economic analyses of prevention: Current issues and future directions. . Prevention Science. 2014;15(6):789–798. [DOI] [PMC free article] [PubMed] [Google Scholar]