Cost and Return on Investment of a Work-Family Intervention in the Extended Care Industry: Evidence from the Work, Family & Health Network

Abstract

Objective

To estimate the cost and return on investment (ROI) of an intervention targeting work-family conflict (WFC) in the extended care industry.

Methods

Costs to deliver the intervention during a group-randomized controlled trial were estimated, and data on organizational costs—presenteeism, health care costs, voluntary termination, and sick time—were collected from interviews and administrative data. Generalized linear models were used to estimate the intervention’s impact on organizational costs. Combined, these results produced ROI estimates. A cluster-robust confidence interval (CI) was estimated around the ROI estimate.

Results

The per-participant cost of the intervention was $767. The ROI was −1.54 (95 percent CI: −4.31 to 2.18). The intervention was associated with a $668 reduction in health care costs (p < 0.05).

Conclusions

This paper builds upon and expands prior ROI estimation methods to a new setting.

Introduction

Work-family conflict (WFC) is a common yet growing concern for U.S. workers. (1, 2) WFC occurs when workers struggle to manage both their work and personal obligations (3–6) and is associated with poor employee health and well-being. (7–13) In addition to its impact on employee health and well-being, high levels of WFC may also affect the company’s financial bottom line. (14–16) Efforts to reduce WFC and improve employee health may directly affect organizational costs associated with health care utilization (17) and sick leave. (12) Many studies have also shown an association between physical and psychological health risks associated with WFC and decreased work performance (7, 8, 13, 18–21) and increased turnover intentions. (7, 8, 18, 22, 23)

In 2005, the National Institutes of Health and the Centers for Disease Control and Prevention established the Work, Family & Health Network (WFHN) to study ways to reduce WFC. Based on two pilot studies (24, 25) and an extensive literature review, the WFHN developed a biopsychosocial model representing the hypothesized causal pathways between an intervention designed to reduce WFC and improvements in employee health, family health, and the employer’s organizational costs such as expenditures on health care utilization and sick leave, diminished employee productivity, and increased absenteeism, and turnover. (26) The conceptual model links WFC interventions to organizational outcomes primarily through two mechanisms: (1) improved psychological and physical health of employees that results from reduced stress caused by WFC and (2) direct administrative changes that WFC interventions may induce (e.g., changes in sick leave policies that redefine absenteeism in administrative systems). Beyond these two mechanisms, the conceptual framework for the WFHN also acknowledges the need to assess intervention effects on common organizational cost drivers (e.g. health care utilization) regardless of a conceptual link to a specific WFC intervention to better garner organizational support for such interventions. The WFHN used this conceptual model to develop a workplace intervention and design a randomized field experiment to test that intervention. (27) The specific a priori hypotheses relating to employer outcomes and a diagram of the conceptual model are available (see Supplemental Digital Content 1).

The WFHN intervention was designed to reduce WFC by encouraging a focus on results rather than a rigidly structured schedule and to increase manager awareness and support of their employees’ needs to manage work and family responsibilities. (28) The intervention was implemented as a part of a group randomized field experiment designed to assess the efficacy of the intervention in two U.S. companies in different industries: an information technology division of a large Fortune 500 company and a medium-sized for-profit extended care company. (27) The goal of this paper is to estimate the intervention costs and return on investment (ROI) of the WFHN intervention as implemented in the extended care company. (29)

Extended care in the United States is a critical component of the health care system that provides assistance to individuals with chronic illnesses or physical or mental disabilities. (30) In 2010, 15,683 certified nursing home facilities were operating throughout the United States and employed approximately 2 million people, who provide care to about 1.5 million individuals. (31) Employees of extended care facilities often experience considerable WFC brought on by physical and emotional challenges of the work, (32) inflexible work schedules, (33, 34) low pay often requiring individuals to work more than one job, (35) and conflict with management. (36, 37) High rates of employee turnover in the extended care industry have been widely documented, with estimates of annual turnover rates among extended care staff ranging from 53% (31) to well over 100%. (38) Furthermore, research has demonstrated that turnover begets turnover, in that the delivery of lower quality care as a result of frequent turnover often creates additional conflict and staff departure. (39, 40) Presenteeism has also been identified as a major problem among nursing staff, (41–44) who make up a large proportion of the workforce in extended care. Researchers have associated both absenteeism and presenteeism due to sickness with job stress among extended care employees. (45)

To assess the financial merits of an intervention to the company, it is necessary to estimate the reductions in organizational costs attributable to the intervention and compare those to the cost of delivering the intervention. ROI is a standard metric often used in business administration to make such a comparison. (46–50) The randomization design and the availability of longitudinal data from the WFHN intervention provide an opportunity to produce rigorous estimates of reductions in organizational costs attributable to the intervention, a feature lacking in much of the prior literature. (15) Previous studies have estimated the cost (51) and ROI (52) of the WFHN intervention as implemented in a Fortune 500 company. Based on a model-adjusted cost of $690 per participant, the ROI of the intervention was 1.68 (95% confidence interval [CI]: −8.85 to 9.47). The WFHN ROI study was the first workplace ROI study to present a confidence interval around the ROI estimate.

In this study, we applied the methodology used to estimate the intervention cost and ROI in the Fortune 500 company to estimate the intervention cost and ROI in an extended care company. Using this methodology, we compared the intervention cost to the changes in cost incurred by the company in four organizational domains: presenteeism, health care utilization, voluntary termination of employment, and sick time used. With the exception of sick time used, these are the same cost domains used in the ROI estimate for the Fortune 500 company and so our results are comparable to the previous WFHN ROI estimate. The ROI estimate is presented with a confidence interval that accounts for the cluster-randomized study design. (53)

Methods

The START Intervention

The intervention, named START (Support. Transform. Achieve. Results. Today.), was designed to enhance employees’ control over their work, to encourage managers to exhibit behaviors supportive of their employees’ family and personal responsibilities, and to reorient organizational focus on results rather than just time spent at work. (15, 54) START was implemented as part of a group-randomized field experiment of 30 U.S. extended care facilities in New England. Fifteen facilities were randomized to each of two conditions: START and usual practice control. START comprised participatory training sessions for employees and managers and additional activities for managers only (described below). Groups of employee and manager champions of the START initiative at each intervention facility, called steering teams, participated in additional meetings. Intervention materials are available for download at www.workfamilyhealthnetwork.org. Methods, measures, and study design are described in more detail by Bray et al. (27)

Prior to the initiation of the intervention, key management staff at each intervention site were invited to participate in START readiness sessions for the purpose of introducing the intervention. Once the intervention began, employees and managers at intervention sites were invited to attend three face-to-face participatory sessions, delivered by external consultants over 4 months: a team induction and “sludge” eradication session, a culture clinic, and a forum. Because staff in extended care facilities cannot leave the floor at the same time, multiple instances of each session were offered in each site. In addition to the three sessions intended for all employees, steering team members participated in two additional meetings over the course of the intervention. Managers were also invited to two additional meetings not attended by employees, so managers on the steering team participated in up to four additional meetings beyond the three face-to-face participatory sessions. (54) All attendees of the face-to-face participatory sessions were encouraged to participate in outside activities to enhance the impact of the sessions.

In addition to the face-to-face sessions, managers participated in two additional components of the intervention. First, managers completed self-paced computer-based training (CBT) designed to increase manager commitment to family supportive behaviors and enhance understanding of the impact of these behaviors on employee health, as well as the time and energy employees can devote to work tasks. (24, 55) CBT was based on behavioral principles of instructional design, including frequent quizzes, self-pacing, immediate feedback, and mastery required, and was delivered in cTRAIN software. (56) Knowledge gains were assessed via pre- and post-test scores. Transfer of training was supported with two separate 2-week trials of behavioral self-monitoring (BSM) using a supportive-behavior tracking iPhone app designed by the research team. Managers set goals for numbers of supportive behaviors and then used an iPod Touch to observe and record the number of supportive behaviors they provided for employees. Managers were asked to complete BSM daily, were provided with immediate feedback within the app, and received individual and group-level feedback after each trial. (24, 57) Finally, after all intervention activities had concluded, key management staff—many of whom had participated in the initial readiness session—were invited to participate in a START “moving forward” session to encourage sustained culture change.

Study Design and Sample

Eligible employees were group-randomized at the facility level into intervention and usual practice control groups using a biased coin adaptive randomization technique. (27) Trained field interviewers conducted 60-minute face-to-face computer-assisted personal interviews (CAPI) with employees and managers at baseline and at 6, 12, and 18 months post-baseline. A 30-minute health assessment was also conducted at the worksite. (27)

Across the 30 sites, 1,994 subjects were eligible to participate in primary study data collection via CAPI. At baseline, 1,706 unique subjects participated in a CAPI interview. Of those, 238 from the first four facilities to implement the intervention were removed from analysis in the current study because considerable changes were made to the intervention protocol following implementation at those sites. Of the remaining 1,468 subjects, 107 individuals from facilities randomized to the intervention were not captured in session attendance data. Because we were unable to estimate an intervention cost for these subjects, they were dropped from the analysis. Seventy-four subjects were excluded from the ROI analyses for not participating in any follow-up data collection. Of the remaining 1,287 subjects, three were excluded for missing demographic information. Finally, 296 subjects were excluded from the main analyses for having missing data needed to calculate organizational costs. The resulting sample for the ROI analysis consisted of 458 intervention participants and 430 control participants (Figure 1). For the cost analysis, 72 subjects who were randomized to but did not participate in the intervention were excluded, leaving 545 subjects who participated in at least one intervention activity regardless of their demographic data or follow-up status. The ROI analysis followed an intent-to-treat approach; thus, individuals randomized to the intervention group who did not participate were included with intervention costs of $0.

Figure 1.

Study Design, Sample Formation, and Response Rates

Intervention Costs

To calculate the cost of the WFHN intervention, we followed the same costing methodology as in Barbosa et al., (51) which calculated the cost of the WFHN intervention delivered in the Fortune 500 company. We first identified and measured the resources used by the intervention and then applied a monetary value to the identified resources. The result is an estimate of the cost to provide the intervention from the perspective of the company.

To identify resources used to deliver the intervention, we developed a taxonomy of activities (described in detail in Supplemental Digital Content 2). Activities were identified and categorized into five categories: development, customization, start-up, implementation, and research. Development and research activities were excluded from the analysis, as the goal was to produce cost estimates applicable to future adopters of the previously developed START intervention that likely would not include a research component.

Development of the taxonomy of activities was completed primarily through interviews with START coordinators and the intervention team, which comprised contracted facilitators of the participatory sessions and the developers of the CBT and BSM components. The WFHN employed START coordinators to assist with the roll-out and delivery of the intervention. They worked closely with staff responsible for organizing employee shifts to schedule intervention activities so that all participants would have the opportunity to attend. Thus, for the purposes of the cost study, we assumed that in a non-research-driven implementation of START, all coordination activities would be performed by human resources (HR) staff employed by the company.

The START coordinator was primarily responsible for customization and start-up activities, but site-level managers and a vice president (VP) from the company’s corporate office provided input. Implementation activities required some support from the START coordinator and other HR staff, but the majority of the effort during the implementation phase was put forth by participating employees and managers.

After all relevant activities were identified, we gathered data on the labor (e.g., time spent by company staff in intervention-related activities) and non-labor (e.g., contracted services, materials, and space related to the intervention) resources used for each intervention activity using the general methodology described by Barbosa and colleagues. (52) Relevant differences from that methodology are noted here. For implementation activities, we estimated labor time using a combination of study attendance data and semi-structured interviews. For the participatory component, we matched attendance data with records from the START coordinator and facilitators that identified the duration of each session. Because there were multiple instances of each session at each site and participants were not linked to specific instances, we applied the site-level average duration to each participant at that site. The time it took to complete outside activities and the proportion of participants in outside activities was reported by the intervention team and START coordinators in semi-structured interviews. Because the specific employees who participated in outside activities were not identified in the study data, a random set of employees equal to the reported proportion of participants was assumed to have completed the outside activities and had the reported time to complete the outside activities applied to the total time they spent on the intervention. Both CBT and BSM included features to track the amount of time each manager spent using the software. We conducted semi-structured interviews with the intervention team to inform the amount of time that managers and HR spent meeting with researchers for each of the two trials and the amount of time required to prepare BSM feedback for each trial.

Non-labor costs comprised office space and materials for intervention activities (e.g., handouts), travel costs for the corporate VP who attended initial readiness sessions at every participating site, equipment and software, and the contract with the external consultants that developed and facilitated the participatory sessions. The size of the rooms varied, so we assumed a standardized space allocation per person by multiplying 15 square feet by the number of people in each session and meeting. (58) Fifteen iPods were purchased for use by managers participating in BSM. The BSM iPod application needed to be purchased only once; the software could then be deployed on all 15 iPods.

To value the resources used to implement START, we applied a unit cost to each activity using an economic costing approach. In this approach, both accounting costs (e.g., the amount paid for the iPod) and opportunity costs (e.g., manager time spent using the iPods for BSM activities) associated with a given resource are factored into the total cost of the intervention. Although certain resources such as staff time or physical space may not require additional payment, their opportunity costs capture the value of a resource in its most highly valued alternative use. (59)

Consistent with economic theory, we valued labor resources devoted to the intervention using the employee’s total hourly compensation. (60) This was calculated by applying the assumed fringe benefit rate of 30% to the employee’s hourly wage. Employees and managers who participated in study data collection reported either a wage or their personal income in ranges of $5,000 at each wave of data collection. For the latter individuals, we calculated the hourly wage by dividing the midpoint of the range by the estimated total number of hours worked, both at the participating company and at other jobs if the participant indicated having another job outside of the extended care company. We applied as an individual’s wage the median value of all the wages reported at each wave of data collection. We imputed wages for employees or managers who did not provide them using conditional mean imputation. (61) We valued labor time for HR staff and the corporate VP using nationally representative median wages for individuals in similar roles. (62)

We valued non-labor resources using the actual costs incurred by the WFHN, when available, or using published estimates for the costs of similar resources. We valued equipment, software, and contracted services at the actual prices paid to acquire them. Space costs were estimated using the median rental rate for class B office space ($19.40). (63) We converted the annual, per square foot rate into an hourly rate for 15 square feet ($0.03322) and applied it to each person-hour of intervention activities. We valued materials such as handouts and posters using the published prices of these resources purchased in large quantities from a national office supply retailer. Finally, we used the 2011 IRS reimbursement rate of $0.51 per mile to value travel costs for the corporate VP. (64)

For the purpose of the ROI analysis, all costs were also tabulated at the individual level. Costs that were not originally measured at the individual level (e.g., start-up and customization costs) were divided equally among all subjects who participated in at least one intervention activity.

Organizational Costs

We selected the four organizational cost domains based on a priori hypothesized impacts of the intervention on organizational costs. (26, 27) Three of the domains—presenteeism, health care utilization, and voluntary termination—were used in the ROI analysis of the Fortune 500 company. (52) Sick time used was not a relevant organizational cost for the Fortune 500 company because sick leave and vacation were combined in its personal time off (PTO) benefit. Instead, Barbosa and colleagues included the value of unused PTO in a sensitivity analysis. In addition to the organizational cost domains discussed above, we hypothesized that the intervention would also reduce workplace injuries and accidents, which would result in reduced workers’ compensation expenditures. We did not include this domain in the ROI analysis because we could not reasonably apply a cost to each episode of workplace injury or accident. Company administrative data on this matter were not made available to us, and we were unable to find an alternative source for a cost estimate. An examination of workplace injuries and accidents during the WFHN study was conducted elsewhere. (65) Information on employee presenteeism and health care utilization came from CAPI interviews, which also provided the demographic information used in this study. Information on voluntary termination and sick leave came from company administrative data collected by study staff with employee consent.

We defined presenteeism as inadequate performance at work. (66) We measured presenteeism using a 0 to 10 self-rating scale of past 4-week work performance taken from select questions from the World Health Organization’s Health and Work Performance Questionnaire (HPQ). (66) The HPQ has been identified as a strong indicator of work performance, especially presenteeism, and is highly reliable and externally valid. (66–68) To monetize the 0 to 10 scale, we followed the procedure used by Mills and colleagues. (20) We assumed that employees work at 100% capacity for a proportion of their work time equal to 0.1 times their self-rating and at 50% capacity for the remaining time. Thus, organizational costs attributable to presenteeism are equal to half of the employee’s hourly compensation multiplied by the total number of hours working at less than full capacity. We scaled presenteeism costs based on the past 4 weeks to represent the entire 6-month interview recall period.

Interview questions about health care utilization were taken from the Economic Form 90. (69) We asked participants about their use of inpatient, outpatient, and emergency medical services in the past 6 months. Because the actual costs borne by the company for an episode of care were not available, we relied on unit costs from the literature. Costs of emergency room episodes ($869 in 2011 dollars), outpatient visits ($155), and days spent in the hospital ($1,746) were taken from French and Martin. (70) Costs of outpatient visits specifically related to alcohol, drug use, or mental health ($178) were taken from Roebuck and colleagues’ estimate of the cost of outpatient substance abuse treatment. (71) We assumed that the company directly incurred the costs for employee health care utilization.

We identified participants who left employment voluntarily using company administrative data. A recent review of the costs incurred by employers due to nursing staff turnover found that the cost to replace an employee ranged from 0.31 to 1.3 times the employee’s annual salary. (72) We estimated an employee’s annual salary by multiplying hourly compensation by the number of hours worked in the past 12 months (according to company administrative data). To estimate the costs of voluntary termination, we multiplied the salary by the median of the range (0.805) found by Li and Jones (72) for nursing staff and administrators and the minimum of the range (0.31) for staff such as nursing aides.

Sick leave used by an employee was captured in company administrative data. We determined the quantity of sick time used in the past 6 months for each employee. An hour of sick leave was valued at the employee’s hourly compensation rate.

ROI Estimation

We applied a 3% annual discount rate to organizational costs incurred after 12 months. We aggregated organizational costs measured at 6, 12, and 18 months after the baseline interview, resulting in post-intervention cost measures for each participant for each organizational cost domain. We modeled organizational costs in total and by domain. Domain-specific models often included participants not included in the main analysis of the combined domains when a domain-specific organizational cost was not missing. The health care utilization cost model also excluded those who did not receive company health benefits.

We used Gamma regression with a log link function to account for the strong positive skew in organizational costs. (73) Covariates in the regression model included demographics (gender, age, and race/ethnicity), educational attainment, the number of data collection waves missing for reasons other than voluntary termination, the calendar quarter in which the baseline interview was conducted to account for potential seasonal effects, and a set of randomization covariates used for the adaptive randomization design. Randomization covariates included the number of staff at an employee’s facility and the retention rate at an employee’s facility. One additional item used in the randomization design, the randomization block (which identified sites randomized at the same time), was excluded from the model due to correlation with calendar quarter. (27) To account for any residual differences between intervention and control participants at baseline, we included as a covariate baseline costs that correspond to the outcome.

We estimated models with cluster robust standard errors, with the facility serving as the unit of clustering. Because of the high frequency of zeroes in each organizational cost domain, the impact of the intervention on each individual organizational cost domain was estimated with a two-part model. (74, 75) In the two-part model, the probability of having a non-zero cost is estimated using a binary choice (e.g., logit) model. Then, the continuous model is estimated on those cases with a non-zero cost. An individual’s expected cost is then estimated by multiplying the probability of having a non-zero cost by the expected value of the cost derived from the continuous model.

We calculated the ROI by estimating the difference between the intervention effect on organizational costs and the cost of the intervention and dividing that difference by the cost of the intervention. To estimate a confidence interval around the ROI point estimate, we used a cluster-robust bootstrap with 1,000 replications. (52, 53) The bootstrap routine drew facilities—and participants randomized in those facilities—with replacement from among all participating facilities. Each replicate consisted of a number of facilities equal to the number of facilities that participated in each study condition. We estimated the organizational cost model within each bootstrap replicate, which resulted in 1,000 estimates of ROI. The lower and upper bounds of the confidence intervals were thus the 2.5 and 97.5 percentiles, respectively, of the resulting ROI distribution.

Sensitivity Analysis

To ensure that results were not exceedingly influenced by assumptions made during the course of the analysis, we conducted three types of sensitivity analyses. First, we increased and decreased the unit costs applied to each organizational cost domain by 20%, one domain at a time. We then re-estimated the combination and domain-specific regression models with the adjusted dependent variable and baseline organizational cost measure, and re-calculated the intervention effect on organizational cost and ROI.

Second, sensitivity analyses examined the impact of reintroducing observations with missing information by using conditional mean imputation to resolve the item nonresponse. Third, we re-estimated the combined cost model to include the impact of overtime hours worked, which were available from the company administrative data. Overtime represents additional labor costs to the company that are incurred when its usual staffing plan is inadequate. Each hour of overtime worked was valued at 1.5 times the employees’ total compensation.

Fourth, we examined the sensitivity of the ROI estimates to the largest component of intervention costs—the contract for participatory sessions—by reducing the cost of the contract by 50%. Finally, we note that one of the control participants had very high health care utilization costs. The individual had estimated health care costs of $198,000 over the course of the follow-up period, which were more than $100,000 greater than the participant with the second highest costs. Thus, this participant potentially had undue influence over our estimates of health care costs. Despite the high costs for this participant relative to others in the sample, a review of the participant’s underlying health care utilization suggested that this individual had a chronic condition requiring daily care in an outpatient facility. Therefore, we maintained this participant in the analysis sample for the main analyses, but we examined the impact of removing this participant in a sensitivity analysis. All analyses were conducted in Stata 13, and all costs are presented in 2011 U.S. dollars.

Results

Intervention Costs

Table 1 presents START costs broken down by category and type. Customization activities cost $4,449, and labor costs made up 100% of this total; staff spent 146 hours on customization activities. Start-up costs were $361,245. Staff spent 300 hours on start-up activities at a cost of $11,381. Non-labor costs, which totaled nearly $350,000, included the contract for participatory session development and facilitation, equipment, software, and mileage incurred by the corporate VP who traveled to each facility for the START readiness session.

Table 1.

Summary of Intervention Costs

Category	Cost type	Resource(s)	Cost (2011$)
Customization
	Labor	146 hours	4,449
	Non-labor	—	0
	Total	—	4,449
Start-up
	Labor	300 hours	11,381
	Non-labor	Participatory session contract, software licenses, iPods, travel	349,864
	Total	—	361,245
Implementation
	Labor	1,956 hours	47,466
	Non-labor	Office space, handout materials	4,675
	Total	—	52,140
Total cost of START			417,834
Total cost of START per participant (n=545)			767

Implementation costs totaled $52,140. Labor costs made up over 90% of the total, as staff spent 1,956 hours on implementation activities. The majority of implementation labor was for employees attending participatory sessions and engaging in other intervention activities; the remainder of the labor was for HR staff helping to facilitate the intervention. Non-labor costs totaled $4,675 and included office space used for intervention activities and intervention materials, such as handouts and posters. The total cost of START was $417,834. Dividing this cost across all employees and managers who participated in at least one intervention activity yielded a per-person cost of $767. Additional details on labor and non-labor costs of the intervention are available (see Supplemental Digital Content 3).

Return on Investment

Table 2 presents descriptive statistics of the ROI analysis sample are presented in Table 2. The typical employee was a white female around age 40 with some college education but not a college diploma. Approximately 10% of study participants had managerial responsibilities. The intervention group had lower self-rated performance scores than the control group (p < .05).

Table 2.

Descriptive Statistics for ROI Sample

Characteristic	Control (N = 530)	Intervention (N = 458)
Sociodemographic Factors
Male	0.08 (0.28)	0.07 (0.26)
Age	41.18 (12.16)	39.47 (12.50)
Race/Ethnicity
White, non-Hispanic	0.65 (0.48)	0.70 (0.46)
Black, non-Hispanic	0.11 (0.31)	0.14 (0.34)
Other, non-Hispanic	0.08 (0.27)	0.07 (0.25)
Hispanic	0.16 (0.37)	0.10 (0.30)
Educational Attainment
Less than high school diploma	0.05 (0.22)	0.07 (0.25)
High school diploma	0.27 (0.44)	0.31 (0.46)
Some college	0.50 (0.50)	0.48 (0.50)
College graduate	0.18 (0.38)	0.14 (0.35)
Employee Characteristics
Costs to employer at baseline†,‡	2,690.43 (2,766.79)	2,661.49 (2,724.96)
Manager	0.12 (0.32)	0.10 (0.31)
Hourly compensation†, §	23.24 (8.64)	22.68 (8.82)
Hours worked per week, past 4 weeks	39.18 (10.54)	38.40 (11.04)
Overall performance self-rating, past 4 weeks*, #	8.56 (1.18)	8.37 (1.13)
Hours of sick time used, past 6 months	19.77 (13.15)	21.01 (12.78)
Health Care Utilization||
Nights in hospital, past 6 months	0.21 (1.11)	0.12 (1.09)
Visits to emergency department/urgent care facility, past 6 months	0.32 (0.92)	0.23 (0.60)
Visits to outpatient provider, past 6 months	1.59 (5.40)	1.27 (3.22)
Instances of outpatient substance abuse or mental health care, past 6 months	0.37 (2.14)	0.39 (2.11)

^*p < .05,

^**p < .01,

^***p < .001, tests are independent sample t-tests or Chi-square tests (race and educational attainment) adjusted for clustering within facility.

Note. Means (standard deviations) based on respondent self-report on 6 months prior to baseline interview.

^ⱡExpressed in 2011 U.S. dollars.

^‡Includes cost incurred by employer based on presenteeism, health care utilization, and sick time used.

^§Includes wage and fringe benefit rate.

^#Using a 0 to 10 scale, how would you rate your overall performance on the days you worked during the past 4 weeks?

^||Unconditional averages.

An analysis of unadjusted organizational costs across the four domains revealed no significant differences between study groups for either the 6 months before or the 18 months after the baseline interview. When adding up all cost domains, average costs between groups differed by less than $30 in both time periods. Costs due to employee presenteeism made up approximately one-half of total organizational costs in both periods. Costs due to voluntary termination made up about one-quarter of total organization costs in the post-intervention period, but were not measured in the pre-intervention period. Health care utilization costs and costs due to sick time used made up the remainder of total organizational costs.

Tables 3 and 4 present results from the multivariate models of organizational costs. Table 3 presents the full model of organizational costs, where the dependent variable is the sum of all four organizational cost domains. The model shows that baseline organizational costs, race/ethnicity, and educational attainment are significant predictors of organizational costs following the intervention. From this model, we estimated that organizational costs were approximately $380 higher for intervention participants than control participants in the 18 months after the baseline interview ($10,345 vs. $9,965, respectively), although this difference is not statistically significant.

Table 3.

Multivariable Model of Organizational Costs

Covariate†	Organizational costs
START participant	380.39 (872.12)
Baseline costs to employer	1.07*** (0.27)
Male	−511.33 (938.96)
Age	16.90 (36.40)
Black, non-Hispanic#	−1,869.57 (1,051.07)
Non-white, non-black, non-Hispanic#	−1,343.85 (965.74)
Hispanic#	−1,990.11* (865.45)
No high school diploma||	−4,740.49*** (846.13)
High school diploma||	−2,938.56*** (733.73)
College graduate||	3,435.21** (1,261.89)
Observations	988
Adjusted Means‡
Control	9,965.18 (787.67)
START	10,345.57 (634.89)
ROI§
Point estimate	−1.54
Confidence interval	−4.31 to 2.18

^*p < .05,

^**p < .01,

^***p < .001

Note. Dependent variables expressed in 2011 U.S. dollars.

^†Marginal effects from generalized gamma regressions with log link and cluster-robust standard errors in parentheses. Models also control for number of missed data collection waves and randomization factors.

^‡Adjusted means (standard errors) computed using recycled predictions.

^§ROI calculated as $\frac{\mathrm{\Delta }E-C}{C}$, where ΔE is the marginal effect of START on organizational costs, and C is the average intervention cost in the ROI sample ($706). Confidence intervals calculated using a cluster-robust bootstrap routine with 1,000 replications (Davison & Hinkley, 1997).

^#Relative to white, non-Hispanic.

^||Relative to “some college.”

Table 4.

Model-Adjusted Intervention Effects on Organizational Cost Domains

	Presenteeism costs	Health care costs‡	Costs due to voluntary termination	Costs due to sick time used
Model-adjusted intervention effect†	360.95 (209.45)	−668.44* (315.08)	919.56 (629.84)	38.11 (29.84)
ROI§	−1.51	−0.05	−2.30	−1.05
Confidence interval	−2.08 to −0.63	−1.29 to 1.83	−4.14 to 0.08	−1.20 to −0.91
N	1,253	727	1,249	1,009

^*p < .05

Note. Dependent variables expressed in 2011 U.S. dollars; models include all observations for which the dependent variable is not imputed.

^†Marginal effects from two-part models with cluster-robust standard errors in parentheses. Models also control for age, gender, race/ethnicity, educational attainment, analogous baseline costs (except voluntary termination, for which no baseline costs exist), number of missed data collection waves, calendar quarter at baseline interview, and randomization factors.

^§ROI calculated as $\frac{\mathrm{\Delta }E-C}{C}$, where Δ is the marginal effect of START on organizational costs, and C is the average intervention cost in the ROI sample ($706).

^‡Model includes only those who receive company health benefits.

The average intervention cost among participants in the ROI study that were randomized to the intervention was $706. This value differs from the $767 per-participant cost estimate from the cost analysis because non-participants were excluded from the cost analysis, whereas the ROI followed an intention-to-treat approach and so included those individuals. Comparing the return of −$380 to the company’s investment of $706 per participant yielded an ROI point estimate of −1.54 (95% confidence interval: −4.31 to 2.18).

Table 4 examines the intervention effect on each of the four domains independent from one another. Model-adjusted estimates of the intervention effect indicate that health care costs were $668 lower, on average, for intervention participants than control participants over the same period (p < .05). Results for the other three domains indicate that costs for intervention participants were higher, but the differences were not statistically significant. Estimated ROI ranged from −2.30 for termination costs to −0.05 for health care costs. The confidence intervals for two of the domains—presenteeism and sick time costs—were strictly negative. Thus, we can conclude with 95% confidence that the intervention yielded a negative return for those two domains independently.

Sensitivity Analysis

Results were not sensitive to changes in the unit costs applied to organizational outcomes by 20% in either direction. For the main model, the ROI point estimate varied between −1.69 and −1.38, with the greatest variation occurring when health care and voluntary termination unit costs were changed. Re-estimation of the domain-specific models using adjusted unit costs similarly had little impact on the results; however, the findings in Table 4 of significant intervention effects on health care costs were robust to the change in unit costs.

When participants with imputed organizational costs were added to the main model, the ROI barely decreased from −1.54 to −1.59. Domain-specific models similarly were not affected by the introduction of participants with imputed organizational costs. When overtime costs were added to the combined model as a fifth cost domain, the ROI decreased from −1.54 to −1.67. When the cost of the participatory session contract was decreased by 50%, the ROI for all organizational cost domains decreased from −1.54 to −1.97. For the domain-specific models, the ROI point estimate remained negative for the presenteeism, voluntary termination, and sick time domains. However, the ROI for the health care utilization domain increased from −0.05 to 0.71. Finally, when the participant with unusually high health care costs was removed, the estimated intervention effect on health care costs decreased from $668 (p < .05) to $344 (not significant).

Discussion

This study estimated the cost and ROI from the perspective of the employer of a workplace intervention designed to reduce employee WFC as implemented in an extended care company in the United States. We focused on the employer perspective, rather than a broader societal perspective, because employers are most likely to invest in workplace interventions if there is a tangible benefit to the company. We estimated the total cost of the intervention to be $417,834, or $767 per participant. We estimated ROI by comparing the cost of the intervention to the impact of the intervention on four organizational cost domains: presenteeism, health care utilization, voluntary termination, and absenteeism due to sick time used. We quantified these cost domains using a portion of employee compensation (wage plus fringe benefits), with the exception of health care utilization, which we valued using unit costs from the health economic literature. (70, 71) After analyzing each domain independently, we found that health care costs were $668 lower, on average, for intervention participants than for control participants (p < .05). Despite this, when all four domains were combined, the company incurred $380 higher organizational costs, on average, for intervention participants compared with control participants. This resulted in an ROI of −1.54 (95% CI: −4.31 to 2.18).

Relative to the Fortune 500 company, the total cost of the intervention at the extended care company was almost $300,000 lower. Although customization and start-up costs were similar between the two companies, they made up a much larger proportion of total costs at the extended care company (88% versus 47%). The largest single component of the intervention cost was the contract for the development and delivery of the participatory sessions. The value of this contract was negotiated by the WFHN and the external consultants who delivered the intervention specifically for the two companies participating in the study. It is likely that, if the START intervention was customized for and implemented in another company, the costs to do so would be lower than they were for the extended care company that participated in the study. As indicated by our sensitivity analysis, although the reduction in the cost of the contract resulted in a positive ROI of the intervention in terms of health care costs, the point estimate under the sensitivity analysis remains well inside the original confidence interval, suggesting a minimal effect of the sensitivity analysis on the ROI estimate.

Differences in implementation costs between the two companies were considerable and were driven by two factors. First, because the average Fortune 500 company employee received higher compensation than the average extended care employee, the per-hour cost on intervention activities was considerably higher for the Fortune 500 company than for the extended care company. Second, the Fortune 500 company had more intervention participants, and thus more hours were spent on intervention activities there than at the extended care company. This may be due to the nature of the work done in the two companies. One of the challenges of conducting a workplace intervention with activities taking place during work hours in an extended care facility is that some employees must remain on the floor at all times. Although intervention coordinators attempted to schedule sessions so that everyone would have a chance to attend, study attendance data suggest that intervention participants in the extended care company attended fewer sessions than participants in the Fortune 500 company as a percentage of sessions available to them (61% vs. 72%). Despite differences in total costs, the per-participant costs were similar across companies because the three cost domains (start-up, customization, and implementation) are accounted for in the calculation.

In terms of ROI, the organizational cost domain with the most influence was voluntary termination. Costs incurred due to intervention participants’ voluntary termination exceeded costs due to control participants’ voluntary termination by $920. Intervention participants who voluntarily left employment at the company received lower compensation, on average, than their colleagues who did not leave voluntarily; however, a higher percentage of intervention participants (17%) than control participants (15%) voluntarily left employment. The difference between intervention and control groups in post-intervention costs due to turnover was not statistically significant. A longitudinal assessment of voluntary turnover being conducted concurrently also indicates that intervention participants were more likely to leave their job voluntarily, but that the impact was not statistically significant.

The lack of an effect of the intervention on voluntary turnover in the extended care company contrasts with the experience in the Fortune 500 company, where the intervention reduced voluntary turnover. (76) These dissimilar findings provide one explanation for why the ROI was positive (but not significant) in the Fortune 500 company but negative (and also insignificant) in the extended care company. In the Fortune 500 company, the unadjusted mean cost of voluntary turnover per employee was over $4,000 lower in the intervention group than in the control group. (52) In the extended care company, we found that mean voluntary turnover costs were $920 higher for the intervention group than for the control group. It is possible that due to the nature of their work, extended care employees did not respond as favorably to, or were unable to take full advantage of, the increased flexibility encouraged by the intervention as much as the employees in the Fortune 500 company.

Previous research has examined the impact of interventions aimed at reducing turnover rates in extended care facilities. One study examined the impact of a workforce development program that focused on staff education and supervisory skills training for those managing staff. After removing two “anomalous participating sites” from analysis, the authors found that the turnover rates were lower (p < .10) at intervention sites relative to control sites. (77) In another study, the authors found that inserting a “retention specialist” into nursing homes had a significant impact on reducing turnover. (78) It is difficult to compare our findings in terms of turnover to these other studies. First, neither study placed a dollar value on turnover. Second, in the case of the workforce development program study, we did not exclude any facilities due to abnormal events that may have impacted turnover rates. It should be noted that the study reported null findings prior to the exclusion of the anomalous sites. Third, both interventions discussed in these studies were focused primarily on reducing turnover. In contrast, the WFHN conceptual model (see Supplemental Digital Content 1) listed employee retention as a distal outcome associated with reductions in WFC.

The other organizational cost domain with considerable influence on ROI was health care utilization. Health care utilization costs averaged $668 less for intervention participants than for control participants (p < 0.05) over the 18 months following the baseline interview. A review of 22 workplace wellness studies conducted in various industries found health care costs averaged $358 less per year for intervention participants than for control participants. (46) The interventions in this review differ in that they were less costly than START, their participants were typically followed for a longer period, and they were intended to improve workplace wellness rather than encourage organizational change. However, the health care cost savings from our study fall into the range of difference-in-difference estimates listed in the review’s appendix. To our knowledge, no study has examined health care savings from a workplace wellness intervention conducted in a health care facility.

It is important to note that a decrease in health care costs in the short term is not necessarily the most desirable outcome. Hypothesized improvements in employee health due to reduced WFC may be realized by way of utilization of appropriate outpatient preventive services. Among WFHN participants, we saw a decline in average outpatient visits per 6-month period, for both intervention and control participants from the pre- to post-intervention period. Although not significant for either group, the decline was larger for control participants.

This study has some important limitations. First, we relied on self-reported data for the presenteeism and health care utilization cost domains. Although the instruments used to collect the data—the HPQ and the Economic Form 90, respectively—are validated and commonly used, administrative data would have been preferable. Unfortunately, work performance data and medical claims data were not available for our analysis. Second, we assumed that health care costs were incurred on a dollar for dollar basis by the company, because the relationship between health care utilization and health care premiums paid by the employer was proprietary information not released to the WFHN. Third, even though our analysis was based on a randomized control field experiment, residual imbalance existed in employee self-rating at baseline. This imbalance was addressed by using multivariable regression models to predict organizational costs. Fourth, the relatively short follow-up period limits the time for intervention benefits to accrue; a larger return may have been measurable given a longer follow-up period. Finally, the main analysis excluded a number of participants for whom we would have had to rely on imputation to estimate organizational costs, which may have introduced selection bias into our results. Participants excluded from the main analysis due to imputation were younger, less likely to be in a managerial position, were paid less, and had lower intervention and baseline organizational costs than those who were included. We addressed this by running a sensitivity analysis in which we included all of these participants with their imputed values and found little difference in the results.

Despite these limitations, our study serves an important purpose. We applied the cost and ROI methodology used to evaluate the first WFHN company partner (51) to a second company with very different characteristics. We expanded the methodology by considering the intervention effect on organizational costs on each domain independently, as well as combined. Although workplace interventions in extended care facilities have been studied, ours is the first study to have examined the impact of a workplace intervention on disparate cost domains, such as presenteeism and health care utilization in the industry. Future research should consider the costs and benefits of an intervention like START from a societal perspective. Efforts should also be made to identify the extent to which facility-specific factors may moderate the impact of workplace interventions in extended care and other similar industries. In addition, in light of the lower levels of participation at the extended care company relative to the Fortune 500 company, research should be devoted to estimating a dose effect of START and similar interventions.