A Scoping Review of Economic Evaluations of Workplace Wellness Programs

Abstract

Objective

Debates about the effectiveness of workplace wellness programs (WWPs) call for a review of the evidence for return on investment (ROI) of WWPs. We examined literature on the heterogeneity in methods used in the ROI of WWPs to show how this heterogeneity may affect conclusions and inferences about ROI.

Methods

We conducted a scoping review using systematic review methods and adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We reviewed PubMed, EconLit, Proquest Central, and Scopus databases for published articles. We included articles that (1) were published before December 20, 2019, when our last search was conducted, and (2) met our inclusion criteria that were based on target population, target intervention, evaluation method, and ROI as the main outcome.

Results

We identified 47 peer-reviewed articles from the selected databases that met our inclusion criteria. We explored the effect of study characteristics on ROI estimates. Thirty-one articles had ROI measures. Studies with costs of presenteeism had the lowest ROI estimates compared with other cost combinations associated with health care and absenteeism. Studies with components of disease management produced higher ROI than programs with components of wellness. We found a positive relationship between ROI and program length and a negative relationship between ROI and conflict of interest. Evaluations in small companies (≤500 employees) were associated with lower ROI estimates than evaluations in large companies (>500 employees). Studies with lower reporting quality scores, including studies that were missing information on statistical inference, had lower ROI estimates. Higher methodologic quality was associated with lower ROI estimates.

Conclusion

This review provides recommendations that can improve the methodologic quality of studies to validate the ROI and public health effects of WWPs.

Workplace wellness programs (WWPs) are employer-sponsored initiatives to promote healthy behaviors among employees. Public and private sectors have used workplace interventions to improve employee health and productivity for decades, ^1,2 with a focus on worker productivity. ³ Studies suggest that WWPs improve employee health by reducing modifiable risk factors, such as physical inactivity, tobacco use, unhealthy eating habits, obesity, high blood pressure, high blood glucose, and high cholesterol. ^{4 -8} These improvements in employee health are thought to increase health-related productivity by reducing absenteeism and presenteeism. ^{4 -6,9 -12} Since their inception, WWPs have expanded to include initiatives such as health promotion, prevention, disease management, and occupational health and safety. ^{1,13 -16}

Today, WWPs are often implemented to improve employee health ⁴ and address rising health care costs, particularly in the United States and other Western societies. ¹⁷ As a result, many economic evaluations of WWPs have focused on return on investment (ROI). ^14,18 Improvements in employee well-being and performance could decrease the organizational costs associated with health care use, high turnover rates, and health-related productivity losses. ^{5 -8,10,18 -20} In the past decade, however, new criticism of the WWP ROI literature argues that the expected cost savings may not materialize, citing a lack of reliable evidence on WWP effectiveness in delivering cost savings or positive ROI. ^21,22

In a series of reviews from 1991 through 2011, Pelletier ^{23 -27} concluded that WWPs would improve health and reduce health care costs if properly implemented. In 2013, Kaspin et al ²⁸ identified program characteristics that were associated with improved economic outcomes. Baicker et al ¹⁸ also concluded that WWPs would reduce health care costs, but this review was criticized in subsequent editorials for inadequate reliable data. ^22,29,30 Lerner et al ³¹ conducted a review using more stringent inclusion criteria than previous reviews and found that only 10 studies were rigorous enough to be evidentiary. Despite finding that 8 of 10 studies showed a positive economic effect, Lerner et al concluded that evidence was insufficient to draw a conclusion on the economic effect of WWPs. In 2014, Baxter et al ³² found that methodologic quality of a study and study design were important determinants for WWP evaluation results. McCoy et al ³³ also noted how business size and type of wellness program could affect the decision to adopt the program and the program’s effectiveness.

To our knowledge, no review has analyzed the heterogeneity of WWP evaluations and their effect on ROI findings. The objective of our review was to describe the effects of study heterogeneity that are not commonly noted in the literature. These effects include inconsistent formulation of ROI, variation in outcomes evaluated, program targets, evaluation length, publication year, conflict of interest, and the lack of statistical inference information for ROI estimates. In addition, we examined the underrepresentation of small businesses in the WWP ROI literature and which methodologic challenges affect ROI findings.

Methods

This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. ³⁴ The lead author (N.U.) primarily performed many aspects of the review. Thus, this review is best classified as a scoping review ³⁵ and was, therefore, not registered at PROSPERO. We identified peer-reviewed articles in PubMed, EconLit, Proquest Central, and Scopus. Our search included all articles that met our inclusion criteria and were published before the date of our final search conducted on December 20, 2019.

The target populations for our search were workplaces, employees, worksites, or workers. The target interventions were wellness, health, health promotion, health prevention, or well-being. The target evaluation method was economic evaluation, including cost benefit, cost effectiveness, cost analysis, economic evaluation, economic analysis, or economic assessments. The outcome was ROI. A detailed list of the search terms by database is available from the authors upon request. We placed no restrictions on publication dates. We excluded publications that were not primary studies (eg, reviews, simulations, or meta-analyses) or not in English. Because “wellness” is often an umbrella term that includes components focused on lifestyle or behavior-related risk reduction and chronic disease management, we included workplace wellness, health promotion, and disease management programs.

We first reviewed articles’ titles and abstracts to determine relevance and fit for this review. Next, we conducted a full-text review of articles deemed relevant. Then, we scanned the reference lists of all identified publications, including those from systematic reviews, meta-analyses, and other reviews, to identify other relevant citations. We excluded articles that evaluated government-sponsored WWPs to maintain a focus on private employer–relevant information. We included only peer-reviewed articles to analyze the validity of recent critiques of WWP ROI studies. ^21,22,29

The lead author (N.U.) read and extracted data from all articles, with targeted assistance from coauthors (G.W., J.B., D.B.). For all articles, we extracted the ROI estimate as reported, regardless of how it was defined. Reported ROI measures included (1) true ROI, expressed as either a ratio or percentage and measured as the ratio of net benefit (the difference between benefits and program costs) to program cost, which has a threshold for positive ROI of 0 ³⁶ ; (2) the benefit-to-cost ratio, which has a threshold for positive ROI of 1; or (3) net benefit with positive ROI as savings exceeding program costs. We recalculated ROI using information from each article to consistently define ROI as the net benefit-to-cost ratio. If the study did not report ROI as its finding but reported program costs and benefits, then we calculated ROI using net benefit-to-cost ratio. To increase consistency across studies, the recalculated ROI is the primary outcome of interest in our review.

We did not discount or adjust monetized values for inflation to have standard valuation across studies. Discounting would require extracting annual flow information for costs and savings, which was not possible for all articles in this analysis. We did not adjust for inflation across studies for 2 reasons. First, most of the included studies already adjusted for inflation when necessary. Second, the recalculated ROI used in this analysis is a ratio of net present values, and inflation adjustments would affect the numerator and denominator equally and so are not needed.

We extracted information from the included articles using a methodologic rigor rubric that we generated based on guidance from 5 checklists. ^{37 -41} Our rubric contained checklist domains of article characteristics, reporting, internal validity, external validity, and power (Table 1; detailed rubric available upon request). We used the domains of reporting, internal validity, external validity, and power to score the quality of the articles. Each domain included items that were scored 0 or 1 based on the presence (1) or absence (2) of information in the included articles. Reporting had 11 items, 3 of which were averaged to compose the score for study sample in this domain, resulting in a total score that ranged from 0 to 8. Internal validity had 13 items, 2 of which were averaged to compose the scores for the appropriate assessment of the outcome measures item and 3 of which were averaged to compose the score for the appropriate cost measures and values item, resulting in a total score that ranged from 0 to 8. External validity and power each had 1 item with a raw score of 0 or 1 based on the presence (1) or absence (0) of each in the articles. The lead author (N.U.) scored all articles based on the rubric domains. The total scores for each domain were summed for a quality index score that ranged from 0 to 18. The second author (G.W.) independently scored 3 articles to calibrate the scoring of the lead author. The lead author discussed any scoring uncertainties with the coauthors to achieve consensus in scoring.

Table 1.

Rubric used to assess rigor and quality of articles that were included in a scoping review, evaluated workplace wellness programs, included ROI measures, and were published before December 20, 2019

Domain	Checklist items	Article information or score
Article
1.1	Who is the author?	Open-ended
1.2	What is the title of the article?	Open-ended
1.3	What year was the study conducted/published?	Open-ended
1.4	If published, in which journal was it published?	Open-ended
1.5	What is evaluated: wellness, disease management, or both?	Open-ended
1.6	What is the country of study?	Open-ended
1.7	What is the industry/company?	Open-ended
1.8	The size of the worksite in which wellness program took place	Small/large
1.8.1	Number of participants and nonparticipants	Open-ended
1.9	What academic department or research center conducted the study?	Open-ended
1.10	What is the funding agency?	Open-ended
1.11	Do authors have any conflicts of interest?	Yes/no
Reporting
2.1	Objectives of the study	0 or 1
2.2	Intervention(s)	0 or 1
2.3	Study sample (scored as average of sub-questions 2.3.1, 2.3.2, 2.3.3, with each sub-question carrying equal weight in contributing to 2.3 score)	0 to 1
2.3.1	Study population	0 or 1
2.3.2	Inclusion/exclusion criteria	0 or 1
2.3.3	Analysis sample	0 or 1
2.4	Type of the economic analysis	0 or 1
2.5	Main outcome(s)	0 or 1
2.6	Intervention costs	0 or 1
2.7	Main finding(s)	0 or 1
2.8	Statistical inference information about the main outcomes (interquartile change, standard errors, standard deviations, confidence interval, P values)	0 or 1
Internal validity
3.1	Were the main outcome measures clearly described?	0 or 1
3.2	Were study subjects randomized to intervention groups? (0: observational case or cohort without control group; 0.25: observational case with control group; 0.50: observational cohort with control group; 0.75: quasi-experimental; 1.00: randomized)	0, 0.25, 0.50, 0.75, or 1.0
3.3	Was the method used to assess the main outcomes appropriate? (Scored as average of sub-questions 3.3.1, 3.3.2, with each sub-question carrying equal weight in contributing to 3.3 score)	0 to 1
3.3.1	When not randomized, was any method used to deal with selection bias?	0 or 1
3.3.2	Appropriate method for outcome estimates	0 or 1
3.4	Were the costs measured and valued appropriately? (Scored as average of sub-questions 3.4.1, 3.4.2, 3.4.3, with each sub-question carrying equal weight in contributing to 3.4 score)	0 to 1
3.4.1	Were the intervention cost measures that were used clearly described?	0 or 1
3.4.2	Were monetized outcomes and intervention costs discounted when the timeframe was >1 year? If not discounted, was the reason explained?	0 or 1
3.4.3	Were costs adjusted to real values/inflation?	0 or 1
3.5	Were control and treated groups (or cohorts) recruited from the same population?	0 or 1
3.6	Were control and treated groups (or cohorts) recruited from the same period?	0 or 1
3.7	Did the analyses adjust for different follow-up lengths in cohort or case-control studies? Was the period between intervention and outcome the same for cases and controls?	0 or 1
3.8	Were attrition/losses from follow-ups taken into account?	0 or 1
External validity
4.1	Did participants in the study represent the entire population from which they were recruited?	0 or 1
Power
5.1	Did study have sufficient power to detect a clinically important effect? (ie, P values)	0 or 1
Total quality index score	Sum the values of reporting (2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8), internal validity (3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8), external validity (4.1), and power (5.1) items.	0-18

To determine if ROI results differed based on the targeted outcome of the WWP, we classified evaluated programs into disease management, wellness, or a combination of the 2. A disease management program targeted diagnosable diseases (ie, asthma, diabetes). A wellness program targeted health risks or behaviors (ie, smoking, exercise, nutrition). Because “wellness” is not precisely defined in the literature or in practice, we attempted to control for the nature of the wellness program being evaluated.

Furthermore, to analyze whether ROI results varied based on how benefits (program outcomes) were defined, we categorized studies based on the cost components included in the ROI analysis, such as costs of health care, absenteeism, or presenteeism. Health care included pharmaceutical claims and medical claims of inpatient, outpatient, and emergency department visits. Absenteeism included lost workdays, sickness absence days, disability days, or time away from work. Presenteeism included measures for productivity loss at work.

We reviewed articles for potential factors that might affect the ROI findings. We categorized the study publication year into 4 groups that attempted to balance sample size and date ranges: before 2000, 2000-2010, 2011-2014, and after 2014. Because short- and long-term effects may differ, we used an indicator variable for studies with follow-up of ≥3 years. We chose the 3-year study duration to roughly balance sample size across categories. We identified conflicts of interest using information on authors’ employment and study funding, setting the indicator equal to 1 if at least 1 author was employed by the funding or program host institution. We classified company size as small (≤500 employees) or large (>500 employees). Size was the only company characteristic that we used in the analysis because of a lack of other information across studies.

We used ordinary least squares regression to examine significant differences in ROI across study characteristics and not to imply any causal inference. Models 1 and 2 included all studies, including those with ROI values as low as –11.61 ⁴² and –6.66, ⁴³ which are extreme outliers ⁴⁴ relative to the interquartile range of ROI estimates. Models 3 and 4 excluded these outliers. Models 1 and 3 included quality index scores as a predictor of ROI, whereas Models 2 and 4 included specific domains that make up the quality index scores to provide more detail on the effects of domains of quality on ROI. For all 4 ordinary least squares regressions, we considered P < .05 to be significant. Because few source articles included the standard error or any statistical inference information for the ROI estimate, we did not adjust the regression for sampling variation within each study; as such, it should not be considered a meta-regression.

Results

We identified and selected for further review 466 unduplicated articles (Figure). Of these, 78 articles met the inclusion criteria after title and abstract screening, 33 of which were included after full-text review. We also included 11 articles from the publications’ reference lists, and we identified 3 articles in an updated search conducted on December 20, 2019, resulting in 47 unique publications included in our review.

Figure.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart showing the process for article inclusion in a scoping review of articles that evaluated workplace wellness programs, included measures of return on investment, and were published before December 20, 2019.

Of the 47 included articles, 30 provided an ROI estimate as a main outcome, 2 of which did not provide any information on the ROI formula and did not have sufficient program cost information for us to recalculate the authors’ ROI findings (Table 2). Thus, we excluded these 2 articles from subsequent analysis. We found 3 articles that had not reported ROI as their outcome but did include program benefits and costs in sufficient detail to allow ROI to be calculated. Therefore, we added these 3 articles to our subsequent analysis. The final sample for the ROI regression analyses included 31 articles with recalculated ROI outcomes. Of these 31 articles, 24 had a positive recalculated ROI and 7 had a negative recalculated ROI.

Table 2.

Summary characteristics in a scoping review of articles that evaluated workplace wellness programs, included measures of return on investment, and were published before December 20, 2019 (n = 47)

Article (publication year)	Journal	Program content	Program duration, y	Country	Company/industry	Size of company	ROI formula in article (unit)	Provided ROI (statistical inference)	Recalculated ROI	Index
Randomized studies (n = 16)
Barbosa et al (2015) 45	Journal of Occupational and Environmental Medicine (JOEM)	Wellness	1.5	United States	Information technology firm	Large	(Benefit – Cost)/Cost (ratio)	1.68 (–8.85 to 9.47)	1.68	18.00
Jones et al (2019) 46	Quarterly Journal of Economics	Wellness	2.5	United States	University of Illinois	Large	Benefit/Cost	2.61	1.61	16.34
Oude Hengel et al (2014) 47	American Journal of Industrial Medicine	Wellness	1	Netherlands	6 construction companies (house, commercial, or industrial building)	NA	[(Benefit – Cost)/Cost] x 100 (percentage)	543	5.43	15.67
van Dongen et al (2016) 48	JOEM	Wellness	1	Netherlands	2 Dutch government institutes	Large	[(Benefit – Cost)/Cost] x 100 (percentage)	–351 (–919 to 210)	–3.52	15.67
van Dongen et al (2017) 43	Health Education Research	Wellness	1	Netherlands	Financial service provider	Large	[(Benefit – Cost)/Cost] x 100 (percentage)	–666 (–1266 to –20)	–6.66	15.50
Song and Baicker (2019) 49	Journal of the American Medical Association	Wellness	1.5	United States	A warehouse retail company	Large	NA	NA	NA	15.34
Groeneveld et al (2011) 50	JOEM	Wellness	1	Netherlands	Construction industry—15 plants	Large	NA	NA	–0.49	15.33
van Dongen et al (2013) 51	JOEM	Wellness	2	Netherlands	2 Dutch academic hospitals	Large	Benefit/Cost (ratio)	–2.21	–3.83	15.00
Noben et al (2015) 52	International Journal of Occupational Medicine and Environmental Health	Wellness	0.5	Netherlands	A Dutch academic hospital	Large	Benefit/Cost (ratio)	11	6.13	14.67
Thiart et al (2016) 53	Sleep	Disease management	0.5	Germany	School teachers	NA	[(Benefit – Cost)/Cost] x 100 (percentage)	208.81 (–296 to 744)	2.09	14.33
Proper et al (2004) 54	Scandinavian Journal of Work, Environment, and Health	Wellness	3	Netherlands	3 municipal services of a Dutch town	Large	NA	NA	0.19	14.00
van Holland et al (2018) 42	Journal of Occupational Rehabilitation	Wellness	3	Netherlands	Dutch meat processing company	Large	[(Benefit – Cost)/Cost] x 100 (percentage)	–1160 (−1415 to −918)	–11.61	14.00
Robroek et al (2012) 55	Health Education Research	Wellness	2	Netherlands	2 health care organizations, 2 commercial services, 2 government branches	Large	NA	NA	NA	13.67
Steinberg et al (2015) 56	JOEM	Wellness	1	United States	Aetna	Large	NA	NA	NA	13.00
Meenan et al (2010) 4	JOEM	Wellness	3	United States	Oahu, Hawaii, hotel workers	Large	NA	NA	–0.95	12.67
Milani and Lavie (2009) 57	American Journal of Cardiology	Disease management	1	United States	A single employer, 2 locations	NA	NA	6	NA	9.33
Quasi-experimental studies (n = 4)
Grossmeier et al (2013) 58	JOEM	Disease management and wellness	3	United States	BP America US employees	Large	Benefit/Cost (ratio)	3	2	13.08
Ozminkowski et al (1999) 59	American Journal of Health Promotion (AJHP)	Wellness	3	United States	Citibank	Large	Benefit/Cost (ratio)	4.73	3.73	12.58
Musich et al (2015) 20	AJHP	Disease management and wellness	4	United States	Dell	Large	Benefit/Cost (ratio)	2.48	1.48	12.58
Serxner et al (2012) 60	AJHP	Disease management and wellness	5	United States	A large financial services corporation	Large	Benefit/Cost (ratio)	2.45	1.45	12.08
Observational cohort studies with control group (n = 3)
Liu et al (2013) 61	Population Health Management	Disease management and wellness	5	United States	PepsiCo	Large	NA	NA	NA	13.34
Liu et al (2013) 62	Inquiry	Disease management and wellness	6	United States	PepsiCo	Large	NA	NA	NA	13.34
Light et al (2015) 63	JOEM	Disease management and wellness	5	United States	Price Cooper, Golub Corporation (large retail grocery chain)	Large	Benefit/Cost (ratio)	4.33	3.33	13
Observational case studies with comparison group (n = 16)
Nyman et al (2012) 5	Medical Care	Disease management and wellness	5	United States	University of Minnesota	Large	Benefit/Cost (ratio)	1.76	0.76	12.58
Kapinos et al (2015) 64	JOEM	Disease management and wellness	9	United States	A large firm	Large	NA	NA	NA	12.25
Jutkowitz et al (2015) 65	JOEM	Disease management and wellness	6	United States	University of Minnesota	Large	NA	NA	NA	12.08
Michaud et al (2016) 66	JOEM	Disease management and wellness	3	United States	University of Minnesota	Large	NA	NA	NA	12
Naydeck et al (2008) 67	JOEM	Wellness	4	United States	Highmark	Large	Benefit/Cost (ratio)	1.65	0.65	11.75
Nyman et al (2013) 68	JOEM	Disease management and wellness	5	United States	University of Minnesota	Large	Benefit/Cost (ratio)	1.63	0.63	11.75
Yen et al (2010) 12	International Journal of Workplace Health Management	Wellness	9	United States	Midwest utility company	Large	Benefit/Cost (ratio)	1.66	0.66	11.25
Caloyeras et al (2014) 69	Health Affairs	Disease management and wellness	10	United States	PepsiCo	Large	No information	1.46	NA	11.08
Nyman et al (2010) 70	JOEM	Disease management and wellness	2	United States	University of Minnesota	Large	Benefit – Cost (US dollars)	–625 947	–0.13	10.42
Goetzel et al (1998) 71	JOEM	Wellness	3	United States	Cincinnati headquarters of Procter & Gamble Company	Large	NA	NA	NA	10.25
Schultz et al (2002) 72	JOEM	Wellness	6	United States	2 manufacturing plants in the Midwest	Large	(Benefit – Cost)/Cost (ratio)	1.3	1.27	9.92
Serxner et al (2003) 73	JOEM	Wellness	1	United States	DaimlerChrysler AG 14 worksites	Large	NA	NA	NA	9.92
Bertera (1990) 74	American Journal of Public Health	Wellness	3	United States	Large multi-location (60 sites) diversified industrial company	Large	Benefit/Cost (ratio)	1.42	0.42	9.75
Griffin et al (2016) 75	Injury Prevention	Wellness	4	United States	Tucson fire department	Small	[(Benefit – Cost)/Cost] x 100 (percentage)	2.4	0.02	9.42
Mattke et al (2009) 76	American Journal of Managed Care	Disease management and wellness	5	Not reported	2 large employers in the consumer goods industry	Large	NA	NA	NA	8.75
Abraham et al (2012) 77	JOEM	Wellness	3	United States	University of Minnesota	Large	NA	NA	NA	8.08
Observational studies without comparison group (n = 8)
Bowne et al (1984) 78	JOEM	Wellness	5	United States	Southwestern home office/Houston	Large	(Benefit – Cost)/Cost (ratio)	1.93	1.93	9.33
Baker et al (2008) 9	JOEM	Disease management	1	United States	American Specialty Health, Inc, multiple workplaces (119 companies)	Large	Benefit/Cost (ratio)	1.17	0.17	9.33
Bevis et al (2014) 79	JOEM	Disease management and wellness	3	United States	Major employer in Orlando area	Large	NA	NA	NA	8
Iijima et al (2013) 80	Industrial Health	Wellness	1	Japan	11 companies (6 wholesale dealers, 3 transportation, and 2 production)	Large	Benefit/Cost (ratio)	1.55	0.55	7.67
Golaszewski et al (1992) 81	Journal of Occupational Medicine	Wellness	15	United States	Travelers insurance company	Large	(Benefit – Cost)/Cost (ratio)	2.4	2.43	7.67
Palumbo et al (2013) 82	Workplace Health and Safety	Wellness	2	United States	1 hospital unit	Small	Benefit – Cost (US dollars)	3747	0.49	7.67
Ozminkowski et al (2002) 83	JOEM	Disease management and wellness	9.5	United States	Johnson & Johnson	Large	NA	NA	NA	7.67
Maniscalco et al (1999) 84	JOEM	Wellness	5	United States	The Lafayette Offshore Business Unit/Louisiana	Small	Benefit/Cost (ratio)	2.51	1.51	5
Summary of frequencies	23 JOEM 3 AJHP	3 disease management, 28 wellness, 16 disease management and wellness	29 three-year and above	36 in United States, 9 in Netherlands, 1 in Germany, 1 in Japan	24 explicitly defined whose program was evaluated	3 small, 44 large	2 net benefit 16 benefit to cost 10 net benefit to cost	5 statistical information	31 recalculated ROI value for analysis sample

Abbreviations: NA, not available; ROI, return on investment.

The mean recalculated ROI was 0.38 (Table 3). The average recalculated ROI shows that companies saved $1.38 for every $1 invested in WWP. Of 31 publications that we included in our ROI regression analyses, 7 included only health care costs in the ROI, 7 included only absenteeism costs, 4 included both health care and absenteeism costs, 5 included health care and presenteeism costs, 4 included absenteeism and presenteeism costs, and 4 included all 3 costs. Twenty-two of the studies that we included in our ROI regression analyses evaluated only wellness programs, 4 evaluated only disease management programs, and 5 evaluated wellness and disease management programs. Eighteen studies had a follow-up length of ≥3 years. Five studies were published before 2000, 7 were published during 2000-2010, 9 were published during 2011-2014, and 10 were published during 2015-2019. Ten studies had a potential conflict of interest. Only 3 studies were conducted in small companies; the mean calculated ROI was 0.67 for these studies.

Table 3.

Descriptive statistics for recalculated ROI, article characteristics, rubric scores, quality indices, and study design among articles that had ROI measures for workplace wellness program evaluations and were included in a scoping review of articles published before December 20, 2019 (N = 31)

Summary item	Value
Recalculated ROI, mean (SD)	0.38 (3.29)
Article characteristics, frequency (%)
ROI with only health care cost	7 (23)
ROI with only absenteeism cost	7 (23)
ROI with health care and absenteeism	4 (13)
ROI with health care and presenteeism	5 (16)
ROI with absenteeism and presenteeism	4 (13)
ROI with health care, absenteeism, and presenteeism	4 (13)
Evaluated program: only wellness	22 (71)
Evaluated program: only disease management	4 (13)
Evaluated program: wellness and disease management	5 (16)
Study duration is ≥3 years	18 (58)
Publication year before 2000	5 (16)
Publication year 2000-2010	7 (23)
Publication year 2011-2014	9 (29)
Publication year 2015-2019	10 (32)
Conflict of interest (based on funding source)	10 (32)
Program in a small company	3 (10)
Statistical inference information	5 (16)
Rigor rubric main domains, mean (SD)
Reporting (8 items)	7.0 (0.8)
Internal validity (8 items)	4.9 (2.1)
External validity (1 item)	0.2 (0.4)
Power (1 item)	0.1 (0.3)
Quality index: items are equally weighted	12.2 (3.0)
Study design, frequency (%)
Randomized studies	12 (39)
Quasi-experimental studies	4 (13)
Observational cohort studies with control group	1 (3)
Observational case studies with comparison group	8 (26)
Observational studies without comparison group	6 (19)

Abbreviations: ROI, return on investment; SD, standard deviation.

The mean reporting score was 7.0, and the mean internal validity score was 4.9. The mean external validity score was 0.2, implying that about 23% of included articles met the external validity criterion. Similarly, about 10% of included articles discussed statistical power. The mean overall quality index was 12.2 points. Of 31 publications, 6 were observational studies without a comparison group (base group), 8 were observational case studies with a comparison group, 1 was an observational cohort study with a control group, 4 were quasi-experimental studies, and 12 were randomized studies.

The ordinary least squares constant represents the average ROI from an evaluation that included only health care costs in the assessment of organizational benefits, exclusively examined a wellness program, had a follow-up length of <3 years, was published before 2000, had no apparent conflicts of interest, was conducted in a large firm, and had either the average quality score (Models 1 and 3) or the average reporting and internal validity scores (Models 2 and 4) (Table 4). Studies with both costs of absenteeism and health care, studies with costs of any presenteeism, studies with a follow-up length of ≥3 years, or studies published after 2000 (except during 2015-2019) were found to be associated with lower ROI than their base or referent categories. Studies that evaluated a program with a disease management component produced higher ROI than evaluations with only a wellness component. Studies with higher reporting scores reported lower ROI than studies with lower reporting scores. Studies with higher internal validity scores reported higher ROI than studies with lower internal validity scores. However, results were not significant across all models.

Table 4.

Mean effects of article characteristics on recalculated ROI for articles that had ROI measures for workplace wellness program evaluations and were included in a scoping review of articles that evaluated workplace wellness programs, included ROI measures, and were published before December 20, 2019 ^a (N = 31) ^b

Article characteristics	Model 1 β (SE) [P value c ]	Model 2 β (SE) [P value c ]	Model 3 β (SE) [P value c ]	Model 4 β (SE) [P value c ]
Costs for absenteeism only d	–0.70 (2.83) [.81]	0.43 (2.48) [.86]	0.62 (1.87) [.74]	1.02 (1.70) [.56]
Costs for health care and absenteeism e	–2.09 (2.90) [.48]	–1.17 (2.51) [.65]	–0.56 (1.91) [.77]	–0.30 (1.72) [.87]
Costs for any presenteeism d ,e	–3.66 (2.37) [.14]	–4.79 (2.08) [.03]	–1.49 (1.61) [.37]	–2.50 (1.52) [.12]
Disease management program only f	1.11 (2.63) [.68]	1.55 (2.25) [.50]	0.60 (1.71) [.73]	0.89 (1.53) [.57]
Disease management and wellness programs f	1.99 (2.72) [.047]	1.25 (2.34) [.60]	0.35 (1.79) [.85]	0.16 (1.60) [.92]
Evaluation duration (≥3 y) g	–2.14 (2.05) [.31]	–2.16 (1.74) [.23]	–0.14 (1.45) [.92]	–0.51 (1.30) [.70]
Publication year (2000-2010) h	–2.45 (2.43) [.33]	–1.34 (2.14) [.54]	–1.75 (1.58) [.29]	–1.21 (1.46) [.42]
Publication year (2011-2014) h	–3.81 (3.06) [.23]	–2.87 (2.63) [.29]	–1.65 (2.05) [.43]	–1.42 (1.85) [.45]
Publication year (2015-2019) h	–2.83 (3.26) [.40]	0.46 (3.03) [.88]	0.49 (2.21) [.83]	1.95 (2.10) [.37]
Conflict of interest	0.16 (2.31) [.95]	–0.89 (2.02) [.67]	0.62 (1.49) [.68]	–0.05 (1.38) [.97]
Programs in a small company i	0.35 (3.25) [.92]	–2.57 (2.95) [.40]	–0.78 (2.10) [.72]	–2.38 (2.00) [.25]
Quality index j	0.05 (0.39) [.90]	– b	0.02 (0.25) [.93]	– b
Reporting domain	– b	–3.69 (1.39) [.02]	– b	–2.13 (1.00) [.049]
Internal validity domain	– b	0.62 (0.46) [.20]	– b	0.34 (0.32) [.31]
Constant	5.59 (4.20) [.20]	5.69 (3.60) [.13]	2.13 (2.88) [.47]	2.78 (2.62) [.31]
Number of observations k	31	31	29	29

Abbreviations: ROI, return on investment; SE, standard error.

^aThe dependent variable for these ordinary least squares regression models is positive ROI, which is 1 if recalculated ROI is positive and 0 otherwise.

^bModels 1 and 3 included average quality index scores (the sum of scores for reporting, internal validity, external validity, and power domains). Models 2 and 4 examined the effects of the 2 domains that contribute most to the score for quality index, controlling for reporting and internal validity domains separately and leaving out the summed quality index score because of collinearity.

^cUsing the t test, with P < .05 considered to be significant.

^dThe base category is only health care cost.

^eThis component includes combinations of costs that include presenteeism.

^fThe base category is “only wellness program.”

^gEvaluation duration is 1 if the study period is ≥3 years.

^hThe base category is publication year before 2000.

ⁱSmall is 1 if the company has ≤500 employees.

^jInternal validity, reporting, and quality index scores were demeaned (ie, sample mean subtracted from each observation).

^kTwo outliers were excluded in the sample for Models 3 and 4.

Discussion

This review addresses some points not previously considered and confirms some findings of previous reviews. It identifies factors underlying heterogeneity across studies and expands on previous findings about the association between study heterogeneity and the magnitude of ROI estimates. Although heterogeneity can never be completely eliminated, this study highlights some key sources of heterogeneity that should be addressed in future studies. Perhaps the most problematic conclusion across studies was that ROI is inconsistently defined in the literature. We acknowledge that the long history of misusing the term ROI in evaluations and reviews of the literature will make it difficult to standardize its use moving forward. Nonetheless, defining ROI using its original, financial definition, ⁸⁵ net benefit-to-cost ratio, is essential if the reasons for using ROI are to speak to financial decision makers.

Only 5 randomized studies reported confidence intervals or statistical inference information for the ROI estimate, making formal meta-analyses impossible. Providing confidence intervals for ROI is not common because ROI is measured as a ratio. An additional method, such as bootstrapping, is therefore needed to estimate confidence intervals or standard errors. This method could be the easiest way to improve WWP evaluations. Unfortunately, the lack of statistical inference information in most of the literature prevents formal estimation of an average ROI or testing of heterogeneity across studies. Therefore, we cannot provide a single, combined estimate of ROI in the literature. The average reported ROI we present should be considered a qualitative summary of the literature, not a quantitative finding. An important corollary of this finding is that most previous reviews claiming to be meta-analyses are, in fact, not formal meta-analyses but, rather, are qualitative syntheses such as the one we present here.

Although results from our recalculated ROI analysis were not significant, we believe they suggest important considerations for future ROI research. For example, WWPs with a specific outcome target could save more money than WWPs with only general wellness or health behavior targets. The health effect of wellness programs is mediated through behavior change, which might be harder to measure in the short run and have less immediate and direct effects on organizational costs than disease management programs. In contrast, disease management programs can directly affect health conditions that drive health care and productivity costs. If the primary objective for implementing WWPs is to control costs, WWPs should directly target the drivers of those costs. Disease management programs may offer a more direct effect on costs than wellness programs. If, however, the primary objective is to improve employee health, then WWPs should target health behaviors, recognizing that cost savings may only accrue in the long term.

Another consideration for future ROI research comes from our mixed results, which showed that conflicts of interest arose because of internal evaluations and were associated with higher ROI than evaluations without conflicts of interest. Although it is possible, and maybe even plausible, that internal evaluators have better access to data, thereby allowing them to better estimate ROI, independent evaluation is essential to increasing confidence in the evidence base. Eliminating conflicts of interest may be one of the most difficult obstacles in the field because of the need to rely on the cooperation of the WWP host companies.

Finally, recent critiques of the WWP ROI literature suggest that studies with greater internal validity scores yield lower ROI estimates. Yet we found that studies with greater internal validity scores (ie, with stronger evidence for causal inference) had higher ROI estimates. In general, evaluation studies, regardless of study design, do not provide the distribution information of the benefits including outliers, which could be one contributor to the positive association in estimation methods and ROI findings.

Limitations

This review had several limitations. One limitation was the small sample size and lack of formal meta-analysis underlying our pooled estimates of ROI. The standard errors did not account for the underlying sampling variation of the ROI estimates drawn from the literature and so did not support formal meta-analytic hypothesis testing. Moreover, the mean ROI did not account for the scale of programs. In theory, ROI handles this issue by being a ratio, but only if programs exhibit constant returns to scale.

Other limitations included measurement errors in data collection and self-selection into program participation. These inherent limitations cannot be eliminated. Randomized clinical trials are difficult if not impossible in some firms for legal and logistical reasons. In addition, some health-related data are not available to independent evaluators for legal reasons, making it necessary to involve an internal collaborator. Finally, this review included only peer-reviewed articles, which may lead to potential publication bias.

Public Health Implications

Our review focused on ROI findings because of the ongoing debate about the findings of the economic evaluation literature. However, the relevant outcomes from employers’ perspectives were varied and subject to change based on companies’ characteristics. For example, a small nonprofit company in one industry might adopt a WWP for corporate citizenship purposes, whereas a large for-profit company in the same industry might adopt a WWP to reduce turnover. Much of the economic evaluation literature has neglected this point.

Our scoping review provides information on areas that can improve methodologic quality for economic evaluations of WWPs. Lack of statistical inference information on ROI is an important reporting issue because we cannot conduct a meta-analysis to derive common effects of WWPs when statistical inference information is missing. The economic evaluation literature needs better reported peer-reviewed studies and attention on WWPs in companies with various characteristics, especially small companies with various reasons for WWP adoption. The advancements suggested in our scoping review will help us understand organizations’ motivations for adopting and implementing WWPs and align private- and public-sector motivations to receive policy support. The goal is for future research to validate whether WWPs can substantially affect public health.

ORCID iD

Nilay Unsal, PhD https://orcid.org/0000-0002-5683-3912