Health Risk Calculator: An online, interactive tool to estimate how health impacts workers’ compensation claim incidence and cost

Abstract

Objective:

The purpose of this paper is to describe and evaluate a web-based, educational Health Risk Calculator that communicates the value of investing in employee health and well-being for the prevention of work-related injuries, illnesses and fatalities.

Methods:

We developed and evaluated the calculator following the RE-AIM framework. We assessed effectiveness via focus groups (n=15) and a post-use survey (n=33) and reach via website analytics.

Results:

We observed evidence for the calculator’s usability, educational benefit, and encouragement of action to improve worker health and safety. Website analytics data demonstrated that we to reached over 300 users in equally in urban and rural areas within three months after launch.

Conclusions:

We urge researchers to consider the ways in which they can communicate their empirical research findings to their key stakeholders and to evaluate their communication efforts.

INTRODUCTION

Over 120 million adults in the United States (US) work and they spend about one-third of their waking time at work, which makes the workplace an important arena for health protection and promotion.¹ Workers in the US suffer from a number of physical chronic health conditions, poor psychosocial health, and over three million annual non-fatal work-related injuries and illnesses on average.^{2, 3} Our research estimates that these health concerns can cost a business with five-thousand workers about half a million dollars per year in lost productivity.⁴ Others estimate that work-related injuries alone cost society more than $250 billion per year⁵ in direct and indirect costs. While there is a plethora of empirical evidence that characterizes this burden, there is little translational research that seeks to understand how to communicate these findings to key stakeholders.⁶

Some calculators exist to help employees and employers understand the relationship between employee health, productivity and healthcare costs^7-10 as well as between work-related injury and injury costs,^11-13 but to the authors’ knowledge, a web-based calculator to help employers understand the connection between employee health and work-related injury does not exist. Such a calculator could help educate employers on the importance of implementing and supporting Total Worker Health® (TWH) policies, programs and practices. Historically, employers have operated their safety program and worksite wellness program in separate silos without coordination. The TWH strategy reflects a relatively recent national effort led by The National Institute for Occupational Safety & Health’s (NIOSH) office of TWH to facilitate the integration of health protection with health promotion at work.¹⁴ The initiative is guided by the NIOSH National TWH Agenda, which has specific community practice goals around “translating new research findings and concepts to applications, practices or technologies that can be utilized and evaluated in a variety of work settings”.¹⁵

Researchers have created numerous health communication materials to assist businesses with implementing TWH business policies and programs; however, few health communication materials exist to help businesses understand the cost implication of adopting these programs. NIOSH’s website contains a webpage devoted to making the business case for TWH using a white-paper style communication method.¹⁶ Additionally, comprehensive, web-based written and video guides exist to help businesses develop TWH policies and programs.^17-19 We argue that educational tools that communicate and quantify the value of a TWH approach are a needed first step in the process of moving business leaders along the stages of change.²⁰

The purpose of this paper is to describe and evaluate a web-based, educational calculator that communicates the value of investing in employee health and well-being for the prevention of work-related injuries, illnesses and fatalities. This study describes the Health Risk Calculator – www.ucdenver.edu/chwe/calculator - that educates businesses on the relationship between health risks and workers’ compensation (WC) costs. The calculator facilitates connecting businesses to community resources to take action to promote employee health. A catalyst for this project was a partnership that the Center for Health, Work & Environment had with Pinnacol Assurance, a WC insurer in Colorado. In Colorado, all businesses with at least one employee must carry WC insurance to ensure workers receive medical care and indemnity compensation (e.g., lost wages) for work-related injuries, illness, and fatalities. As a WC carrier, Pinnacol Assurance offers safety consultation services to policyholders. From 2010 to 2014, they piloted a worksite wellness program service with their policyholders and partnered with the Center for Health, Work & Environment at the Colorado School of Public Health and others from academia and consulting to evaluate their Health Risk Management (HRM) program. This partnership resulted in several peer reviewed publications outlining the need for businesses to address the health, safety, and productivity of employees.^{4, 21-25} However, the partners were aware of the need to disseminate these findings to businesses in a way that resonated with them to ultimately help facilitate action to enhance their employee’s health and safety at work.

METHODS

Web-Application Calculator Development

Algorithm development

Data source

In order to estimate workers’ compensation claims and costs, we obtained data from the Pinnacol Assurance HRM study that was conducted from May 1, 2010 through December 31, 2014. The participants in the study included 314 businesses from diverse industries, such as services and construction/mining, and geographical locations in Colorado. Additionally, businesses ranged in size: small (<50 employees, n=166), small/medium (50-99 employees, n=70), medium (100-499 employees, n=68), and large (500+ employees, n=10). Once a business enrolled in the study, employees were eligible to take a health risk assessment (HRA) as well as participate in other worksite wellness activities. Employers received aggregated reports of employee needs and progress. More information on the study can be found in previously published studies.^{4, 21-23}

During the study, a total of 16,926 employees completed 25,993 HRAs. In brief, the HRA included validated questions about chronic health conditions and overall health from an NQCA-certified tool provided by Wellsource ²⁶ and the World Health Organization’s Health and Work Performance Questionnaire (HPQ)²⁷ and the more abbreviated HPQ Select.²⁸ We linked the HRA data to WC claims data provided by Pinnacol Assurance through a robust linkage process whereby both sets of data were sent to an independent third party to be de-identified and then provided to researchers for analysis.

Inputs and Outputs

The calculator inputs reflect demographic and health-related variables. In total, we included 34 health-related predictor variables of interest in the model as they were shown to be previously associated with healthcare costs.^{29, 30} We also controlled for nine demographic variables. We limited the number of calculator input variables that were visible to the user to 12 health risk factors: culture of health rating, overall health status, stress at work, poor sleep, abnormal body mass index, tobacco use, high cholesterol, depression, migraines, heart disease, arthritis, and diabetes. We selected these 12 health risk factors based on their relationship with claim occurrence²³ as well as how actionable they were for the employer to provide resources for health improvement. We also included five workforce composition input variables: industry, number of employees, aging workforce, full-time/part-time status, and hourly/salary status. We strove to minimize the number of visible input variables to balance valid outputs with ease of user engagement and comprehension. The following variables were omitted from the calculator display and set at the average value in the dataset for each industry: alcohol use, weekly amount of exercise, high blood pressure, stress at home, financial stress, fruit and vegetable intake, back pain, cancer, asthma, stomach issues, IBS, lung disease, fatigue, osteoporosis, annual physical exam, bi-annual dental care, secondhand smoke, seat belt usage, helmet usage, drinking and driving, safe lifting practices, anxiety, gender, income levels, education levels, and job categories.

The outputs reflect anticipated volume of WC claims. Once we linked the HRA to the WC claims data, we created an indicator variable to mark whether a worker had a closed, non-zero cost, WC claim within one year after their HRA response. Next, we worked with Pinnacol Assurance to obtain the average claim cost across Pinnacol Assurance’s entire portfolio of businesses (>60,000 policy holders), not just the claims represented in our study dataset. This was done to ensure that cost estimates accurately reflected projected claim costs. All WC claim costs were adjusted to 2013-dollar values.

Statistical analysis

We first randomized the dataset into a training set (n=19,493; 75%) and a verification set (n=6,500; 25%). To protect against overfitting the model, the training set was used to develop the model, while the verification set was used to assess the model performance.³¹ We modeled the probability of a claim on the individual level using multivariable logistic regression with random intercepts for employee using the training set, and then evaluated model performance with the verification set. No variable selection took place, all demographic and health risk factor variables described above were included in the multivariable model. The resulting predictor coefficients from this model were used as the basis for the calculator.

User experience

We partnered with a communications and web development firm to design and build the educational, interactive, and user-friendly online calculator. Our goal was to apply the algorithm described above to interactively display the inputs and outputs in a way that was quick and easy to use. The communications and web development firm assisted us with mapping the user journey and coding the web app. We designed the app’s written copy, images, and flow with businesses (owners, safety and health professionals, executives) and business affiliates (safety/worksite wellness consultants, insurance agents and brokers) in mind because of the role both play in implementing workplace health and safety policies and practices. The firm programmed the app as a “white label app,” which meant that it could be embedded into any existing webpage or portal. We also created an FAQ page to answer commonly asked questions and direct users to contacts and resources. We hosted a link to the calculator and the FAQ on Pinnacol’s policyholder portal throughout the evaluation study period. The link took users to a standalone website.

The homepage of the calculator is visually appealing with a simple call to action: Get Started. Once users click Get Started, they begin a 3-step journey including, 1) Estimate Your Compensation Claims & Costs by Next Year, 2) Personalize Your Risk, and 3) Your Results (see Figure 1). The user first observes the baseline relationship between their industry and business size and frequency and cost of claims without accounting for their business’s employee health risks. This is calculated by setting all estimated coefficients from the multivariable logistic regression model at their industry average. At this stage, the user can only manipulate the industry and business size inputs. The probability of the claim is multiplied by 100 to provide the unadjusted predicted number of claims per 100 employees. This estimate is then multiplied by the industry-specific average cost of a claim to obtain the expected cost of all claim(s) per 100 employees. The probability of a claim changes as these two inputs are changed by the user.

Figure 1.

Screenshots of the Health Risk Calculator.

In the second step, the user is able to customize the results by observing how the two claim outcomes change when they manipulate workforce composition and health risks inputs. The estimated coefficient for each input is multiplied by response option the user selects (e.g., percent of employees who experience work stress). The user then observes the estimated adjusted predicted number and cost of claims. For predictor variables that are in the underlying model but not presented as calculator inputs online, the values are held constant at their industry average (e.g., 14% alcohol use for mining/construction compared to 5% for educational services).

Finally, the user goes to the Results Page, which includes visuals to display the unadjusted industry average and adjusted predicted number and costs of claims. We encouraged users to complete a survey after submitting their inputs and receiving their results by clicking a “Take Survey” button. This is how we collected the evaluation survey data used in the evaluation study described below.

Web-Application Evaluation

We evaluated the calculator using the RE-AIM program evaluation framework³² following Suggs³³ recommendations for evaluating new technologies for health communication (see Table I). This framework evaluates interventions in five areas: Reach, Effectiveness, Adoption, Implementation, and Maintenance. For the purpose of this study, we focused on Effectiveness – satisfaction with the calculator and knowledge and behavioral intentions after using the calculator during the beta and soft-launch phases. We also assessed Reach – the number of users that accessed the calculator - and Implementation - the consistency with which the calculator was delivered as planned – after official launch on the Pinnacol website under their policyholder portal (see Table I).

Table I.

Health Risk Calculator development and evaluation

Timeline	Jun. – Sept. 2015	Oct. – Dec. 2015	Jan. 2016	Mar. – Jun. 2016
Phases	Development	Beta-test	Soft launch on CHWE hosted site	Launched on Pinnacol website under policyholder portal
RE-AIM evaluation activities
Reach				Website analytics
Effectiveness		Focus groups
		Post-use surveys	Post-use surveys

Participants and recruitment

We evaluated the calculator amongst two target audiences. First, we recruited policyholders who represented business representatives, such as owners or safety managers. Second, we recruited affiliates who represented Pinnacol Assurance safety services consultants, worksite wellness professionals, and insurance brokers, essentially the professionals that work with businesses. The Pinnacol Assurance HRM program director helped us recruit participants who were familiar with the HRM program. They identified policyholders who were engaged in the HRM program as well as affiliates who were involved in the HRM either as employees working on the program or as individuals who consult with businesses that were involved in program. The researchers then sent an email inviting them to participate.

Data collection

We collected data throughout the beta, soft-launch, and launch phases to assess each component of the RE-AIM framework. To assess effectiveness, we first conducted focus groups with our target audiences during the beta-test to gain a better understanding of user experience including what they thought about the calculator’s design, functionality, and key messaging. We conducted one focus group (n=6) with policyholders and another (n=9) with affiliates. Both focus groups lasted 90 minutes, and were voice recorded. During the first 10 minutes, participants used the calculator on their own and took a post-use online survey. The survey asked questions about demographics, acceptability, appropriateness, feasibility, knowledge gained, and intentions after using the calculator. The target audiences completed separate surveys, but some of the questions overlapped for comparability. The link to the user survey was embedded in the calculator at the end when they were reading their results. The rest of the session included a guided discussion on how they interacted with the calculator, what they thought about its content and design, and what they learned. Second, during the soft-launch phase, we emailed the calculator link to a sample of affiliates and policyholders. Similar to the focus group participants, we asked the participant to use the calculator and provide their feedback via the same post-use survey. After official launch, we tracked calculator reach via Google website analytics. Finally, we describe our implementation success in the context of our reach findings.

Data analysis

We conducted qualitative and quantitative analyses of all evaluation data. We analyzed the focus groups by reviewing audio transcripts for common themes by target audience. We generated descriptive statistics of the post-use survey responses, using chi-square tests and independent samples t-tests, as appropriate. First, we conducted a within target audience group comparative analysis of policyholders and affiliates who completed either a beta-test or soft-launch survey to determine whether the data from both timepoints could be combined. Our null hypothesis was that respondents would not differ demographically nor would mean responses be significantly better among the soft-launch survey respondents, compared to the beta-test survey respondents. Second, we combined beta-test and soft-launch survey data and conducted a between target audience comparative analysis of policyholder and affiliate survey responses. Our null hypothesis was that mean responses would not be significantly different between our target audiences. Finally, we generated descriptive statistics of the web-analytics results.

WEB-APPLICATION EVALUATION RESULTS

Focus groups

Focus group participants had many consistent comments and questions about the calculator. They all liked the clean and simple design and they understood the message it was trying to convey. They also all thought that the calculator helped to start a conversation about employee health and its association with workplace safety, because it left them wanting more. They were also motivated to take action right after using the calculator but wanted more help in determining what steps to take next.

The groups diverged in their opinions regarding how the calculator would be used in practice. Policyholder representatives focused on how they might use this calculator with their employees to improve buy-in for their company’s wellness program. They highlighted the common difficulty they have getting employees engaged in these programs and thought that the calculator could help them facilitate interest and participation. They also mentioned that it was hard for them to estimate the health status of their workforce when they did not have access to specific employee health information such as body mass index (BMI), sleep patterns, and chronic disease conditions. On the other hand, affiliates keyed in on using the calculator as a marketing and decision tool when interacting with policyholders. They requested more information on how to use and interpret the calculator’s results so that they can effectively communicate it to policyholders. Participants felt confident in navigating the tool and estimating inputs.

We used the results from these focus groups to improve the calculator content and to generate marketing materials to increase awareness and adoption of the tool after implementation. First, the feedback allowed us to refine language used within the calculator. We also decided to include more information on how to use the calculator and how to interpret the results in the FAQ page. Second, we used the focus group feedback to generate our marketing and communications plan, including promotional materials and website content. For example, the policyholder group felt the calculator had credibility because it was developed by the University through evidence-based research. So, in our marketing materials and the FAQ page, we highlighted the public-private partnership between the University and Pinnacol Assurance

Survey results

A total of 33 users from our target audience responded to the post-use online survey during the beta-test and soft-launch. Table II presents the demographics of our target audiences by beta-test and soft-lunch timepoint. A chi-square test comparing participant genders between the beta-test and soft-lunch groups indicated that there were significantly more females in the policyholder soft-launch group (100%) than in the beta-test group (40%) (χ² (2, N = 13) = 6.24, p<0.01). No other participant demographics differed by group. Generally, across our target audiences at each timepoint, about half were over the age of 45. Participants held a variety of positions such as business owner and insurance broker. More than half of the policyholders worked in the services industry. In the beta-test group, both policyholders and affiliates were mostly based in an urban area of Colorado whereas the soft-launch group came from both urban and rural areas. Many of the policyholders in both groups indicated they had some safety and wellness programming but had workers who experienced a WC claim in the past year.

Table II.

Description of post-calculator use survey sample by target audience

	Policyholders		Affiliates
	Beta-test (n=5)	Launch (n=9)	Beta-test (n=8)	Launch (n=11)
	N (%)	N (%)	N (%)	N (%)
Age
18-29	0 (0%)	0 (0%)	1 (17%)	0 (0%)
30-44	1 (20%)	2 (25%)	1 (17%)	4 (44%)
45-59	3 (60%)	4 (50%)	3 (50%)	5 (56%)
60+	1 (20%)	2 (25%)	1 (17%)	0 (0%)
Gender, Female*	2 (40%)	7 (100%)	2 (40%)	3 (43%)
Position
Business owner	1 (20%)	2 (25%)	n/a	n/a
Human resources rep	2 (40%)	3 (38%)	n/a	n/a
Manager/supervisor	2 (40%)	0 (0%)	n/a	n/a
Safety/risk management	0 (0%)	2 (25%)	n/a	n/a
Insurance agent	n/a	n/a	4 (67%)	2 (22%)
Pinnacol Assurance safety services employee	n/a	n/a	1 (17%)	5 (56%)
Pinnacol Assurance health and wellness employee	n/a	n/a	0 (0%)	1 (11%)
Other	0 (0%)	1 (13%)	1 (17%)	1 (11%)
Industry
Finance	0 (0%)	1 (13%)	n/a	n/a
Manufacturing	0 (0%)	1 (13%)	n/a	n/a
Mining/construction	2 (40%)	0 (0%)	n/a	n/a
Public administration	0 (0%)	1 (13%)	n/a	n/a
Services	3 (60%)	4 (50%)	n/a	n/a
Transport/comm/ele/gas/san	0 (0%)	1 (13%)	n/a	n/a
Business size
<100 employees	1 (20%)	2 (25%)	n/a	n/a
100 – 499 employees	3 (60%)	4 (50%)	n/a	n/a
>500 employees	1 (20%	2 (25%)	n/a	n/a
Region
Denver area	5 (100%)	3 (38%)	5 (83%)	6 (67%)
Front range	0 (0%)	1 (13%)	1 (17%)	1 (11%)
Northwest	0 (0%)	0 (0%)	0 (0%)	2 (22%)
Southeast	0 (0%)	1 (13%)	0 (0%)	0 (0%)
Southwest	0 (0%)	3 (38%)	0 (0%)	0 (0%)
Current business safety, health, and well-being practices
Safety activities, % yes	4 (80%)	8 (100%)	n/a	n/a
Worksite wellness activities, % yes	3 (60%)	7 (88%)	n/a	n/a
Certified Health Links Healthy Business, % yes	1 (20%)	2 (25%)	n/a	n/a
Workers’ compensation claim(s) in past year, % yes	5 (100%)	6 (75%)	n/a	n/a

^*Policyholder group chi-square test comparing beta-test and launch responses statistically significant at p <0.01

n/a Participant was not asked this survey question.

Note. Frequencies may not add up to the full sample size due to missing data. Thus, percentages represent the number of respondents divided by the percentage of respondents that responded to the survey question.

First, we compared mean survey responses within target audience by timepoint (beta-test and soft-launch). There were no significant differences in the policyholder mean survey responses between the beta-test and soft-launch timepoints. However, the affiliates had significantly different beta-test and soft-launch survey responses to three questions. Affiliates who completed the post-use survey during the soft-launch were more likely to report that the calculator was user friendly (M = 4.50, SD = 071) than affiliates in the beta-test (M = 3.67, SD = 0.82), t(13) = −2.16, p=0.02. Affiliates who completed the post-use survey during the soft-launch were more likely to report that the calculator reflected the importance of safety, health, and well-being (M = 4.11, SD = 0.33) than affiliates in the beta-test (M = 3.17, SD = 0.75), t(13) = −3.35, p<0.01. Finally, affiliates who completed the post-use survey during the soft-launch were more likely to report that the calculator would move policyholders to approve of health, safety, and well-being (M = 3.67, SD = 0.50) than affiliates in the beta-test (M = 3.00, SD = 0.63), t(13) = −2.28, p=0.02.

Second, we averaged the survey responses across timepoints within target audiences and then compared average responses between target audiences (see Table III). As the within group results suggest above, combining affiliate survey responses for three of the questions may bias the between-target audiences results towards the null. However, our between-target audiences analysis suggests that affiliates rated these questions significantly more positively than policyholders. For example, affiliates indicated that the calculator was more user friendly (M = 4.19, SD = 0.83) than the policyholders (M = 3.46, SD = 1.05), t(27) = 2.08, p=0.02. After using the calculator, affiliates also reported more knowledge than policyholders. For example, affiliates said they knew more about where to find information to enhance employee health and well-being (M = 4.67, SD = 0.36) than policyholders (M = 3.38, SD = 0.42), t(26) = 2.34, p=0.01. After using the calculator, both policyholders and affiliates reported the highest agreement to questions about their intentions to explore Pinnacol safety services resources (M = 3.77, SD = 0.93; 4.27, SD = 0.59, respectively) followed by intentions to join Pinnacol’s HRM program (M = 3.62, SD = 0.96; 4.20, SD = 0.68, respectively).

Table III.

Results of post-calculator use survey by target audience – Data combined from beta-test and soft-launch surveys

	Policyholders (n=14)	Affiliates (n=19)
Survey question	M (SD)	M (SD)
Acceptability - The health risk calculator was very user friendly**	3.46 (1.05)	4.19 (0.83)
Appropriateness - This calculator reinforces the importance of employee safety, health, and well-being*	3.46 (1.05)	3.73 (0.70)
Feasibility - After using this calculator, employees at my business/policyholders would approve of employee health, safety, and well-being**	2.92 (0.86)	3.40 (0.16)
Knowledge - After using this calculator, I know…
More about (our) businesses industry average workers’ compensation risk	3.69 (1.93)	4.20 (1.37)
How health risk contributes to (our) businesses workers’ compensation risk*	3.84 (1.28)	4.53 (1.06)
Where to find more information on how to enhance employee safety**	3.54 (1.45)	4.67 (1.40)
Where to find more information to enhance employee health and well-being***	3.38 (1.50)	4.67 (1.40)
Intentions - After using this calculator, I will…
Share the Health Risk Calculator results	3.23 (1.17)	n/a
Join Pinnacol’s free worksite wellness program	3.62 (0.96)	n/a
Explore Pinnacol Safety Services resources	3.77 (0.93)	n/a
Apply for or renew my business’s Health Links Certification	2.92 (0.86)	n/a
Encourage policyholders to use the Health Risk Calculator	n/a	4.07 (0.70)
Help policyholders start or improve their business wellness program	n/a	4.06 (0.70)
Help policyholders start or improve their business safety program	n/a	4.20 (0.68)
Encourage policyholders to explore Pinnacol Safety Services and resources	n/a	4.27 (0.59)
Encourage policyholders to explore Pinnacol Worksite Wellness Services	n/a	4.20 (0.68)
Encourage policyholders to apply for or renew their business’s Health Links Certification	n/a	3.80 (0.68)

^***Independent samples t-test comparing mean survey responses between target audience for survey questions that were comparable - statistically significant at p < 0.01;

^**p < 0.05;

^*p < 0.10

n/a Participant was not asked this survey question.

Note. All question responses were on a 1-5 Likert scale, strongly disagree to strongly agree. M = mean and SD = standard deviation.

Website analytics results

Overall, we reached a total of 324 users in the first three months after launch. The majority were new users (n = 226, 70%), and fewer were returning users (n = 98, 30%). The majority of users (n = 313, 96%) accessed the Calculator directly by typing in the URL, and then spent, on average, 1:12 minutes using the Calculator. The users that accessed the calculator via Pinnacol Assurance’s website (n = 11, 3%) spent, on average, 3:04 minutes using the calculator. The majority of users overall (95%) accessed the Calculator from their desktop, not their mobile/tablet device. Few accessed the FAQ page (n = 27, 8%). Approximately half of the users came from urban areas: Denver, CO (n = 157, 48%), Aurora, CO (n = 24, 7%) and Colorado Springs, CO (n = 10, 3%). The other half represented over 40 other rural cities across Colorado. As can be seen in Figure 2, the majority accessed the calculator right after launch.

Figure 2.

Number of user sessions by week after calculator launch on Pinnacol Assurance’s policyholder portal

We believe our reach was limited by implementation challenges. We designed the calculator to be a “white label” app so that it could be embedded directly into Pinnacol Assurance’s policyholder portal. The way the calculator was deployed at the official launch was through a link that was embedded on a webpage within policyholder portal. The link directed users to the calculator on a separate webpage. Due to where the link to the calculator was placed on the policyholder portal, the visibility and therefore accessibility to the tool was limited upon sign in.

DISCUSSION

There is a need to understand how to communicate workplace health-specific research findings to businesses. Such communications should help businesses take action to implement TWH policies and programs. Our study presents an effort to develop and evaluate a technology-based health communication solution to help businesses understand the value of investing in employee health and well-being for the prevention of work-related injuries, illnesses and fatalities. Our strategy followed methods to comprehensively identify target audiences, develop and tailor key messages, identify and deploy the appropriate health communication technology channel, and evaluate the technology application.³³ Results from both our qualitative and quantitative analyses provide evidence for the calculator’s efficacy in terms of usability, educational benefit, and encouragement of action to improve worker health and safety. Furthermore, despite implementation challenges, our website analytics demonstrate that we were still able to reach 324 people from equally urban and rural areas of Colorado in the first three-months after launch.

Unexpectedly, we observed that affiliates rated post-use survey questions about acceptability, appropriateness, feasibility and knowledge significantly higher than business representatives. This may be due to the fact that affiliates already consult with businesses on workforce health and safety, and thus may be primed to the calculator’s content. Many of the businesses that Pinnacol Assurance and we work with are small businesses with less than 100 employees. These types of businesses often rely on intermediaries for information and assistance in adopting and implementing TWH policies and practices.³⁴ Indeed, the affiliates mentioned that they could see themselves using the calculator as a health communication tool when working with businesses. Our study suggests that the calculator may be more effectively used amongst affiliates to help businesses understand the value of investing in workforce health and safety.

Our study adds to the work that researchers have already done to publish health-related, employer-focused web-based calculators in the peer-reviewed literature. Three studies describe employer-focused calculators estimating health impacts on healthcare costs and worker productivity.^{7, 9, 10} Like the present study, the researchers intended their calculator to educate and invite the user to take action. However, unlike the present study, none of these studies describe strategies to develop the user experience or the use of an application to drive organizational behavior change. Furthermore, though some of the studies did not intend to share their calculator widely,^{8, 10} those that did intend to do not provide program evaluation data.^{7, 9} Our study adds to this literature by providing an example strategy to develop an evidence-based, technology application to communicate important health messages in the workplace as well as a strategy to evaluate the application via the RE-AIM framework.³² Our combined evaluation findings suggest that we were able to reach several hundred users within three months of launch and that perceptions of the acceptability, appropriateness, feasibility, knowledge gained, and intentions after using the calculator differed by target audience.

There are several avenues for future health communication research in the field of TWH. Based on our focus group with business representatives, we argue that future health communication technology interventions should allow for easier customization. User’s company specific data should be directly input into the calculator, for example from an HRA survey, so that users can accurately estimate how their employees’ health impacts WC costs. Improving this piece of the user experience may help make the calculator more user friendly and understandable for businesses. Future research should also consider how use of the calculator results in the adoption and implementation of organizational TWH policies and programs. For example, after a safety program manager uses the calculator, does it result in implementation of new workplace health and safety strategies? Do users make changes that ultimately impact worker health and safety outcomes? There is also an opportunity to develop other TWH-focused calculators using the methods outlined in this study. For example, a calculator that integrates more sources of data, such as healthcare, disability, and workers’ compensation claims, can help businesses understand the complex interplay between health, safety, and well-being and health and productivity outcomes.

Update on website analytics results after study completion

Since collecting the website analytics data for the study period (March 1 – May 31, 2016), we have continued to use Google Analytics to track and measure the use of the calculator. After we completed data collection for this evaluation study, we linked to the calculator’s webpage from the Calculator to the Center for Health, Work & Environment’s community and practice webpage where we house open access business-facing resources. We also marketed the calculator through email marketing, social media, and presentations at conferences. After several months, we directly embedded the Calculator on its own page on our website - www.ucdenver.edu/chwe/calculator. From June 1, 2016 to March 27, 2019, we reached a total of 2,423 users (68% of them were unique users). Most users (66%) accessed the calculator through an organic word search (Google) or social media. Users spent an average of 1:45 minutes using the Calculator. The majority of users overall (79%) accessed the Calculator from their desktop, versus their mobile/tablet device. When looking at users who accessed the Calculator from cities in Colorado, 61% of them came from urban cities and 38% came from rural cities. The combined website analytics data from both the study and post-study period indicates that we were able to reach 2,747 users in both rural and urban regions of Colorado.

Strengths & Limitations

The calculator represents a first of its kind educational tool. The calculator’s algorithm is based on a diverse sample of 314 businesses from different industries and regions of Colorado and almost 17,000 employees with various demographic factors. Additionally, we developed the calculator through a participatory approach with our target audiences to ensure the content and design was user friendly. Finally, we also included a qualitative and quantitative evaluation component of the project to evaluate reach, effectiveness, and implementation results.

While the sample represented in the calculator’s algorithm was diverse, they represent individuals who were willing to participate in an HRA and only workers residing in Colorado. Additionally, the algorithm also represents an analytical model that was internally validated, but not externally validated with a different sample. Also, we did not collect pre-calculator use survey data and cannot say whether knowledge and actions changed from before to after using the calculator.

The implementation of the calculator was limited in its accessibility. Users were required to login to a web portal to access the tool. Furthermore, due to challenges, we were not able to embed the calculator in the policyholder portal. This limited our implementation fidelity and impacted our reach and ultimately the sustainability of the calculator.

Conclusions

In conclusion our study demonstrates a method to successfully translate and disseminate health and safety messaging and calls-to-action to businesses via a web-based Health Risk Calculator. While our study demonstrates that implementing health communication strategies in practice can be challenging, we were still able to reach hundreds of our target audience in both urban and rural regions. Our effectiveness evaluation of the calculator indicates that it can be most successfully used amongst individuals who consult with businesses on workforce health and safety. We encourage TWH researchers to work comprehensively to develop and evaluate innovative tools to ensure public health impact.