Design and baseline data of a randomized trial comparing two methods for scaling-up an occupational sun protection intervention

Abstract

Background

Successful methods for scaling-up evidence-based programs are needed to prevent skin cancer among adults who work outdoors in the sun.

Methods

A randomized trial is being conducted comparing two methods of scaling-up the Sun Safe Workplaces (SSW) intervention. Departments of transportation (DOTs) from 21 U.S. states are participating and their 138 regional districts were randomized following baseline assessment. In districts assigned to the in-person method (n = 46), project staff meets personally with managers, conducts trainings for employees, and provides printed materials. In districts assigned to the digital method (n = 92), project staff conduct these same activities virtually, using conferencing technology, online training, and electronic materials. Delivery of SSW in both groups was tailored to managers' readiness to adopt occupational sun safety. Posttesting will assess manager's support for and use of SSW and employees' sun safety. An economic evaluation will explore whether the method that uses digital technology results in lower implementation of SSW but is more cost-effective relative to the in-person method.

Results

The state DOTs range in size from 997 to 18,415 employees. At baseline, 1113 managers (49.0%) completed the pretest (91.5% male, 91.1% white, 19.77 years on the job, 66.5% worked outdoors; and 24.4% had high-risk skin types). They were generally supportive of occupational sun safety. A minority reported that the employer had a written policy, half reported training, and two-thirds, messaging on sun protection.

Conclusions

Digital methods are available that may make scale-up of SSW cost-effective in a national distribution to nearly half of the state DOTs.

Trial registration: The ClinicalTrials.gov registration number is NCT03278340.

1. Introduction

In the field of implementation science, scale-up is the effort to increase the impact of successful prevention interventions to benefit more people on a lasting basis [1]. Scale-up methods must be lower cost and increase reach to provide greater access to benefits [[2], [3], [4]]. Going to scale with a research-tested intervention is more than replication in large populations. Options, knowledge processes, and technologies are expanded to build capacity and influence decision-makers [1] and understand implementation and cost of programs [5]. Costs of national distribution can be daunting, so affordable scale-up methods are needed to maintain sufficient intervention effectiveness, as interventions' effectiveness often declines [6] due to reduced dose and fidelity [7] and adaptions to fit contextual/budget parameters [6]. Information on costs and intervention delivery resources are under-reported in dissemination and implementation [8].

A randomized trial is being conducted to compare two methods of scaling up, nationwide, an effective occupational sun protection intervention, Sun Safe Workplaces (SSW), that promotes policy and education for outdoor workers [[9], [10], [11]]. Americans laboring outdoors are regularly exposed to high doses of solar ultraviolet radiation over many years [12,13] that elevate risk for skin cancer [[14], [15], [16], [17], [18], [19]], especially keratinocyte cancers [[20], [21], [22]] but also melanoma [23,24]. A method utilizing digital technology to conduct virtual communication with workplace managers is being compared to the original intervention method that relied on in-person contacts. The intervention is being disseminated to state departments of transportation (DOTs), public employers with large numbers of employees who work outdoors. The evaluation will explore whether the scale-up method that uses digital technology produces a lower implementation rate than the original in-person method by testing the following hypothesis:

H: Compared to employers in the in-person scale-up method, fewer employers receiving the digital scale-up method will deliver education and implement other sun safety action for employees.

Analyses are also exploring whether SSW can be delivered cost-effectively to worksites using the digital scale-up method, which contains scale-up costs. “Cost-effective” in our case means that digital method should be chosen over in-person despite a lower worksite implementation rate if the cost savings from the digital method are at least as much as a decision-maker would accept in exchange for fewer implementing worksites.

Several frameworks have guided scale-up, such as collaborative models that stress planning, finances, and systems [25], interactive systems approaches [26], and process models such as EPIS (exploration, preparation, implementation, and sustainability) [27]. The present trial was based on the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [28,29], a robust framework for program dissemination [30]. RE-AIM can address costs and economic outcomes across general and multiple dimensions of the model [31,32] including at the organizational and individual levels [31,33]. It applies to the health policy arena by estimating public health impact and integrating policies with health promotion strategies [34]. In RE-AIM, cost influences several aspects of dissemination (e.g., intervention intensity is positively related to effectiveness but negatively to implementation) [35]. The economic analysis will examine key resource elements that include distribution, effectiveness, and replication and implementation costs [8]. The scale-up methods, randomized trial design, and results from the baseline survey of managers on occupational and personal sun protection are presented in this paper.

2. Materials and methods

2.1. Experimental design

The randomized trial is underway and designed to compare the distribution of SSW using the original in-person method to a digital method for meetings and training. The protocol originally included 200 fire departments and state departments of transportation (DOTs) enrolled in two waves (to make it more feasible) into a randomized, pretest-posttest two-group experimental design. Employers were to be stratified by size (large v. small; median split on number of employees) and solar intensity (a proxy for climate and UV), which had been significant covariates in our past research [36]. After pretesting, regional districts in the DOTs were randomized to the in-person or digital scale-up method, originally in a planned 1:3 ratio. The intervention period lasts 20-months for each district, after which posttesting will be completed. All procedures were approved by the Western Institutional Review Board.

2.2. Changes to design after commencement of the trial

After the trial commenced, the sample size was reduced to 138 because of budget constraints created by a reduction in the budget from the awarding agency. Also, the sample was altered to include just state DOTs. This also occurred because of budget constraints when a) cost of identifying fire departments was higher than expected, b) the American Association of State Highway and Transportation Officers (AASHTO) was able to attract a large number of state departments of transportation to enroll, and c) state DOTs had similar structures with managers and employees organized into multiple regional districts. Regional districts became the unit of randomization rather than individual departments. Another change was that the policy measure was changed from an outcome variable to a moderator variable, because in state DOTs, written policies on sun protection are set at the central state office, with employee education delivered at the regional district level. As all 50 state DOTs were invited to participate, stratification by size and solar intensity region was eliminated. Instead, randomization occurred within state and each state had regional districts assigned to in-person and digital scale-up strategies (except one state that had only a single participating district, which was assigned to in-person strategy). Combined across states, districts were assigned in a 1:2 ratio to in-person to digital strategies. Finally, plans to pretest employees were eliminated due to budget constraints and reluctance by state DOT managers to provide access to employees prior to participating in the study.

2.3. Recruitment procedures

State DOTs in all 50 states were invited to participate in the trial by a letter sent to the state DOT directors from the AASHTO Chief Executive Officer along with a description of the study. Project investigators followed up by telephone and email to secure agreement to participate. A key contact at the state office provided information on the state DOT districts, including district locations and characteristics, and manager names to be sent the pretest. Eligibility criteria for DOT regional districts were a) being a regional district in a participating state DOT, b) located in the United States, c) having the state DOT provide written safety policies at pretest, and d) having at least 6 managers complete the pretest. Eligibility criteria for managers included a) being in a management or front-line supervisory position in a regional district; b) supervising outdoor workers; c) consenting to participate, and d) completing the pretest. The manager cohort included Safety Officers/Leads, Safety Coordinators, Maintenance Administrators/Supervisors, Resident/Construction Engineers, Maintenance/District Superintendents, Operations/Infrastructure Managers, and Project/Section Supervisors. Managers were sampled by position at pretest and persons holding those same positions will be posttested. Survey invitations were sent by email followed by weekly reminders for up to 5 weeks. After that, persistent non-responders received a printed survey by U.S. mail, with a return-addressed stamped envelope. If the position holder changes, we will attempt to posttest both the new and original position holders. At posttest, eligibility criteria for employees to be assessed are a) being employed part/full-time in a participating regional district, b) working at least part of daytime hours outdoors, and c) consenting to participate. Employees will be sampled at posttest from employee rolls. Districts were the unit of randomization within each state. After each state completed the pretest protocol, the project statistician randomized districts into scale-up strategy (in-person or digital).

2.4. Sun safe workplaces intervention

The SSW program was designed by our team and demonstrated effectiveness in a randomized trial [[9], [10], [11]]. It promoted comprehensive occupational sun protection (i.e., adoption of sun safe policies and provision of sun safety education) to ensure employee sun safety was promoted and supported long term. The intervention was based on principles of Diffusion of Innovations Theory (DIT) and two theories of relationship development – social penetration theory [37,38] and stages of relational development [[39], [40], [41]]. In DIT, organizational diffusion is achieved through changes in management, policies, and procedures [[42], [43], [44], [45]] facilitated by increasing perceived need for workplace sun safety, demonstrating that policy/education fit the organizational mission/practices, and helping plan for policy/education, realizing that policies/education and/or the organization may need to change to improve fit and changes need to be clarified to managers/employees [[46], [47], [48]]. In meetings and follow-up communication with managers, program staff aimed to a) reduce managers' uncertainty [49,50] about sun safety; b) highlight advice from national health authorities to build credibility/trust [51,52]; and c) obtain commitment to adopt and implement the policies/education [[53], [54], [55]].

Operationally, SSW consisted of personal visits with managers and materials promoting sun protection policies and education [56]. It started with a program announcement packet and in-person meetings between senior managers and program staff that covered: 1) Introduction to SSW; 2) Sun Safety Practices in the Workplace; 3) Sun Safety Policy for Outdoor Workers; 4) Sun Safety Policy Adoption; 5) Sun Safety Policy Reinforcement and Maintenance. The SSW website was provided to workplaces with promotional materials on: skin cancer and UV, risk factors, personal risk assessment, workplace prevention strategies (i.e., sunscreen, sunglasses, long-sleeved shirts, hats, shade, scheduling), prevention at home, and online workplace audit and policy writing tool. Program staff conducted in-person Sun Safety Training to staff during worksite visits that presented: 1) The U.S. Skin Cancer Problem; 2) The Sun, UV Rays and Skin Cancer; 3) Assessing Your Personal Risk; and 4) Practicing Sun Safety. A Sun Safety Tool Box provided support materials for policy and education (i.e., worksite audit and facts sheets on sun safety and barriers to policy implementation and theory-based printed posters, risk assessment brochures, and tip cards for employees). In monthly follow-up contacts, staff sustained the relationship with managers, supported their decisions on sun safety policies and education, and problem-solved barriers [57,58].

2.5. Design of digital scale-up method

The digital scale-up method for distributing SSW aims to virtualize the in-person contacts, trainings, and policy tools, using conferencing technology, an online training platform, and the SSW website to reduce cost and expand reach, deliver standardized, engaging content, provide flexibility to fit the worksite's schedules and environments, and increase portability [59]. The digital method has the same goals and schedule, and similar components, as the original SSW and advocates that employers adopt comprehensive sun safety policies and employee education. The virtual meetings and follow-up communication are being conducted by multi-model synchronous web-based video conferencing (i.e., Zoom) or telephone conference call technology. In most cases, a single key contact manager meets with program staff but where possible staff seeks to meet with additional managers to enhance knowledge sharing, idea generation, synergistic conversations, and collaborative learning [60,61]. An online version of the Sun Safety Training was produced that could be used in groups or individually. The content was updated and expanded to address UV and skin cancer, personal skin cancer risk, sun protection practices on the job (e.g., UV Index, shade, protective clothing, hats, sunglasses and sunscreen), and skin self-examination. Finally, all printed materials, Sun Safety Tool Box, posters, risk assessment brochures, and tip cards were converted to a digital format and are available on the SSW website, where managers can download them for use with employees.

Initially, we planned to use a social media platform in the scale-up of SSW. However, all of the state DOTs had highly restrictive cybersecurity policies that prohibited the use of social media in the workplaces. Thus, this component of the digital method was eliminated.

2.6. Tailoring delivery on readiness to innovate

To improve communication with managers in both scale-up methods, procedures were developed to tailor the communication with managers to their readiness to adopt occupational sun safety, based on DIT and a protocol developed in another project: Agenda Setting (i.e., need for sun safety), Matching (i.e., fit of sun safety with existing policy/procedures), Structuring (i.e., initial implementation of policies/actions), and Clarifying (i.e., communicating with employees, other managers, and clients to garner support and counter opposition). By tailoring on readiness, program staff connects with districts where they are in the innovation process so they provide the most relevant coaching and resources. A tailored report on the DOT's current support and actions for occupational sun safety is prepared for state-level key contacts based on the existing sun safety content in their written policies and managers' responses to a pretest survey and a checklist of workplace sun safety actions they complete during the first meeting. The report presents a plan for implementing sun safety for employees with resources matched to state contact's readiness.

2.7. Outcome measures

2.7.1. SSW implementation

SSW implementation is the primary outcome in the cost analysis. A primary measure will be managers' reports of any training or other sun safety actions, using measures modified from the trial testing the effectiveness of SSW and implementation of school sun safety policy [62,63]. Managers will report if training was provided to employees, and to supervisors and managers:

• 1.Training on the health risks of sun exposure is provided to employees
• 2.Training regarding sun safety is delivered to managers and supervisors

They also report on nine sun protection actions in the workplace:

• 1.Employer monitors UV Index and work scheduled is adjusted for harm associated with UV level.
• 2.Employees wear UV-protective clothing or uniforms (shirts with sleeves; long pants), hats, and/or eyewear when outdoors.
• 3.Employees wear sunscreen with SPF 30 or greater when outdoors.
• 4.Messages are communicated to employees about protecting their skin and eyes from the sun while outdoors at work.
• 5.Employer provides sun protection resources, such as sunscreen, UV-protective clothing/ uniforms, hats, and/or eyewear to employees. “Provides” means your employer gives employees these items or gives employees money to purchase them for use at work
• 6.Temporary or permanent shade structures are provided in the work environment.
• 7.Employer requests that staff employed by contractors/subcontractors comply with the sun safety policy while working in my employer's work environments.
• 8.Employer encourages employees to regularly check their skin for signs of skin cancer either by themselves or by a physician.
• 9.Employer conducts a risk assessment of sun exposure and sun protection for employees in the work environment

For each of these items, managers respond yes, no, or don't know.

At posttest, employees will report whether they received the sun safety training and whether employers communicated about sun safety on the job through oral, written, or electronic messages. This will be a secondary measure, validating managers' reports. In our prior research, managers' and employees' reports of sun safety education were highly correlated [36].

2.7.2. Employees' sun protection

The secondary outcome is employees' sun protection at work assessed at posttest. Employees will report frequency of sun protection practices on 5-point frequency scales, including sunscreen with SPF 15 or greater and with SPF 30 or greater, sun-protective clothing such as long-sleeved shirts and long pants, hat with any type of brim and with a wide-brim all the way around (including hat with flap on the back the protects ears and neck), sunglasses, shade use, limit exposure to midday sun, and have sunscreen, hat and eye protection at work at all times (1 = never, 5 = always). They will also report on prevalence of sunburn in the past 12 months anywhere and on the job (yes/no; number of times). These are standard, validated, and reliable measures from past studies [[64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74]]. Managers will complete these measures, too.

2.7.3. Policy measures

Given that policy is set by the central state DOT office, written sun protection policy was changed to be a potential effect moderator. Human resources and safety documents were excerpted and coded by research assistants blind to condition at pretest. Composite scores will assess presence, strength, intent, and responsibility from a coding protocol developed previously [9]. Specifically, it assesses 15 policy categories: engineering controls (physical environment of the workplace), administrative controls (workplace procedures), and employee education (workers' sun safety). Each category receives a point (0,1) for presence (total score = 15) and a 3-level strength score (−1 = not allowed, 0 = not mentioned, 1 = allow/ recommend, 2 = require; total score = 30). Policies on engineering controls (scheduling/shade) and sun safety practices (hats/protective clothing/protective eyewear) could exist for reasons other than sun safety (e.g., to prevent injury), so those categories receive a point (0,1) when sun protection is explicitly cited (total intent score = 5). A 3-level responsibility score is assigned, noting who provides protection equipment (0 = not specified, 1 = employee, 2 = employer, total score = 10). These ordinal composite scores are continuous and summed across categories. Training ensured inter-coder reliability exceeds 0.70. There is no basis for the minimum number of components that must be changed to improve workplace sun safety, so we will assess both presence and extent of change. The ordinal measures can detect expansion of existing policies. The proportion of employers who have any policy will be estimated. We will also assess extent (number of elements) and strength (−1 = not allow, 0 = not mentioned, 1 = allow/ recommend, 2 = require summed across policy components) of policy.

2.7.4. Cost measures

Costs associated with the in-person and digital scale-up methods will be recorded. Using a micro-costing approach, resource allocations for each component of the scale-up methods will be identified (e.g., in-person and virtual visits; follow-up communications; staff-delivered and virtual training, web resources, and mailed materials). Both labor and non-labor elements associated with in-person and virtual meetings, staff-delivered and virtual trainings, printed materials, follow-up communications, and website resources, and induced employer costs of policy and education implementation, will be recorded. For project personnel, the proportion of FTE across activities will be estimated by contemporaneous staff self-report into categories (e.g., recruitment, intervention development, intervention delivery). In posttest surveys, senior managers will identify organizational sun safety actions induced by SSW and attach cost estimates to each (i.e., hours spent by manager and employees implementing action; items purchased to implement actions) [75]. Costs will be summed to produce overall cost estimates.

2.7.5. Manager and employee awareness and attitudes toward occupational sun protection

Managers' attitudes toward skin cancer prevention and occupational sun protection and awareness of sun protection policies and procedures are assessed in the managers' pretest and posttest survey. Perceived susceptibility to skin cancer is assessed by two Likert-type items for self and employees within the organization (1 = strongly disagree; 5 = strongly agree). Support for occupational sun protection is measured by two Likert-type questions: I personally support adopting policies to protect employees from the sun at my organization and Employees should take action on their own to protect themselves from the sun while at work, without my organization telling them to do so (reverse coded). Managers rate occupational sun protection (i.e., changing formal written policies, procedures, and training in my organization to protect employees' skin and eyes from the sun) on six innovation characteristics suggested by DIT: [[42], [43], [44], [45]] necessary, too expensive, compatible with work procedures, too complicated, acceptable to employees, and would improve organization's existing risk management or employee wellness programs (1 = strongly disagree; 5 = strongly agree). Managers and employees report if the organization has a formal written policy, administrative procedure, or training standard on sun protection of employees and unwritten informal standard operating procedures on sun safety (i.e., presence and content of procedures). Those aware of a policy also indicate degree of acceptance (How much do employees that report to you agree with the formal written policy; Do I agree with the formal written policy [1 = not at all; 3 = very much]) and compliance (How much are the employees that report to me following the policy; Do I follow the policy [1 = not at all; 5 = completely]) with the policy on sun protection.

2.7.6. Measures of potential effect moderators

Community and organizational characteristics are being collected and tested as moderators of the scale-up methods. At the community level, population size, education, and socioeconomic status are being obtained from the U.S. Census. Mean annual hours of sunshine will be recorded from the National Weather Service. For each state DOT and regional district, number of employees and job type (e.g., public works, public safety, etc.) are collected.

Also, potential moderators are collected in manager pretest survey. Job characteristics assessed include years working for state DOT; direct supervision of day-to-day activities of employees who work outdoors; and frequency of being involved in decisions about workplace policy, procedures, and training related to safety and health of employees (1 = never; 5 = all of the time). Based on DIT [[42], [43], [44], [45]], managers' opinion leadership (I am frequently asked to give my opinion concerning safety and health of employees by other people [1 = strongly disagree; 5 = strongly agree]) [76] and cosmopolite-ness (readership of printed or online publications for professional in your field; attendance at statewide, regional, or international professional conferences in past 5 years; and awareness of other public organizations like yours that have policies, rules, or standard operating procedures intended to improve the sun protection of employees) [77] are assessed. Managers report on the amount outdoor work: Whether they work mostly outdoor, mostly indoors, or outdoors and indoors equally; and number of hours spent working outdoors in a typical week in the summer (April to October) and winter (November to March). Their tanning desires are measured by an item asking if they thought they looked better with a tan (1 = strongly disagree; 5 = strongly agree [78]). Managers are also asked whether the organization has any policy that makes it more difficult for employees to use sun protection. Finally, skin type (always burn and unable to tan; usually burns but can tan if work at it; sometimes mildly burns and then tans easily; rarely burns and tans easily) [79], skin cancer history and demographic characteristics of managers and employees are collected: age, education, Hispanic ethnicity, race, and gender. We plan to assess job characteristics, outdoor work and skin type, skin cancer history, and demographic characteristics in posttest surveys of employees, too.

2.7.7. Sample size

2.7.7.1. Cost analysis

The primary analysis will model the cost-effectiveness of digital relative to in-person scale-up strategy, based on the estimated rate of employers (i.e., regional districts) who implement policies and education and the cost of delivering each intervention. Secondary analyses will explore the representativeness of districts who implement the interventions. The proportion of regional districts adopting sun safety education in the trial establishing the effectiveness of SSW was used to estimate the sample size of districts. To be conservative, we expected implementation to be slightly lower in the in-person scale-up group than in the original trial and designed the sample size of the digital scale-up group to be larger, on the possibility it produces a lower implementation rate than in-person method. In the SSW trial, the proportion of districts implementing education was 0.80 (40 of 50), but we assumed that in-person scale-up method will have a lower rate of implementing education of 0.600. We further expected up to 10% of the organizations to be unavailable/refuse to provide follow-up data. A sample of 40 regional districts assigned to the in-person group, with 36 providing full data, allows for 95% confidence intervals (CI) for proportions of 0.424 to 0.759 when true proportion of districts providing education is 0.600. For the digital scale-up method, the sample was designed to have similar precision even if the proportion implementing education is one-tenth as large as the in-person group, i.e., 0.060 implementing education. A sample of 100 districts, with 90 providing full data, assigned to digital scale-up group allows for 95% CI for proportions of 0.021 to 0.131 when the true proportion providing education is 0.060 using exact clopper pearson calculations. Comparisons of education have >80% power to test implementation rate differences between interventions. The smallest sample size would be 42 and 84 respectively, but 84 does not provide for an estimated nonzero lower limit of the 95% CI for implementation, a key figure for calculating cost-effectiveness. Thus, the sample size was increased to provide a nonzero lower limit of the 95% CI. We invited all eligible managers in the DOT regional districts, with the aim of pretesting 840 (6 per district) so we could reasonably expect to posttest 672 (5 per district; 80% follow-up).

2.7.7.2. Hypothesis test

We estimated the power for the hypothesis test for comparison between the two scale-up methods on proportion implementing employee education and other sun safety actions, with the sample sizes of n = 40 in the in-person scale-up group and n = 100 in the digital scale-up group. As noted, we assume the proportion of districts in the in-person scale-up group who implement education is 0.600. The rate may be lower in the digital scale-up group but by how much is unknown. Thus, based on the rate of 0.600 for districts in the in-person group, we estimated power for a range of proportions implementing policy in digital group for a low n = 36 for in-person and n = 90 for digital (expecting 90% with complete data in our original sample size calculations). In the actual enrolled population, power for comparisons exceeds 80% for proportions of 0.300 in the in-person group versus 0.060 in digital group, even if the resulting sample sizes are only 40 and 80 respectively.

2.7.7.3. Research question

Finally, for employees' reported sun protection practices, sample size is adjusted for the design effect due to clustering within DOT regional districts (i.e., intra-class correlation [ICC]), which inflates variance and Type I error, making significance testing too liberal. We assumed 50 employees per regional district on average and a total of n = 2,300 in the in-person scale-up method (n = 46 regional districts) and n = 4550 in the digital scale-up method at posttest. Adjusting for an ICC of up to 0.02, which usually is much lower for individual behaviors, reduces effective sample size to n = 1079 in the in-person method and n = 2614 in the digital method, but a small difference in proportions of 0.60 to 0.65 is detectable with 80% power.

2.7.8. Analysis methods

2.7.8.1. Cost-effectiveness analysis

The economic evaluation will explore whether SSW can be delivered cost-effectively by a digital scale-up method to worksites producing a lower implementation rate than by an in-person method but at substantially lower cost. To begin the cost analysis, we will estimate implementation of education in two ways, i.e., if at least 1 manager reports training for districts and the average number of managers who report it. We will create similar measures of implementation of other sun safety actions, including if at least 1 manager reports implementing the action and the average number of managers who report it. Implementation will be defined conservatively, i.e., district both delivers training and other sun safety actions, and liberally, i.e., either training or other sun safety actions. Next, we will use an incremental cost-effectiveness ratio (ICER) to summarize the economic effects of the digital scale-up method (“intervention”) relative to the in-person scale-up method (“control”). The ICER is the incremental program cost (C) of digital method per incremental change in implementation rate (E [for effect]): ICER = (C _digital – C _in-person ) / (E _digital – E _in-person ). In Fig. 1 , digital scale-up is cost-effective relative to in-person if the ICER is in the gray area to the right of the dashed line, which represents the dollar value of an additional unit of effect (i.e., either maximum willingness to pay (WTP) for an additional implementing worksite or minimum willingness to accept (WTA) cost savings in lieu of a worksite that does not implement). If the ICER is to the left of the dashed line, in-person scale-up is considered cost-effective relative to digital. If digital scale up were both less costly and more effective than in-person scale-up, a negative ICER would imply economic “dominance” of digital over in-person. Conversely, a positive ICER (e.g., both numerator and denominator negative) would indicate the average cost savings for a non-implementing worksite that would have implemented sun safety policy under in-person scale-up but not under digital scale-up. In this case, cost-effectiveness depends on one's willingness to accept a given level of digital scale-up cost savings from a worksite not implementing at or below its cost savings (see Fig. 1). We will not have access to WTP/WTA data from participating agencies or worksites, so we cannot state definitively that either digital or in-person scale-up methods will be cost-effective relative to the other. However, we will strive to validate our ICER results using external comparators (e.g., budgets for known implemented health-related policies and programs). Note that this is not an analysis of SSW simply delivered “as is” to more worksites. Content remains the same, but the delivery mechanism and associated resources differ between the two scale-up methods.

Fig. 1.

Approach to Cost-effectiveness Analysis Comparing Digital to In-person Scale-up Strategy.

For policy purposes, development costs will be considered “sunk” and excluded, so the economic evaluation will be that of an existing program. However, to the extent possible, digital method development costs will be tracked and reported as appropriate. Research and evaluation costs will be removed. Univariate and multivariate sensitivity analyses will model representativeness. Regional district size will be examined to evaluate the digital scale-up method's ability to (cost-effectively) contact individual workers. It may be cost-effective but if mostly small districts implement, digital scale-up's representativeness (i.e., “reach”) is limited among individual workers. Further, we will calculate ICERs within each of the five U.S. Census regions, with similar ratios indicating representativeness across regions. Our cost analysis will not include or account for potential feedback effects, such as business interactions that expose other worksites to the digital method or long-term health outcomes, such as reduced skin cancer incidence.

We will extend the economic analysis by recalculating ICERs based on number of employees that report pre-post improvement in sun protection (see below). This analysis will address effectiveness, i.e., program cost required to motivate positive behavior change by employees.

2.7.8.2. Statistical analysis for hypothesis testing

Analyses will also test the hypothesis that the digital scale-up method would result in less implementation of training and other sun safety actions by the district than the in-person method. The unit of analysis will be the district, using logistic regression analyses and Poisson regression (or negative binomial regression, if the assumptions of the Poisson model is not fulfilled) (i.e., Proc MIXED, Proc GLIMMIX), adjusting for significant (p < 0.10) covariates. Representativeness of implementation achieved by the digital method will be explored by adding to the regression models variables such as size and U.S. Census region and other moderators potentially affecting implementation. These variables will be added as fixed factors and their interactions with condition examined at p = 0.05 (2-tailed), unadjusted for multiple comparisons. Using the correlation structure from the primary outcome variables, we will assess order effects to determine which components may drive the implementation of policy and education.

Districts' engagement with the two scale-up methods may differ and affect implementation rates achieved by them. We will explore mediation of education and other sun safety actions implementation rates by program exposure assessed by process measures as ancillary analyses, using mediational analyses from Judd and Kenny [80] and elaborated by MacKinnon [81]. To satisfy the conclusion of mediation, significance will be determined via interval estimation using a bootstrap estimate [82]. Ratio of indirect to total effect will estimate proportion of effect mediated by engagement.

3. Results

3.1. Characteristics of participating state departments of transportation

DOTs in 21 states agreed to participate (see map in Fig. 2 ). The DOTs ranged in size from 997 to 18,415 employees and were located in four Census regions (7 South, 6 Midwest, 4 Northeast, 4 West). There were 138 regional districts from the state DOTs enrolled, pretested, and randomized (range = 1 to 27 per state). Randomization resulted in 46 districts being assigned to the in-person scale-up method and 92 to the digital scale-up method. One state had a single district and was assigned to the in-person scale-up strategy.

Fig. 2.

Participating State Departments of Transportation (n = 21).

3.2. Characteristics of regional district managers

A total of 1113 managers (49.2%) in 138 regional DOT districts completed the baseline survey out of 2262 managers invited. The managers' characteristics are displayed in Table 1 . As would be expected for senior staff, they are generally long-term employees (nearly 20 years on average), almost all make decisions about workplace safety and health policy, procedures, and training, and serve as opinion leaders on those topics. Many regularly read professional publications and attend conferences, which can expose them to innovative ideas such as occupational sun safety. Nearly all managers are male. Education is widely distributed, with one-fifth having a high school education, one-third some education beyond high school, and 43.4% a college degree. Many of these managers work outdoors at least part time on the job (66.5%). Nearly a quarter (24.4%) have skin types at high-risk for skin cancer and several have been diagnosed with skin cancer. This is not too surprising given that nearly all managers are white (<9% another race and only 5.6% Hispanic). Sun tans are viewed favorably by some managers.

Table 1.

Baseline characteristics of the participating regional district managers at baseline.

Characteristic	Overall Sample
N	1113
Years working for state DOT	19.77 (9.33)
Involvement in making decisions about workplace safety/health policy, procedures, and training (mean = 3.56, sd = 1.12) All/most of the time Some of time Rarely/never	50.5% 33.3% 16.2%
Working outdoors Mostly outdoors Outdoors and indoors equally Mostly indoors Hours spend working outdoors in a typical week In summer ((April to October) In winter (November to March)	24.9% 41.6% 33.5% 17.18 (12.15) 13.41 (11.52)
Innovativeness I am frequently asked to give my opinion concerning safety and health of employees by other people (innovativeness)1	3.41 (1.10)
Cosmopolite-ness Regularly read any printed or online publications for professionals in your field Yes No/don't know Attended a statewide, regional, national, or international professional conference for your organization in the past 5 years Yes No/don't know	58.6% 41.4% 67.9% 32.1%
Tanning desirability (I think I look better with a tan)	3.24 (0.90)
Age (mean, SD years)	48.88 (8.46)
Education2 High school graduate or less Education beyond high school 4-year college graduate or more	21.9% 34.7% 43.4%
Skin type Always burn and unable to tan Usually burns but can tan if work at it Sometimes mildly burns and then tans easily Rarely burns and tans easily	4.9% 19.5% 44.9% 30.7%
Skin cancer history Yes No/don't know	10.2% 89.8%
Hispanic ethnicity Hispanic/Latino Not Hispanic/Latino	5.6% 94.4%
Race/ethnicity American Indian/Alaska Native Asian Black/African American Native Hawaiian/Other Pacific Islander White More than 1 race	1.5% 1.0% 5.2% 0.0% 91.1% 1.2%
Gender Male Female Other	91.5% 8.5% 0.0%

¹Education beyond high school includes technical or vocational education beyond high school, some college education, or two-year college graduate.

²1 = strongly disagree; 5 = strongly agree.

Randomization successfully balanced the two conditions on managers characteristics in the pretest survey, with only two variables showing differences by treatment group. Managers in the in-person scale-up group spent more time working outdoors (least square means 14.50 h, standard error 1.30 h) than managers in the digital scale-up group (least square means 12.77 h, standard error 1.25 h) (F = 5.98, p = 0.015).

3.3. Occupational sun protection

Managers were supportive of occupational sun safety (Table 2 ). Managers felt personally vulnerable to skin cancer (51.6% strongly agree or agree) and that employees were at risk for getting skin cancer (65.3% strongly agree or agree). They strongly supported adopting policies on occupational sun protection at their organization (85.5% strongly agree or agree). However, most managers also felt that employees should take responsibility for sun safety at work without the organization telling them to do so (83.1% strongly agree or agree). Managers considered the innovation characteristics of occupational sun safety suggested by DIT to be favorable, including being necessary (64.7% strongly agree or agree), compatible with work procedures (71.9% strongly agree or agree), not too expensive (93.2 strongly disagree, disagree, or neutral), not too complicated (92.2% strongly disagree, disagree, or neutral), acceptable to employees (64.5% strongly agree or agree), and improve on the organization's existing risk management or employee wellness programs (72.3% strongly agree or agree).

Table 2.

Policy, training, and other sun safety actions reported by regional district managers at baseline.

Characteristic	Overall Sample
N	1113
Occupational Sun Safety Risk and Support:
Susceptibility to skin cancer1	3.55 (0.88)
I personally support adopting policies to protect employees from the sun at my organization2	4.16 (0.83)
Employees should take action on their own to protect themselves from the sun while at work, without my organization telling them to do so.2	4.12 (0.90)
Awareness of Policy and Procedures:
Organization has a formal written policy, administrative procedure, or training standard on sun protection for its employees Yes No/don't know	30.5% 69.6%
How much do employees that report to you agree with the formal written policy, administrative procedure, or training standard on sun protection for employees3 (n = 328 managers reporting a policy)	2.32 (0.52)
How are the employees that report to you following the policy4 (n = 328 managers reporting a policy)	3.51 (0.75)
Organization has unwritten standard operating administrative, or training procedure, or any other unwritten rules, intended to improve sun protection of employees Yes No/don't know	40.2% 59.8%
Aware of any other public organizations like yours that have policies, rules, or standard operating procedures intended to improve the sun protection of employees Yes No/don't know	8.7% 91.3%
Innovation Characteristics of Occupational Sun Safety:1
Necessary	3.69 (0.93)
Too expensive	2.40 (0.84)
Compatible with work procedures	3.79 (0.75)
Too complicated	2.37 (0.81)
Would be acceptable to employees	3.63 (0.83)
Would improve my organization's existing risk management or employee wellness programs	3.85 (0.76)
Composite innovation characteristics5	3.70 (0.59)
Training on Occupational Sun Safety:
Training on the health risks of sun exposure is provided to employees	48.1%
Training regarding sun safety is delivered to managers and supervisors	40.2%
Composite measure of sun safety training6	51.5%
Workplace Sun Safety Actions:
The UV Index is monitored and outdoor work activities are modified with regard to risk of harm associated with the UV Index level	9.3%
Employees wear UV-protective clothing or uniforms (shirts with sleeves; long pants), hats, and/or eyewear when outdoors	40.1%
Employees wear sunscreen with an SPF of 30 or greater when outdoors	25.0%
Temporary or permanent shade structures are provided in the work environment	16.5%
Messages are communicated to employees about protecting their skin and eyes from the sun while outdoors at work.	65.7%
My employer requests that staff employed by contractors/subcontractors comply with the sun safety policy while working in my employer's work environments	8.3%
My employer provides sun protection resources, such as sunscreen, UV-protective clothing/ uniforms, hats, and/or eyewear to employees. “Provides” means your employer gives employees these items or gives employees money to purchase them for use at work	59.5%
My employer encourages employees to regularly check their skin for signs of skin cancer either by themselves or by a physician	26.0%
My employer conducts a risk assessment of sun exposure and sun protection for employees in the work environment	7.8%
Composite measure of sun safety actions7	2.54 (2.03)

¹Combined scale: Compared to people working in other organizations or companies in the United States, I am more likely to get skin cancer in my lifetime and the employees in my organization are more likely to get skin cancer in their lifetime compared to people working in other organizations and companies in the United States (1 = strongly disagree; 5 = strongly agree; Cronbach's alpha = 0.77 (standardized).

²1 = strongly disagree; 5 = strongly agree.

³1 = not at all; 2 = somewhat; 3 = very much.

⁴1 = not at all; 2 = a little; 3 = somewhat; 4 = mostly; 5 = completely.

⁵Cronbach's alpha =0.81 (standardized).

⁶Cronbach's alpha = 0.83 (standardized).

⁷Cronbach's alpha = 0.74 (standardized).

Only a minority of managers reported that there was a written sun protection policy or unwritten standard operating, administrative, or training procedures at their workplace (Table 2). Also, they felt that employees generally agreed with and followed the sun protection policy or procedures, when it was present. Further, training and other actions to support occupational sun protection were occurring in some workplaces (Table 2). About half of managers reported that training on sun protection on the job occurred with employees and/or managers. Two-thirds of managers said that messages on sun protection were conveyed to employees and three-fifths said that their employer provided personal protection resources such as sunscreen, protective clothing, hats, or eyewear to employees (i.e., employers gives these resources to employees or provides money for employees to purchase them). However, managers felt that only a minority of employees were using sunscreen or sun-protective protective clothing. Very few managers reported other occupational sun protection actions such as providing shade in the work environment, monitoring the UV Index and modifing work activities for UV levels, asking contractors/subcontractors to comply with sun safety policy in the organization's work environment, conducting risk assessments of sun exposure and sun protection, or encouraging employees to get regular skin checks with self- or clinical exams. Overall frequency of these other sun protection actions averaged 2.54 out of 9.

Randomization balanced the treatment groups on baseline occupational sun protection training and actions. The only apparent difference was that more managers (23.9%) in the in-person scale-up group reported that their organizations encourage employees to regularly check their skin for signs of skin cancer either by themselves or by a physician than managers (17.4%) in the digital group (F = 4.79, p = 0.031). There were no differences in policy awareness and support, education of employees, or other sun safety actions by treatment group.

3.4. Personal sun protection practices

Managers were inconsistent in their sun protection practices (Table 3 ), with only a mid-range value on the composite sun protection score. The most frequent personal protection for sun exposure was wearing sunglasses (80.6% always or often) and a hat with any type of brim (65.0% always or often). However, wide-brim hats were not commonly worn by managers (23.3% always or often) and only a minority wore sunscreen routinely (28.4% always or often on the face and 34.1% on exposed body parts). Still, many reported that they had sunscreen, a hat and eye protection with them at work when outdoors (65.7% always or often). Sun-protective clothing (35.5% always or often) and shade (42.0% always or often) also were used less frequently by managers. Few managers limited their time outdoors during midday (24.4% always or often), the daily period of highest UV.

Table 3.

Sun protection practices and sunburn prevalence reported by regional district managers at baseline.

Characteristic	Overall Sample
N	1113
Sun Protection Practices (mean, sd)1
Sunscreen with an SPF 15 or greater on my face through the use of aftershave, face lotion or make-up.	2.61 (1.33)
Sunscreen with an SPF 30 or greater on my exposed body parts (not including aftershave, face lotion or make-up).	2.98 (1.12)
Clothing specifically to protect my skin from the sun, such as long-sleeved shirts and pants.	3.06 (1.10)
Hat with a wide-brim all the way around (this includes a hat with a flap in the back that protects the ears and the neck).	2.48 (1.24)
Hat with any type of brim.	3.78 (1.08)
Sunglasses.	4.18 (1.02)
Limit my exposure to the sun during the mid-day hours.	2.84 (1.00)
Stay mostly in the shade when it's available.	3.29 (0.96)
Have sunscreen, a hat, and eye protection with me at work when I am outdoors.	3.79 (1.01)
Composite sun protection score2	3.16 (0.65)
Sunburn
Number of sunburns in past year (mean, sd)	1.62 (2.99)
Number of sunburns in past year that occurred while working outdoors (mean, sd)	1.10 (2.56)

¹1 = never, 2 = rarely, 3 = sometimes, 4 = often, and 5 = always.

²Cronbach's alpha (standardized) = 0.73.

Further, over half of managers had been sunburned in the past year. Many reported that this over-exposure occurred at work, with two-fifths reporting that they were sunburned at least once in the past 12 months while working (Table 3).

4. Discussion

Occupational sun exposure receives limited attention [83,84], despite its link to skin cancer [[14], [15], [16]]. Health promotion approaches that incorporate policy with education such as in our S SW intervention can impact safety, disease management [85], and disease prevention [[86], [87], [88], [89], [90], [91], [92], [93], [94], [95], [96], [97], [98], [99]] by working synergistically to integrate health promotion and safety procedures [85], clarify personal/organizational responsibilities (e.g., who provides sunscreen and protective clothing), and formally direct employees to take precautions to overcome low perceived risk, personal preferences (say for tanning), and other barriers, as well as equalize gender and age differences in health practices [12,100,101].

Our team has successfully recruited two-fifths of the state DOTs in the United States to participate in the randomized trial on the cost effectiveness of two scale-up methods. These departments employ over 100,000 workers, many of whom work outdoors. As the managers reported, employees work for many years at the state DOTs (managers average nearly 20 years) and spend substantial time outdoors both in the summer and winter, making them vulnerable to solar UV skin damage including skin cancer. The focus on a single industry provides some measure of control against confounding of organizational characteristics but the 21 states are diverse in terms of climate and ambient UV levels, and racial composition of the workforce, that alter their approach to on-the-job sun protection. The sample of managers in this trial is similar to the one we recruited from cities, counties, and special taxing districts in Colorado, with the exception that they were longer-term employees, more variable on education, and more predominately male.

A very pronounced knowledge-practice gap appears to exist in occupational sun safety in these organizations, according to reports from managers. Many of the managers recognized that they and their employees are at risk for skin cancer and are supportive of occupational sun safety, but only some managers said their organizations were taking steps to address it. Only half of managers said training was occurring, but two-thirds said the organization was communicating about sun safety with employees. Employees were more likely to receive training than managers. This may be an oversight given that managers are spending time outdoors on-the-job and training may motivate them to be opinion leaders for occupational sun safety. Two-thirds of managers said that organizations were providing personal protection resources, but most employees were not using them according to managers. Training and communication to employees may need to be expanded to get more use of personal protection. Further, the organizations were not taking advantage of the UV Index to alert employees or adjust work schedules for the risks posed by high UV, although we have found in the past that adjusting work schedules is one of the least likely actions by employers. In this case, concerns over safety of employees (i.e., motorist can see them better during daylight) may favor daytime work schedules. The organizations also are not providing shade, although this too may be challenging for employees who are traveling long distances to monitor and work on roadways. Still, shade canopies can be mounted on some equipment (e.g., tractors) and vehicles and other environmental objects such as trees, underpasses, and buildings cast shade and can be used during breaks to reduce UV exposure (although auto glass blocks UVB better than UVA).

Still, the initial support for occupational sun protection and the fact that state DOTs were taking some actions on sun safety bodes well for the successful scale-up and implementation of our evidence-based sun safety intervention. There does not appear to be strong resistance from the management of DOT regional districts to occupational sun safety. Rather, it may be that sun safety is a lower priority than other safety issues such as safe driving practices, fall prevention, hazardous materials, and construction site safety. Also, well-designed training and resources for sun safety at a reasonable cost may not be widely available to risk management and training staff.

Information on costs (and intervention delivery resources) is essential for moving research into practice [8,28] and under-reported in dissemination and implementation [8]. Cost will undoubtedly be relevant when attempting to reach and serve these very large organizations in the trial that maintain transportation infrastructure of over 1.9 million square miles. The size of the regional districts makes them more manageable organizational units for testing the scale-up methods. However, the regional districts do pose many challenges to implementation of occupational sun safety, with most districts having employees based in multiple locations, from central offices and yards to remote maintenance facilities. Travel costs are substantial to reach employees with in-person training and work with local managers across locations within these districts to take other actions on sun safety. Digital communication can reach the outlying locations for far lower costs; however, digital communication may not be easy with some of these districts. Many locations of remote yards are small and in very rural areas, without a lot of technology resources. Many jobs require employees to travel to outlying locations and rely on using vehicles and heavy equipment, not work in offices and locations where computers are readily available. However, these circumstances are true for many outdoor jobs, increasing the potential generalizability of the study. Facilitating implementation is the presence of formal risk management and training personnel and procedures in all of the state DOTs, which was typically the initial point of contact for the study.

The trial design has a number of strengths and weaknesses. The participation of a large proportion of a major U.S. public sector industry adds to the realistic nature of the test and generalizability. The prospective, randomized trial design is rigorous and will control several confounders among the regional districts. The multiple levels of analysis, from policy coding to surveys of managers and employees will provide a more comprehensive assessment of the scale-up processes and impact. The scale-up methods are based on a theoretical framework used extensively in the implementation sciences and the assessment of cost, as well as impact on implementation activities and employee behavior, will provide key information for decision makers charged with distributing evidence-based programs and adopting them with at-risk groups.

However, the study is limited by a single industry, although it does have a variety of outdoor occupations, and the fact that it is a public rather than private employer. Public employers may have longer term employees, have less concerns over profits that limit resources for health programs, and provide other types of occupations. Public employers supported by taxes also can experience budget constraints. The implementation and some of the cost measures will be reported by managers, rather than obtained from observations, which are impractical in this geographically large study area but which are subject to social desirability biases, demand effects, and memory errors. We will conduct a small-scale validation study on these reports. State DOTs centralize some of the risk management functions, so the regional districts will vary in their abilities to make independent decisions on all sun safety actions.

This paper was written during the first months of the COVID-19 pandemic in the United States. While DOTs were classified as essential employers in all states, all DOTs suspended in-person training and out-of-town travel was inadvisable during stay-at-home orders in many states. Thus, the in-person scale up method has been suspended during the spring and summer 2020. The digital scale-up procedures have continued in a very limited number of districts where managers were still willing to engage with our intervention staff. As states transition to safer-at-home and other forms of opening, the scale-up methods will be re-started. We should be able to intervene and posttest in 10 states with little interruption to the study timeline but posttesting will be delayed in 11 states to provide sufficient time to implement scale up procedures through the spring and summer 2021. We will handle this deviation in intervention duration by including a covariate for the time between baseline and posttest period and use least squares means for the adjusted post analysis that accounts for possible attenuation of effects due to the longer interval of time.

5. Conclusions

Evidence-based programs have been successfully scaled-up for HIV treatment/prevention [102,103], obesity prevention [3,104], and health service delivery [103,105], yet a research-to-practice gap remains between effective programs and real world application [26]. It is hoped that this trial identifies cost-effective methods for disseminating program widescale and help to realizing their promised benefits of the S SW intervention to fulfill one of the priorities of the Surgeon General in his recent call to action for skin cancer prevention [106].