Process Evaluation of Two Participatory Approaches: Implementing Total Worker Health^® Interventions in a Correctional Workforce

Abstract

Background

Correctional Officers (COs) have among the highest injury rates and poorest health of all the public safety occupations. The HITEC-2 (Health Improvement Through Employee Control-2) study uses Participatory Action Research (PAR) to design and implement interventions to improve health and safety of COs.

Method

HITEC-2 compared two different types of participatory program, a CO-only “Design Team” (DT) and “Kaizen Event Teams” (KET) of COs and supervisors, to determine differences in implementation process and outcomes. The Program Evaluation Rating Sheet (PERS) was developed to document and evaluate program implementation.

Results

Both programs yielded successful and unsuccessful interventions, dependent upon team-, facility-, organizational, state-, facilitator-, and intervention-level factors.

Conclusions

PAR in corrections, and possibly other sectors, depends upon factors including participation, leadership, continuity and timing, resilience, and financial circumstances. The new PERS instrument may be useful in other sectors to assist in assessing intervention success.

INTRODUCTION

For the half million correctional officers in the United States [Bureau of Labor Statistics, 2014], the work environment is exceptionally stressful due to inmate interactions that carry the constant threat of confrontation or violence [Morgan, 2009]. Correctional officers (COs) work all shifts, including weekends and holidays, and are subject to mandatory overtime on short notice [Swenson, 2008]. The job requires sitting or standing in static positions on hard surfaces, for extended periods, interrupted by frequent carrying of heavy equipment and less frequent high intensity emergency responses, usually to inmate violence [Warren et al., 2015]. Poor indoor air quality, erratic temperature control, overcrowding, and high noise levels are also characteristic of the correctional setting [Martin et al., 2012; Bureau of Labor Statistics, 2014].

COs have among the highest injury rates and poorest health of all the public safety occupations [Brower, 2013; Bureau of Labor Statistics, 2014, 2015]. Their life expectancy has been estimated at between 58 and 59 years of age nationally [Cheek, 1982; Institute of Justice, 2005]. In Connecticut, the average life expectancy for active and retired correctional staff is 66 years. While Connecticut correctional workers are living on average 8 years longer than the life expectancy for COs nationally, their life span is 15 years shorter than that of other Connecticut residents [State of Connecticut Comptroller’s Office, 2015]. CO health has not been heavily researched, but studies have identified health problems including cardiovascular disease [Abdollahi, 2002; Morse et al., 2011], musculoskeletal disorders [Warren et al., 2015], alcohol and drug abuse [Morgan, 2009], high risk of suicide [Stack and Tsoudis, 1997; Tiesman et al., 2010; Konda et al., 2012], and high rates of anxiety and depression [Tiesman et al., 2010]. Other psychosocial decrements include job stress, burnout, and work-family conflict [Carlson and Thomas, 2006; Bourbonnais et al., 2007; Obidoa et al., 2011; Lambert et al., 2015].

In searching the literature for health interventions designed for COs, we found only two studies. Crawley and Crawley [2008] conducted qualitative research that helped the New Jersey Department of Corrections to develop a training program addressing work-family issues for new COs, and Booth et al. [2009] developed a cognitive behavioral stress management program for COs. Thus, there remains a paucity of studies targeting the improvement of CO health and well-being [Armstrong and Griffin, 2004; Brower, 2013]. Moreover, few programs designed to improve CO health have been evaluated for effectiveness [Brower, 2013]. Evidence-based interventions for other public safety sector workers, such as police and firefighters [Elliot et al., 2007; Kuehl et al., 2014], are possible models for CO interventions, but even those programs lack congruence with the specific challenges of the correctional workforce, such as working in a closed environment, the constant threat of violence from inmates, and public indifference or even disdain [Brower, 2013].

In the past, the state of Connecticut’s Department of Corrections (DOC) had attempted to improve CO health through conventional administratively driven workplace health promotion (WHP) that included various wellness-related committees and training initiatives. However, CO health problems were unresponsive to DOC’s administratively driven programs. Administratively driven WHP interventions tend to focus on individualized change in COs, but interventions that include organizational change show greater promise [Schaufeli and Peeters, 2000]. Such changes might include making overtime voluntary, improving shift schedules, rotating work assignments, and anticipating risks to emotional health [Finn, 2000]. Moreover, administratively driven WHP interventions typically do not permit the participation of front-line workers.

Participatory Action Research

Participatory Action Research (PAR) has been recognized for its effectiveness in the design, implementation, and sustainability of health interventions, particularly where source-based methods have failed [Henning et al., 2009; Punnett et al., 2009; Choi et al., 2011; McVicar et al., 2013], the “source” being the intervention developer or sponsor [Wandersman et al., 2008]. As a user-based model in which the people or groups that benefit from the developed intervention participate in the research process, PAR can accelerate the translation of effective interventions into practice [Wandersman et al., 2008]. PAR is based on three basic premises: (i) participation, from the individuals directly affected by a common problem; (ii) action, which involves constant critical reflection, awareness, confrontation, dialogue, and negotiation; and (iii) research, in which participants create and test new problem-solving processes to bring about meaningful and lasting change [Glassman and Erdem, 2014].

Participatory ergonomics (PE) programs are analogous to PAR but they are specific to occupational settings and do not necessarily advance research questions. PE programs directly involve workers as subject matter experts of their jobs [Kuorinka, 1997; Haines et al., 2002]. More recently, a combined approach that integrates health protection and health promotion, and is compatible with PE activities, has emerged as the Total Worker Health^® (TWH) Program of the U.S. National Institute for Occupational Safety and Health (NIOSH) [Henning et al., 2009; Punnett et al., 2009; Punnett et al., 2013; NIOSH, 2015].

Health Improvement Through Employee Control (HITEC)

In 2006, the Center for the Promotion of Health in the New England Workplace (CPH-NEW), a NIOSH Center of Excellence for Total Worker Health, launched HITEC (Health Improvement through Employee Control), an intervention research program aimed at improving CO well-being in Connecticut [Morse et al., 2011; Cherniack et al., 2015]. HITEC is a collaboration between the University of Connecticut (UConn) research team and DOC administration and labor organizations that has lasted for a decade and is ongoing. Findings from the HITEC-1 (2006–2011) study confirmed Connecticut’s COs have numerous health problems early in employment, with hypertension, obesity, depression, and musculoskeletal disorders exceeding levels in comparison populations in the first 3 years of work [Morse et al., 2011; Warren et al., 2015].

HITEC-1 compared two intervention strategies: one driven by DOC administration with the help of health professionals (i.e., top-down), and one driven by frontline COs that involved participatory intervention design (i.e., bottom-up). The top-down approach was ineffective in reducing health risks and had lower participation and CO acceptance than the bottom-up approach [Cherniack et al., 2015]. Because of the failure of the top-down administration-directed arm of HITEC-1, it was replaced by a modified Kaizen-type approach in HITEC-2, launched in 2012, using Participatory Action Research (PAR). Kaizen is a participatory approach that uses a structured continuous improvement method to quickly identify problems with work processes, and test and refine solutions for implementation [von Thiele Schwarz et al., 2015]. The kaizen approach was deemed sufficiently different from the bottom-up arm intervention, because of its resemblance to focused shorter term action programs used by DOC to address security problems (e.g., gang formation). It also had a joint labor-management structure, which permitted a degree of administrative involvement which had some positive effects (i.e., especially middle manager involvement) during HITEC-1 [Reeves et al., 2012].

HITEC-2’s Two Total Worker Health Programs

In HITEC-2, the two modified programs were implemented, each in a separate facility, and were compared to determine differences in both health outcomes and implementation. The study parameters were consistent with PAR and TWH concepts. First, in keeping with participatory methodology, CO participation was required for the design and implementation of the health and safety interventions. Second, interventions were required to target improvement in health by considering both the DOC work environment and organizational structure, and the behavior of individual COs [Cherniack et al., 2015].

One program featured Kaizen Event Teams (KET) that were multi-level (i.e., COs plus middle/senior managers such as lieutenants, captains, deputy wardens, wardens) and focused on one specific project for a defined 4-month duration, following which that KET team disbanded. The other program had a continuous participation Design Team (DT), consisting only of front-line COs, tasked with designing a series of four interventions without a fixed time limit or sequence over about a 4-year period. In the absence of direct managerial involvement, the DT interacted with a Facility Steering Committee (FSC) consisting of middle- and senior-level DOC managers that was responsible for providing the DT with oversight, support, and resources, as well as input on intervention feasibility.

This is primarily a methods paper that presents the development and use of an evaluation system for assessing process execution for a participatory intervention. Secondarily, it is applied explicitly to the HITEC-2 interventions in order to illustrate utility and encourage similar work by examining the relationship between the developed process domains and intervention effectiveness. The presentation of methods overlaps with actual program evaluation, and the authors have tried to clarify where overlap occurs.

MATERIALS AND METHODS

Study Design

Two correctional facilities in different geographical locations, herein Sites A (KET) and B (DT), were matched from all 18 DOC facilities, on the basis of workforce size, average age of COs, workers’ compensation claims, and readiness to change (established by survey). The UConn research team oversaw human subjects and data protection, administered surveys, and physical testing, and evaluated program effectiveness. The full study protocol was approved by the Institutional Review Board at the University of Connecticut Health Center.

General oversight of both the KET and DT programs was provided by a joint labor-management Study Wide Steering Committee (SWSC), which met bimonthly. The SWSC identified the four interventions using a PAR process [Cherniack et al., 2015], maintained uniformity of the study design across sites, reviewed study instruments, provided continuous staffing of committees and teams, secured staff release time for program activities, provided needed resources not available at the facility level, and acted as a liaison between DOC administration, facilities, and health intervention teams. Table I provides a comparison of the KET and DT program characteristics.

TABLE I.

Comparison of Kaizen Event Team and Design Team Program

Activity	KET program (site A)	DT program (site B)
Composition of teams	COs, wardens, administrators, specialists	COs only
Duration of interventions	<120 days	Determined by DT
Number of teams	Four separate KETs (one team to work on each of four intervention topics)	One single DT (one team to work on all four intervention topics)
Completion of four intervention topics	Yes	Yes
Sequence of four interventions	Determined by SWSC	Determined by DT
Program facilitator	Yes	Yes
Facility steering committee	Integrated within KET	Separate from DT/consultative
Study-wide steering committee	Yes	Yes

CO, Correctional Officer; KET, Kaizen event team; DT, design team; SWSC, study-wide steering committee.

Intervention planning tool

One challenge to PAR methodology is the tension between participant autonomy and structured validation in establishing study content. To promote standardization, both the KET and DT programs were required to (i) use the same intervention planning tool; and (ii) design interventions around a priori health priorities. The intervention planning tool used by both the KETs and DT was the “Intervention Design and Analysis Scorecard” (IDEAS) developed by CPH-NEW for their Health Workplace Participatory Program [HWPP; Robertson et al., 2013]. Its seven-step process is presented in Figure 1.

FIGURE 1.

IDEAS tool steps 1–7. IDEAS, intervention design and analysis scorecard.

There were differences in the application of IDEAS between the DT and KET. Site B’s DT completed steps 1–5 and then proposed three intervention designs to a Facility Steering Committee (FSC) to vet and refine potential interventions. The arrows in Figure 1 represent potential multiple iterations of steps 5–7 as the DT and the FSC negotiated ratings, revisions, and ongoing modifications. Site A (KET) had no FSC as supervisors were part of the KET, and the KETs completed steps 1–7 without upper-level facility oversight (see left column of Fig. 1).

Four intervention priorities

The KETs and DT were asked to design and implement interventions targeting four health priorities: physical environment/ergonomics, physical fitness, weight management/nutrition, and safety/injury reduction. The workplace ergonomics intervention, Building Improvement Linked to Design (BILD), was directed toward procurement policies and facility design. The Work to be Fit (W-2 BFIT) intervention targeted CO physical fitness. The Better Food through Education and Design (BFED) intervention focused on weight management through improved nutrition and work environment factors that affect eating patterns. The Structured Work-related Injury Prevention through Ergonomics (SWIPE) intervention focused on safety and reduced injury severity by addressing CO injury related to inmate incidents. The KETs and DT were provided a program facilitator (member of the UConn research team), held meetings during their work day at a mutually agreed upon time, had six to eight overall team members, and received input from middle and senior-level management.

Program Evaluation Framework

In order to evaluate and compare the two intervention programs, each with different types of participation and continuity, several quantitative domains were established and validated that do not appear to have precedence for workplace PAR. Outcome evaluation is a preferred approach for evaluating intervention efficacy and effectiveness, as well as for determining resource allocation [Saunders et al., 2005; Rabin et al., 2008]. In addition to gauging outcomes, comprehensive evaluation also assesses the implementation process to identify promoting or inhibiting factors [Wandersman et al., 2000].

Implementation process

Documentation and evaluation of the process for implementing continuous improvement participatory programs should include considerations such as stakeholder commitment of time and effort, needed resources, future planning, expert facilitation and support, feedback and feedforward mechanisms, adherence to training principles, structure and flexibility, and macro-ergonomics [Haims and Carayon, 1998]. These considerations influenced our decisions regarding which aspects of the implementation process to evaluate, and also made clear that in HITEC, these processes occurred at multiple geographic and functional levels (from micro to macro). Ultimately, program domains at six levels (i.e., team, facility, DOC, state, facilitator, and intervention) were incorporated into the HITEC-2 intervention evaluation (characterized in Table II). The evaluation instrument developed by the HITEC-2 study team, entitled the Program Evaluation Rating Sheet (PERS), is described below. It is an experience-based empirical instrument that was constructed and validated based on less structured evaluation methods and observations from HITEC-1. The main departure of PERS from linear/sequential evaluation is a recognition that intervention content and process may occur simultaneously at different organizational levels.

TABLE II.

Program Evaluation Rating Sheet (PERS)

Site name:___
Intervention name:___
Start and end dates:___
Activity	Criteria	Rating Use binary rating (0 =did not occur,1 =occurred) to rate interventions designed with IDEAS. Activities not relevant to the team’s work should be rated N/A, and activities that have not yet taken place should be rated TBD.
I. Team-level factors
1. Met recommended initial recruitment goal for team membership	Six to eight members recruited for team (recommended by IDEAS tool)
2. Team members played instrumental role in gathering information and designing surveys/assessment tools	Team members distributed/collected “Team metric” surveys
3. Frequent team meetings held during period of active intervention development and implementation	KETmet 2×/month; DTmet1–4×/month
4. Meetings attended by minimum recommended number of team members	Three to four members present in >1/2 of meetings (recommended by IDEAS tool)
5. Meetings attended by union leadership	One union leader present in >1/2 of meetings
6. Completed IDEAS tool in recommended amount of time	For KET, <120 days total
7. Team invited involvement of experts and advisers	At invitation of team, internal or external experts contributed to process
8. Team members played instrumental role in communicating interventions to other COs in facility	At least two members have talked about intervention with other COs outside of team
9. Team conducted quantitative assessment of intervention effectiveness	Team metrics utilized (use of pre–post surveys)
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ____ Highest possible score = __ Percentage score = ___
II. Facility-level factors
10. Regular review meetings held with facility steering committee	FSC meets at least bimonthly
11. Supervisor assisted with scheduling meetings	Supervisor assists with scheduling on a meeting-by-meeting basis, based on CO schedules/availability
12. Attendance of warden or deputy warden at meetings (KET only)	Warden or deputy warden attended at least 30% of meetings
13. Attendance by bargaining unit representative (NP-4) at KET or FSC meetings	An NP-4 representative attended at least 30% of meetings
14. Approval of final intervention plan was received at FSC-level (DT only)	Endorsement by FSC
15. Sufficiency of time resources	Time resources provided were sufficient to carry out intervention designed by team
16. Outreach and dissemination throughout facility	Intervention information communicated via roll call announcements, flyer posting in facilities; HITEC 2 newsletter
17. Consistent participation and prompt replacement of key administrative/supervisory personnel during period of active intervention development and implementation	No longer than a1month gap in participation by key administrative/supervisory personnel (i.e., lieutenants, captains, deputy warden)
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ____ Highest possible score = __ Percentage score = __
III. DOC-level factors
18. Funding approved for intervention (if needed)	Funding approved for any DOC funding beyond facility budget
19. DOC approval of final intervention plan was received at SWSC level (if needed for intervention)	Endorsement by SWSC
20. Outreach and dissemination throughout DOC	Intervention information communicated via publication in DOC newsletter, website, flyer postings at all facilities
21. Commissioner or deputy commissioner intervened to resolve derailed intervention efforts (if needed)	Commissioner or deputy commissioner got involved, regardless of outcome
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __
IV. State-level factors
22. Financial support received at state level (if needed)	State budget did not adversely impact intervention funding
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __
V. Facilitator-level factors
23. Facilitator experience with IDEAS Tool implementation	Facilitator has implemented IDEAS Tool previously at least one time
24. Facilitator followed steps of the IDEAS tool	Program facilitator used IDEAS tool for intervention planning
25. Facilitator invited involvement of experts and advisers (if requested by team members)	At invitation of program facilitator, internal or external experts contributed to process
26. Principal Investigator intervened at facility level to resolve derailed intervention efforts (if needed)
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __
VI. Intervention-level factors
27. Assessment activities are designed to identify work and non-work risks to employee well-being	Team metrics: pre-intervention assessments (surveys, focus groups) included both work and non-work factors that impact health
28. Both organizational and individual behavior changes were considered during intervention development	Intervention planning addressed improvements to organizational policies/practices as well as employee health/safety behavior, in IDEAS Step 2
29. Work and non-work factors were both focus of the final intervention	Work and non-work factors addressed in final intervention per IDEAS Step 5 (as applicable to DT and KET)
30. Intervention included skill building in workforce	Yes or no
31. Participatory engagement in assessment of intervention/program	Twenty-five officers participated in Team Metrics (pre-post surveys) within facility
32. Participatory engagement of workforce in intervention/program	More than 20 workers (non-team members) participated in/attended intervention/program
33. Final intervention targeted change of DOC policy or infrastructure	Yes or no
34. Cost	Detailed (facility and programmatic) cost analysis of intervention was conducted
35. Sufficiency of financial resources	Financial resources provided for the intervention were sufficient
36. Intervention completed in specified time period	Consistency with start and start date
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __
VII. Implementation effectiveness
Adoption	Final intervention received approval for implementation in facility, at all necessary levels (FSC, SWSC)
Fidelity	Final intervention was implemented as intended by program developers, as specified by “Schedule of implementation activities” in IDEAS Step 6
Cost effectiveness	Final intervention met criteria for acceptable and realistic cost, as specified by “Resources/Cost” Key Performance Indicator in IDEAS Step 3
Penetration	Final intervention impacted people in the organization it was intended to benefit, as specified by “Scope” Key Performance Indicator in IDEAS Step 3
Sustainability	Final intervention was institutionalized within the facility or DOC through integration into policies and practices, ongoing and stable operations, or permanent funding or equipment
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __
VIII. Intervention effectiveness
Team metric indicator	Positive change on short survey (pre/post)
Study metric indicator	Favorable response on HITEC Core Survey item assessing program awareness or involvement
Health metric indicator	Positive change onT5 physical testing reassessment
Sum ratings within level, divide by the highest possible score, and give a percentage score. (Subtract NA and TBD ratings from highest possible score prior to calculating percentage score to the right.) →		Sum of ratings = ___ Highest possible score = __ Percentage score = __

CO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen event team; DT, design team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be Fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

Table II explains the criteria used to inform quantitative evaluation domains. Team-level time/effort commitments were assessed by the number of team members, frequency of meetings, and activities to advance team goals. Facility-level commitment and resources were assessed by FSC team composition, frequency of FSC meetings, FSC selection of an intervention for adoption, and other factors related to the facility environment. DOC-level commitments and resources were assessed by DOC senior leadership support and feedback for DT and KET proposed interventions. Support was quantified as approval of funding, broader publicizing of the interventions, and iterative resolution when intervention efforts were stymied. At the state level, we assessed whether funding was available for interventions and whether cumbersome procurement could be accelerated. At the facilitator level domain, we assessed the program facilitator’s experience with and use of the IDEAS planning tool in prior applications, responsiveness to team requests for expert assistance (e.g., nutritionist, ergonomist), and engagement of the principal investigator to resolve problems when intervention efforts were derailed. At the intervention level, we assessed participatory engagement resulting from use of the IDEAS tool [Robertson et al., 2013], intervention alignment with the TWH approach [CPH-NEW, 2015], as well as considerations related to financial resources and intervention timing.

The heuristic nature of PERS is recognizable. Evaluation items are quantitatively derived, but utility and content are also observed outcomes. These are reasons why PERS is an object of continuous development.

Implementation outcomes

Five implementation outcomes, adoption, fidelity, cost, penetration, and sustainability were assessed to evaluate if the implementation process resulted in intended outcomes [Chaudoir et al., 2013]. The evaluation metrics are derivative and not internally developed. We used Proctor et al. [2011] definitions. Adoption is the decision to use an intervention; fidelity is the extent to which the intervention was used as intended; cost-effectiveness gauges whether the financial resources needed for the intervention were acceptable and realistic; penetration is the extent to which the intervention was integrated within the setting; and sustainability reflects the extent of maintenance or institutionalization of the new intervention. The five implementation outcomes measures are delineated discretely in Table IX.

TABLE IX.

Ratings on PERS for Health Interventions at Sites A and B: Implementation Effectiveness

Implementation effectiveness	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
Adoption	0.5	1	NA	1	0	1	1	1
Fidelity	0.5	1	NA	1	NA	TBD	0	1
Cost effectiveness	0.5	1	NA	1	NA	TBD	1	1
Penetration	0.5	1	NA	TBD	NA	TBD	1	TBD
Sustainability	0.5	1	NA	TBD	NA	TBD	1	TBD
Sum	2.5	5	NA	3	0	1	4	3
Total possible	5	5	NA	3	1	1	5	3
% Of total possible	50	100		100	0	100	80	100

CO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

Program Evaluation Rating Sheet

The Program Evaluation Rating Sheet (PERS) (Table II) was developed to document the implementation of the KET and DT programs. It lists key activities within the implementation process as well as the five implementation outcomes. Using a binary rating system (0 =did not occur, 1 =occurred), program facilitators completed a PERS for each designed intervention to indicate whether the listed activities occurred or not. Activities not relevant to the team’s work were rated NA (not applicable), and activities that had not yet taken place were rated TBD (to be determined). Ratings were then summed within each level, divided by the highest possible score, and given a percentage score. NA and TBD ratings were subtracted from the highest possible score prior to calculating the percentage score.

The resort to a simple binary rating system, rather than a system of weights and quality factors, reflects deference to the current maturity of the evaluation process. We chose to not oversell specificity, in part to discourage fixation on detail that is not yet empirically supported. As a consequence, there is inconsistency between the detail of each particular measure and the simplicity of the scoring. By imposing limited variation on the scoring system, it was understood that in the analysis, summed scores likely would be less indicative than the presence or absence of key factors.

Measures

Four data collection methods were used to generate ratings for the PERS.

IDEAS worksheets

Completed during or directly after KET and DT meetings, the worksheets documented objective team events, including discussion topics and decisions made, providing key information about the planning process (i.e., number of steps completed, time to complete steps, upper level feedback received).

Meeting minutes

Recorded minutes taken at team (KET, DT) and committee (FWC, SWSC) meetings were used to assess administrative performance (e.g., number, frequency, and scheduling of meetings; team/committee size, composition, attrition/replacement; meeting attendance by members; level of support for final intervention plans, funding allocations, etc.).

Team metrics

The KETs and DT developed short pre- and post-intervention surveys (herein called “team metrics”) to assess intervention efficacy; they were customized for use within the facility to assess individual or organizational changes resulting from designed interventions. Respondents identified themselves using a personalized code only known to them, which was used to link surveys at the two time points. Team metrics were implicit in the HITEC-2 study design to segregate purely participatory derived outcomes from non-controvertible outcomes measures set by the study team [Cherniack et al 2015].

Study metrics

The HITEC-2 research team designed and administered a HITEC Core Survey and physical testing measures to evaluate health effects (herein called “study metrics”). They were conducted in 2013 (n =326 COs) and will be collected again in 2016. The HITEC Core Survey was a 30-min tool that gathered demographic information and assessed employee perceptions of health, related attitudes and behavior, working conditions, and socio-emotional factors associated with a healthy culture at work.

To further clarify, team metrics evolved from surveys and informal measures developed by the KET or DT and were idiosyncratic to the designed intervention and the site. These included pre- and post-intervention surveys on exercise and air quality, developed by the KETs and DT. Study metrics were identical for both sites and defined a priori by the study team. As an example, the Center for Epidemiologic Studies Depression [CES-D; Radloff, 1977] and Job Content Questionnaire [JCQ; Karasek et al., 1985] scales were components of the HITEC Core Survey, and served as study metrics along with physical measurements of blood pressure, body fat, and exercise tolerance.

RESULTS

Tables III–X present the PERS ratings regarding the implementation process and outcomes associated with interventions at Sites A and B. For ease of reading, we present one table for each of the PERS form’s eight sections (i.e., team level, facility level, DOC level, state level, facilitator level, intervention level, implementation effectiveness, intervention effectiveness). The tables include a critical subjective differentiation: the facilitator’s assessment of the intervention effort as successful (S), unsuccessful (U), or in-process/unresolved (I). Figure 2 presents the implementation timeline.

TABLE III.

Ratings on PERS for Health Interventions at Sites A and B: Team level

Team-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall Intervention Assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
1. Met recommended initial recruitment goal for team membership	1	1	1	1	1	1	1	1
2. Team members played instrumental role in gathering information and designing surveys/assessment tools	1	1	NA	1	NA	TBD	1	1
3. Frequent team meetings held during period of active intervention development and implementation	1	1	1	0	1	NA	1	1
4. Meetings attended by minimum recommended number of team members	1	1	1	1	0	1	1	0
5. Meetings attended by union leadership	1	0	0	0	1	1	1	1
6. Completed IDEAS tool in recommended amount of time	0	1	NA	1	NA	NA	NA	NA
7. Team invited involvement of experts and advisers	1	1	1	NA	NA	NA	1	1
8. Team members played instrumental role in communicating interventions to other COs in facility	1	0	NA	1	1	1	1	1
9. Team conducted quantitative assessment of intervention effectiveness	1	1	NA	1	NA	TBD	1	1
Sum	8	7	4	6	4	4	8	7
Total possible	9	9	5	8	5	4	8	8
% Of total possible	89	78	80	75	80	100	100	88

NCO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

TABLE X.

Ratings on PERS for Health Interventions at Sites A and B: Intervention Effectiveness

Intervention effectiveness	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
Team metric indicator	0	1	NA	TBD	NA	TBD	1	TBD
Study metric indicator	TBD	TBD	TBD	TBD	TBD	TBD	TBD	TBD
Health metric indicator	TBD	TBD	TBD	TBD	TBD	TBD	TBD	TBD
Sum	0	1	TBD	TBD	TBD	TBD	1	TBD
Total possible	1	1	TBD	TBD	TBD	TBD	1	TBD
% Of total possible	0	100					100

CO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available, PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

FIGURE 2.

Timeline of HITEC-2 health interventions (2012-present). HITEC, health improvement through employee control; KET, Kaizen event team; DT, design team; BILD, building improvement linked to design; W-2 BFIT, work to be fit; BFED, better food through education and design; SWIPE, structured work-related injury prevention through ergonomics.

It should be noted that not all four intervention topics (BILD, W-2 BFIT, BFED, SWIPE) are presented. The SWIPE intervention has yet to occur at Site A; the BFED intervention at Site B was pre-packaged and adopted early without the IDEAS tool. To qualify for inclusion in this study, an intervention was required to have completed Step 2 of IDEAS. Two interventions, the KET’s W-2 BFIT and DT’s SWIPE intervention, are presented twice. The first attempts were unsuccessful, but a second iteration of each (still in-progress), appears more successful.

Site A: KET Interventions

At Site A, KETs have thus far worked on three health priority areas in the following order: BILD, BFED, and W-2 BFIT. Program facilitators characterized BILD as unsuccessful in addressing two health subtopics: indoor air quality (IAQ) and noise. The IAQ intervention was the cleaning and refurbishment of the heating, ventilation and air conditioning (HVAC) system, and the noise interventions included soundproofing and the provision of new headsets/earbuds for COs in place of shoulder-mounted handsets. BILD met 89% of team-level activities (Table III), 100% of facility-level activities (Table IV), 75% of DOC-level activities (Table V), 0% of state-level activities (Table VI), 100% of facilitator-level activities (Table VII), and 75% of intervention-level activities (Table VIII). This KET did not complete the IDEAS design process in the recommended amount of time (120 days), did not communicate its interventions DOC-wide, and did not receive expected state-level financial support for soundproofing, despite iterated and detailed installation plans. Although this intervention was categorized as unsuccessful because identified outcomes were unrealized, it is more accurately classified as partially successful. There was review and pilot testing of sound-insulating materials by the KET. Earpiece piloting was repeatedly delayed by the diffidence of the manufacturers in supplying in-the-ear hardware and suitable electronics for testing. As a result, the KET exceeded the 120 day limit. The KET-developed interventions for IAQ (HVAC system) were successful in terms of adoption, fidelity, cost-effectiveness, penetration, and sustainability (Table IX). BILD was the first HITEC-2 intervention project. Both the study team and the KET concluded that the complexity of the projects exceeded the capacities of the KET, despite there being more time and resources invested than in any other intervention.

TABLE IV.

Ratings on PERS for Health Interventions at Sites A and B: Facility level

Facility-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall Intervention Assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
10. Regular review meetings held with Facility Steering Committee	NA	NA	NA	NA	1	NA	0	1
11. Supervisor assisted with scheduling meetings	1	1	1	1	0	1	1	1
12. Attendance of warden or deputy warden at meetings (KET only)	1	1	1	1	NA	1	NA	NA
13. Attendance by bargaining unit representative (NP-4) at KET or FSC meetings	1	0	0	0	1	1	1	1
14. Approval of final intervention plan was received at FSC-level (DT only)	NA	NA	NA	NA	0	TBD	1	1
15. Sufficiency of time resources	1	1	1	1	1	TBD	1	1
16. Outreach and dissemination throughout facility	1	1	NA	1	NA	1	1	1
17. Consistent participation and prompt replacement of key administrative/supervisory personnel during period of active intervention development and implementation	1	1	1	1	1	1	1	1
Sum	6	5	4	5	4	5	6	7
Total possible	6	6	5	6	6	5	7	7
% Of total possible	100	83	80	83	67	100	86	100

NCO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be Fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics, FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

TABLE V.

Ratings on PERS for Health Interventions at Sites A and B: DOC Level

DOC-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
18. Funding approved for intervention (if needed)	1	NA	NA	NA	0	TBD	1	1
19. DOC approval of final intervention plan was received at SWSC level (if needed for intervention)	1	1	NA	NA	0	TBD	1	1
20. Outreach and dissemination throughout DOC	0	1	NA	1	NA	TBD	1	TBD
21. Commissioner or deputy derailed intervention efforts (if needed)	1	NA	NA	NA	NA	NA	NA	1
commissioner intervened to resolve
Sum	3	2		1	0		3	3
Total possible	4	2		1	2		3	3
% Of total possible	75	100		100	0		100	100

NCO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined, NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

TABLE VI.

Ratings on PERS for Health Interventions at Sites A and B: State level

State-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
22. Financial support received at state level (if needed)	0	0	NA	NA	NA	NA	NA	NA
Sum	0	0
Total possible	1	1
% Of total possible	0	0

CO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be Fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS, Intervention Design and Analysis Scorecard.

TABLE VII.

Ratings on PERS for Health Interventions at Sites A and B: Facilitator level

Facilitator-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
23. Facilitator experience with IDEAS Tool implementation	1	1	1	1	0	1	0	1
24. Facilitator followed steps of the IDEAS tool	1	1	1	1	1	1	1	1
25. Facilitator invited involvement of experts and advisers (if requested by team members)	1	1	1	1	1	TBD	1	1
26. Principal Investigator intervened at facility level to resolve derailed intervention efforts (if needed)	1	NA	1	NA	1	NA	1	NA
Sum	4	3	4	3	3	2	3	3
Total possible	4	3	4	3	4	2	4	3
% Of total possible	100	100	100	100	75	100	75	100

NCO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS =Intervention Design and Analysis Scorecard.

TABLE VIII.

Ratings on PERS for Health Interventions at Sites A and B: Intervention level

Intervention-level factors	SITE A: Kaizen event teams				SITE B: design team
	BILD	BFED	W-2 BFIT-1	W-2 BFIT-2	SWIPE-1	SWIPE-2	BILD	W-2 BFIT/BILD
Overall intervention assessment	Unsuccessful	Successful	Unsuccessful	In-progress	Unsuccessful	In-progress	Successful	In-progress
27. Assessment activities are designed to identify work and non-work risks to employee well-being	1	1	NA	1	NA	TBD	1	1
28. Both organizational and individual behavior changes were considered during intervention development	1	1	NA	1	0	TBD	1	1
29. Work and non-work factors were both focus of the final intervention	NA	1	NA	1	0	TBD	NA	1
30. Intervention included skill building in workforce	1	1	NA	TBD	0	TBD	1	1
31. Participatory engagement in assessment of intervention/program	0	1	NA	1	NA	TBD	1	TBD
32. Participatory engagement of workforce in intervention/program	1	1	NA	1	NA	TBD	1	1
33. Final intervention targeted change of DOC policy or infrastructure	1	1	NA	TBD	1	TBD	1	1
34. Cost	1	1	NA	1	0	TBD	1	TBD
35. Sufficiency of financial resources	0	0	NA	0	NA	TBD	1	1
36. Intervention completed in specified time period	0	1	NA	1	NA	NA	NA	NA
Sum	6	9		7	1		8	7
Total possible	9	10		8	5		8	7
% Of total possible	67	90		87	20		100	100

CO, Correctional Officer; DOC, Connecticut Department of Correction; KET, Kaizen Event Team; DT, Design Team; BILD, Building Improvement Linked to Design; W-2BFIT, Work to be fit; BFED, Better Food through Education and Design; SWIPE, Structured Work-related Injury Prevention through Ergonomics; FSC, Facility Steering Committee; SWSC, Study-wide Steering Committee; TBD, To be determined; NA, Not available; PERS, Program Evaluation Rating Sheet; IDEAS =Intervention Design and Analysis Scorecard.

BFED was characterized as successful. Outputs included the creation and posting of educational posters on nutrition, the addition of healthier vending machine options, tailored healthy choices menus from local take-out restaurants, and reduced pricing of bottled water. BFED met 78% of team-level activities, 83% of facility-level activities, 100% of DOC-level activities, 0% of state-level activities, 100% of facilitator-level activities, and 89% of intervention-level activities. The lack of state-level activity refers to the fact that financial support was not provided for one intervention alternative proposed by the KET: a drinking fountain filtration system that would allow COs to refill personal water bottles as an alternative to soda. Unlike the BILD intervention, the state-level reticence was not fatal to program execution. Deficiencies included the failure of KET members to communicate about interventions to other COs, the failure of a union representative to attend KET meetings, and the unavailability of non-budgeted state funding. Nonetheless, the BFED interventions achieved 100% of the five implementation outcomes, and the posters and educational materials were adopted by DOC for all of its facilities.

Tables III–X list W-2 BFIT twice. The original KET was unsuccessful (W-2 BFIT-1) due to the lack of facility management support. A new KET was revived several months later (W-2 BFIT-2), promoted by active supervisor support. W-2 BFIT-1 proposed use of the facility’s courtyard by COs for a formalized “exercise break” with the installation of stationary exercise equipment. Facility leaders demurred, citing CO absence during codes. KET members abandoned the idea, concerned that managerial objection might expand to other breaks, such as smoking breaks. Fewer than 10% of COs smoke, but the withdrawal of an established “right” is a fundamental provocation for COs. W-2 BFIT-1 met 80% of team-level activities, 80% of facility level activities, and 100% of facilitator-level activities. There was no line officer union representation at KET meetings.

The second attempt, W-2 BFIT-2, is “in-progress” and shows early promise. W-2 BFIT-2 was revived through a new idea proposed by the same supervisor who had raised the earlier concerns over courtyard exercise. A hallmark of program has been the “Pedometer Challenge,” which involves teams of COs wearing pedometers to increase physical activity through tracking steps walked per day, in combination with complementary fitness- and health-related educational materials. The Pedometer Challenge is an elaboration on a walking program, a type of health promotion activity that had been previously unsuccessful. Its effectiveness is a function of local variation and innovation that combines the features of team competition and translation of total distances walked into a graphical depiction of walk to a desirable location (e.g., a map plotting a walked route to New Orleans). The KET became engaged in peer-level interventions with a sub-group of COs who also work as personal trainers or are competitive athletes. W-2 BFIT-2 has already met 75% of team-level activities, 83% of facility-level activities, 100% of DOC-level activities, 100% of facilitator-level activities, and 86% of intervention-level activities. The evaluation showed that KET meetings were held less frequently than advised and that no union representative was present. W-2 BFIT-2 achieved the implementation outcomes of adoption, fidelity, and cost-effectiveness; penetration and sustainability are yet to be determined. The apparent successes reflect the emergence of an underlying health and fitness culture that is ordinarily suppressed.

Site B: DT Interventions

At Site B, the DT thus far has worked in three health priority areas: SWIPE, BILD, and W-2 BFIT. The intervention proposed in SWIPE-1 was a “Stress Lounge,” a room in the high-security area where COs could recover from the stress of responding to emergency incidents, and regain composure for more routine work challenges. The FSC rejected the stress lounge idea, citing space limitations. However, as with the proposed W-2 BFIT-1 courtyard exercise breaks, the main reason for the stress lounge demurral was management concern that COs would abuse their break privileges. SWIPE-1 met 80% of team-level activities, 67% of facility-level activities, 0% of DOC-level activities, 75% of facilitator-level activities, and 20% of intervention-level activities (because only IDEAS Step 1 was used). DT meeting attendance was low, supervisors did not assist with scheduling, the proposed intervention was not approved by FSC or by DOC administration, funding was not approved, and the program facilitator was inexperienced with using the IDEAS tool. SWIPE-1 was not adopted, so the remaining implementation outcomes could not be assessed. SWIPE has recently been revived as SWIPE-2. SWIPE-2, currently in-progress, revives the stress lounge intervention more than a year after the proposal was abandoned. New facility leaders reviewed the DT’s earlier proposed intervention and decided to create the stress lounge. Thus far, SWIPE-2 has met 100% of team-level activities, 100% of facility-level activities, and 100% of facilitator-level activities. These ground-up measures appear to capture the critically necessary components of functional and nonfunctional interventions (see Tables III and IV). It is still too early in the process to assess DOC- and intervention-level activities, and implementation outcomes (apart from adoption). The effectiveness of SWIPE-2 has been contingent on the DT approach where program recommendations are iterative and there is no formal termination date. Both the SWSC and the DOC Commissioner had advised against abandoning the concept. It should be noted that specific time window compliance was a stipulation of the HITEC-2 study design and not an essential characteristic of scorecard evaluation.

The BILD interventions targeting the improvement of IAQ and were highly successful. A particularly interesting outcome was the training of DT members on indoor environmental quality issues by the Connecticut Department of Public Health using their Tools for Office Buildings (TfOB) training program. The DT members used the skills they obtained through TfOB to develop IAQ related interventions, including the training of inmates on systematically cleaning air vents/ducts, and the development of a new standardized cleaning protocol for the facility. BILD met 100% of team-level activities, 86% of facility-level activities, 100% of DOC-level activities, 75% of facilitator-level activities, and 100% of intervention-level activities. The DT did not have regular FSC meetings and did not have a facilitator with previous IDEAS tool experience. Nonetheless, BILD achieved 80% of the five implementation outcomes (adoption, cost-effectiveness, penetration, sustainability). BILD also had a demonstrable and dramatic downstream effect that exceeded DT recommendations. The DT worked actively with facility managers as temporary team members to develop a multi-level plan that ranged from preventing vent obstruction to new zoned ventilation. Pre- and post-team metric surveys were used to assess effectiveness. The effort was sophisticated and targeted, and the perceived improvement in IAQ was dramatic. The DOC responded by offering what was deemed impossible, complete replacement of the antiquated ventilation system. This may be the greatest accomplishment of the BILD intervention, although the outcome exceeded the proposed interventions.

The W-2 BFIT intervention, characterized as in-progress by the program facilitator, targets the improvement of foot, leg, and back health. Physical pain in the back and lower extremities was seen as a root cause of CO lack of exercise for physical fitness. Recognizing that foot, leg and back issues result from the demands of the job as well as individual habits, the DT also focused on the work environment, incorporating some elements of the BILD approach. The resulting interventions were (i) a health fair that targeted the improvement of foot health and physical fitness; (ii) the permanent installation of ergonomic mats in the workplace for use by COs who stand for long periods on hard surfaces; and (iii) a structured intervention around footwear selection and morbidity assessed by screening and education from a podiatrist and physical therapist. W-2 BFIT/BILD met 88% of team-level activities, 100% of facility-level activities, 100% of DOC-level activities, and 100% of facilitator-level activities, and 100% of intervention-level activities. DT meeting attendance was low, but W-2 BFIT/BILD achieved the implementation outcomes of adoption, fidelity and cost-effectiveness; the outcomes of penetration and sustainability are yet to be determined.

DISCUSSION

This process evaluation was conducted to introduce the PERS methodology. PERS was internally developed to follow the Health Workplace Participatory Program and the steps of the IDEAS tool, and to evaluate the participatory process for intervention studies in corrections. PERS is not a final word in PAR evaluation, in general. At its current level of generality, a contrarian observation might dismiss its superiority over qualitative observation, given the time commitments. Quantitative evaluation has proved to be particularly useful in a labor-management combined activity, where some level of tension and interplay is unavoidable and necessary. PERS-based information has enabled dialogue by identifying critical processes and linking them to outcomes, being an alternative to postulation.

There are several features of the PERS that distinguish it from other assessment tools used to evaluate TWH-type programs, such as the approach from the American College of Occupational and Environmental Medicine [Loeppke et al., 2015]. First, because workforce and management participation is an a priori exigency, participation is not an isolated category but is imbedded into multiple domain at the team, facility, DOC, state, facilitator, and intervention levels. Second, the operational level approach is both iterative and bottom-up, and does not presume managerial directives or predominant managerial responsibility. Third, management “buy-in” or commitment has contributory not formative status. All of this assumes a managerial flexibility toward an approach that requires shared responsibilities and bounded structures, although also within fiscal identified organizational constraints.

HITEC-2’s two participatory programs, based on the KETs and DT, were compared in order to illustrate use of the PERS instrument and to assess its utility by amending raw scores with study team observations on process. The PERS was developed within the multiple sites of the HITEC-2 study and it, therefore, is not site specific, nor is it fully generic.

PERS and Organizational Hierarchy

Carrying out PAR research means operating within the parameters of an organization and its systems. Workplaces are at best an indirect polity, and PAR is intrinsically foreign to hierarchical organizations, such as DOC, because its core principles of equality, power-sharing, and inclusiveness conflict with a military-type structure. Although the experience with DOC limits generalization, even with these evident constraints, the PERS-based evaluation improved participant understanding of “fit” by sub-dividing and identifying the particular items affecting performance. In DOC, remediation occurs quickly and effectively when mandated. For example, anti-gang security measures can be immediate when transmitted as orders or directives. However, change occurs without participation or engagement, and the corrections culture is unmuted. DOC’s hierarchical organizational structure is characteristic of law enforcement and the military. Although there is strong mutual appreciation of common risks and stresses, there are limitations to participatory interventions if security or chain of command is compromised. There are inherent tensions between line officers, supervisors, and facility leadership (deputy wardens and wardens), and the process of subordination and promotion to higher ranks encourages builtin defaults to exclusionary seniority or a suspicion of supervisory use of information for reward or punishment.

For example, the concept of “job stress” in corrections may seem sufficiently obvious and ubiquitous that further characterization is unnecessary. The locked environment and dorm units house up to 130 inmates in partially divided quarters, monitored by a single “hypervigilant” CO. Security-driven reporting policies mandate that a CO demonstrating mental health symptoms must be reported to a supervisor. Hence, the barriers to use of the employee assistance program or familiarity with a Human Resources (HR) representative are high.

The PERS captures the step-by-step, facility-to-team intercourse, that is a foundation for participatory programs, and which avoids challenge to unbending organizational structures. By weighting the process over the outcome decision, the participatory team can probe and negotiate with arbiters of formal structure. In Tables IV (facility-level factors) and VIII (intervention-level factors) this process of working within a hierarchical work culture is demonstrated in the experience with the “stress lounge.” The DT was innovative in its program development, even soliciting recommendations from an institutional building architect. The facilitator (see Table VII) performed at a relatively high level despite limited experience with the newly implemented IDEAS tool. The issues had been taken to the SWSC, but barriers at the facility level and from DOC administrators stymied the DT’s project for professed policy and security reasons. Over time, facility leadership changed, including the promotion of a key supervisor familiar with HITEC-2 and an enthusiast for participatory intervention. The iterative process of including supervisor attendance with DT meetings has resulted in the more promising work toward a “stress lounge” in SWIPE-2. Identification of the critical stage and establishing a process for continuous engagement, softened a “hard” administrative refusal. In a hierarchical and highly structured organization like corrections, iterative processes, occurring within an iterative time span, have proved to be resilient and are abetted by an evaluation process that assesses multiple items at several organizational levels.

An unanswered question in the use of PERS is the identity of the evaluator. Normally, this is an HR or management function. In HITEC-2, the role initially fell on the study team, and is in the process of transfer to an internal training group. Neither HR nor a dedicated study team are long-term solutions and, therefore, the seemingly simple issue of the identity of the evaluator is not yet resolved.

PAR and Supervisory Support

The dilemma for PAR in corrections, and by extension to other workplace sectors, resides in the tension between line worker autonomy, which is essential for innovation and persistence, and supervisor direction, which is essential for administratively generated cohesion. In this context, the concept of “supervisor support” becomes overly simplistic. No supervisors explicitly opposed program efforts, but many offered nuanced support, sometimes based on reluctance toward CO autonomy. In some instances, supervisors were supportive of programs to improve CO health but demurred on the participatory nature of the program. For example, the DT program’s BILD (IAQ) intervention was resisted by the maintenance department when COs first proposed the idea, because COs were perceived as peddling institutional myths and operating above their expertise. The facilitator introduced maintenance supervisors to the detailed process used in the IDEAS tool by the DT. The example distinguishes a technical management group from operational supervisors; the idea of “management support” is broken down into practical constituent elements. The PERS formalized this element of supervisory facilitation as facility-level factors, as distinguished from DOC-level factors, involving administrative supervision.

The PERS definition of facility-level factors was also useful in delineating circumstances where CO participation was supported, but there was resistance to specific interventions. While there was broad support for workforce engagement with fitness and stress relief, the KET’s W-2 BFIT intervention of a courtyard exercise break was rejected for being outside of the walls. A geographically confined pedometer program was acceptable, although it involved amending prohibitions to worn ancillary devices. The resurrection of a fitness program that had sufficient alternative plans underscores the importance of persisting within an organization that is typified by transfer, retirements, and contract conflicts, and periodic changes in workplace climate. The PERS also highlights the importance of frontline COs within intervention programs, but also demarcating the limits in a highly structured administrative environment with a priority on safety [Borda, 2001; Cherniack et al., 2015].

Although the resistances and endorsements of supervisors were recorded, the PERS domains were not sufficiently detailed to capture individual quality of thought and action. This type of assessment requires subjective evaluation by a third party or enhanced, and possibly non-validated, weighting of evaluation variables. In DOC, the supervisors’ bargaining unit has been an elemental ally of the HITEC-2 effort, even requesting and receiving TWH-type services for their own members outside the main study. Leadership/supervisory quality may not be best realized by conventional evaluation methods or the PERS.

Another factor that seemed to facilitate CO participation was the presence of a union representative. This is not equivalent to supervisor support, but in a strong union environment, it appears to be an important component of facility-level factors. The DT indicated that CO participation in the presence of supervisors was greatly enhanced when a union representative attended the meeting. Union leadership varies, is subject to periodic change, and is weakened under current labor market conditions. A vigorous labor-management cooperative process may require some level of labor independence and protection. For example, attendance by a bargaining unit representative may well prove critical at times of labor-management contention, but in HITEC-2 several interventions proceeded without this important inclusion. The absence of an independent labor presence, which is the norm in American workplaces, poses a different set of challenges to workforce participation.

Site-Specific Observations

Although this manuscript is primarily an introduction of the PERS, and is somewhat site agnostic, because there were only two study sites, some level of site-level comparison seems useful. The KET was more effective for well-defined, results-directed interventions that benefitted from the active participation of supervisors. The DT was better suited for longer term, more resilient proposals. The determinants of success appeared to be more reliant on the presence or absence of key factors that were influential in both approaches. Accordingly, the most effective approach may include elements of both the DT and KET, or favor one over the other (e.g., direct versus indirect supervisor involvement, short versus long intervention interval), depending on the nature of the intervention [Cherniack et al., 2015]. In future efforts, where research-based comparisons are unnecessary, workable structures likely will incorporate elements from both approaches.

The determination that the BILD intervention was unsuccessful at the KET site, due to incompletion within an assigned interval, was a study site specific stipulation. From the perspective of HITEC-2, the project was unsuccessful. However, the team-level score was high (89%), with the single negative being the lack of completion in the allotted time interval. Again, the time limit was an artifact of Kaizen implementation.

In addition to participation, continuity is a key element of implementing continuous improvement participatory programs [Haims and Carayon, 1998]. The two HITEC-2 programs varied in their degree of continuity: by design, the KETs were episodic and the DT continuous. However, both programs had alternating cycles of intense and innovative activity and dormancy [Borda, 2001; Cherniack et al., 2015]. One key to program continuity was the overall administrative structure (i.e., KETs/DT, FSC, SWSC, study team). For instance, the study team was instrumental in maintaining stability through cycles of ebbing and flowing engagement, and the DOC leaders facilitated process by troubleshooting and intervening when program efforts were stalled. This is illustrated by the DT’s SWIPE (IAQ) intervention. There was difficulty maintaining team consistency and morale in early stages, but there was ultimately success due to essential re-engagement efforts on the part of wardens and labor representatives.

Timing was important to continuity during the implementation process. The timeframes were originally intended to be different for the KETs and DT, but in practice there was convergence. The DT had no required timeframe for carrying out interventions, and the KETs had a specified 120 day timeline. However, the successful interventions for both the KET and DT programs were implemented over a more extended time period; the KET’s fitness program had two iterations, and the DT’s BILD lasted 18 months. This suggests that a timeframe longer than 120 days may be required for any participatory approach at DOC to be successful. This observation runs counter to the management literature that associates innovation with speed [Lawton, 1993; Banu Goktan and Miles, 2011]. The reasons for longer timeframes are various, but despite efforts to facilitate upper-level management support for interventions, the SWSC met either monthly or bi-monthly, and the scheduling of FSC review due to work demands often took several weeks. The long duration of implementation worked to the disadvantage of KETs; the DT was hampered by the absence of supervisors to assist with scheduling, communications, and back-up. The KETs had an advantage: the involvement of supervisors on KETs facilitated CO attendance because (i) supervisors were able to assist in scheduling subsequent meetings at times when the greatest number of team members was available to attend; and (ii) supervisors’ regular presence at meetings seemed to sustain the regularity of CO attendance.

Line-officer and supervisor tensions are characterized differently in the KET and DT environments, a situation not specific to the PERS, but reflecting study design. COs on the KETs reported that their participation during team meetings was constrained due to the presence of supervisors, which was an important team-level factor. The issue was minimally relevant to the CO populated DT. However, increased candor without supervisors was compromised by the vulnerability of intervention proposals to dismissal by the FSC and to diffidence from supervisors. The DT seemed to experience more resistance due to operating outside of their usual CO role. These concerns may have been especially relevant because all DT interventions targeted organizational change.

The PERS and Program Resilience

A key determinant of implementation success was program resilience. Program resilience seemed especially important for accommodating a democratic/flexible process and for remounting interventions that had been blocked. This ability to adapt allowed the programs to maintain functional continuity even when teams disbanded and re-assembled with new members, when staff changes took place due to job reassignments and retirement, when union involvement fluctuated with elections and contract negotiations, and when interventions were stalled or unsuccessful. Adaptation required the long view of accepting obstacles as temporary challenges that were part of the PAR process. In some circumstances, delay worked to the program’s advantage. HITEC-1 and -2 investigators directed earliest efforts to the organizational climate: constructing the SWSC, engaging separately and then together labor and management representatives, and building the programmatic layers, before any interventions were attempted [Cherniack et al., 2015].

The installation of a redundant structure was not incidental, but came out of many years of study team experience with joint labor-management health and safety committees. The program facilitator and study team were particularly instrumental in ensuring program resilience. As the DOC moves in the direction of fully participatory programs characterized by internal regulation [Haims and Carayon, 1998], identifying an internal alternative to the study team will be a challenge, but is central to HITEC research as it moves into its final phase in preparation for broad dissemination. The PERS does not provide a domain-based measure of program resiliency. Team-level factors include adherence to the IDEAS tool, which requires feedback, iteration, and alternate plans. The concept is brushed upon in Implementation Effectiveness (Table VII), but a seamlessly effective intervention does not implicitly require resiliency. There is no immediate solution in this version of the PERS because a qualitative tool would dilute objectivity and therefore, reproduction and a quantitative tool, unsupported by observation, would serve little benefit.

Financial Considerations

Our evaluation indicated the importance of establishing financial resources for the KET and DT programs. DOC and the State of Connecticut are functioning in a world of yearly budget rescission. Financial commitments which appeared viable, such as the KETs’ BFED (water filling stations) and BILD (soundproofing) interventions were withdrawn due to ongoing budget cuts. A detailed cost analysis to determine whether interventions are possible with funding available is one of four key performance indicators considered in the IDEAS design process. The metric for Intervention Effectiveness includes realistic cost effectiveness, and the importance of budgeting is a key DOC-level factor. Just as expected costed interventions went unfunded, a universal presumption of absent funding would have prevented the unexpected purchase of the new HVAC system at Site B. Evidence for effectiveness in health interventions may enhance investment in worker health at a time of rampant health and workers’ compensation costs.

Limitations of PERS

The PERS evaluation method did have limitations. First, not all criteria listed were achievable by all intervention efforts (some were not applicable or incomplete), preventing a point-by-point comparison of intervention implementation. Second, our binary rating system is too crude and did not capture variability. It needs further refinement and validation. Our percentage ratings allowed for an overall assessment of project success, but future research might introduce more timely, participatory and refined methods of measuring successful process. Refinement might include weighting or tighter definition of necessary steps. At this time, important contextual factors are not readily reduced to a relative score.

Our process evaluation excluded less tangible measures. The quality of supervisors, who championed PAR and TWH principles and maintained profound workforce trust, was important, but we did not attempt its capture in PERS. This was also true for capturing transformational leadership and group resilience.

Conclusions and Suggestions for Future Research

The participatory intervention programs conducted at DOC led to creation of a program evaluation instrument (the PERS) to assist in identification of factors associated with intervention success within the correctional workplace, and may also be useful in other sectors. Enabling factors included consistency of meetings and individual and extended time frames for interventions. Selecting a relatively simple and accessible intervention at onset seems to enhance successful team efforts. At DOC and other organizations with frequent turnover of key staff and leadership, there is utility in building program resilience through multi-level activity and governance.