Table II.
Program Evaluation Plan
| Evaluation Measures | Evaluation Questions | Data Sources | Tools / Procedures | Data Analysis or Synthesis | Use of the Findings | |
|---|---|---|---|---|---|---|
| Program Implementation - Process Evaluation | ||||||
| Recruitment | ||||||
| Recruit Work Organizations | Fidelity – the extent to which researchers followed the “wish list” for recruitment, contractors’ adherence to the partnership agreement, and the surrounding context. | 1.To what extent were contractors recruited within the “wish list”? 2.Did contractors follow the partnership agreement? |
Researchers | 1.Items from the “wish list.” 2. Items from the partnership agreement & meeting notes. |
Content analysis of the initial partnership and post-intervention meeting notes. | Inform the criteria for recruitment and feasibility of items on the partnership agreement. |
| Designed Activities | ||||||
| Participatory Ergonomics (PE) Training Intervention | Fidelity – the extent to which the program was implemented as planned and the context in which it occurred. | To what extent were the training intervention and training content implemented as researchers planned? | Researchers | Descriptions of original training, training logs, and delivered training presentations. | Comparison of intended training program delivery to the actual training program delivery. | Determine what was feasible to provide in training and modifications for the training content. |
| Activity Outputs | ||||||
| Delivered & Received Program | Reach – proportion of the intended audience that participates in the training (attendance). | Was the training program delivered to at least 80% of the workers? | Researchers | Training logs | Calculate the proportion of workers who participated in each training component. | If attendance is low, consider adjusting delivery of training to improve reach. |
| Frequency – the amount of intended units of each delivered training component. | To what extent was the amount or number of intended units of training delivered? | Researchers | Training logs | Calculation of frequency of attendance for each training component. | Explore the context to delivering the training to make adjustments. | |
| Duration of workers in training. | How long did the workers receive the training? | Workers | Training logs | Sum the duration of training. | Explore the context surrounding duration to make adjustments. | |
| Engagement of workers with the intervention. | 1. To what extent did workers actively engage in the training? 2. Were workers receptive to the program? |
Workers | 1. Field notes. 2. Self-report surveys & focus groups. |
Content analysis | Determine ways to improve active engagement & receptiveness. | |
| Context – Aspects of the larger social & political environment that may influence the program. | Were there aspects of the larger environment that influenced implementation? | Contractors, union reps, local business reports. | Field notes, worker job logs, contractor & union interviews, and local business journal reports. | Review of the local construction economy from 2007 through 2011. | Modify the program to work within the limitations of the context. | |
| Program Efficacy – Summative Evaluation | ||||||
| Short Term Impacts | ||||||
| Learning | Skills, awareness, knowledge, attitudes | To what extent did workers identify problems (e.g. risk factors) and solutions (e.g. work methods & tools or equipment)? | Workers | 1. Self-report surveys. 2. Observation field notes. 3. Interview field notes. |
1.Quantitative analysis. 2. Content analysis. 3. Content analysis. |
Consider modifications to target increased learning. |
| Intermediate Impacts | ||||||
| Actions | Behavior, practice, decision-making | To what extent did workers perform solutions (e.g. work methods & tools or equipment) to make the job easier? | Workers & Researchers | 1.Self-report surveys. 2. Observation field notes. 3. Video analysis. 4. Interview notes. |
1.Quantitative analysis. 2. Content analysis. 3. Video analysis. 4. Content analysis. |
Modify training to target increased learning and actions. |
| Long Term Outcomes | ||||||
| Health & Injury Risk | Symptoms, risk factors, MSD | To what extent did workers’ reports of discomforts, exposures to risk factors, & MSD change? | Workers & Contractors | 1.Self-report surveys administered at baseline, 3 and 12 months. 2. OSHA 300 Logs. |
Quantitative analysis | Interpret these results based on findings from the process & impact evaluations. |