Table 2.
Summary of the proposed multi-level, mixed methodology for the initial implementation evaluation of WorkChat.
| Process Evaluation Domain | Type of method, source of data, examples |
|
|---|---|---|
| Quantitative Data | Qualitative Data | |
| Acceptability (career readiness counselors [CRCs]) | Training Evaluation Questionnaire; Intervention Acceptability Measure (sample questions: satisfied with the training you received to teach WorkChat?; Overall, how satisfied are you with WorkChat as a service for your students?) | CRC Interviews (example questions: how we can improve the CRC training experience? What did you like the most/least about WorkChat? Can you share your thoughts on what was appealing about teaching students how to use WorkChat?) |
| Acceptability (students) | Treatment Acceptability Rating Form (sample question: WorkChat was enjoyable) | Semi-structured interview for students (sample question: What did you like the most about WorkChat?) |
| Usability (students) | System Usability Scale (sample question: I think I'm good at using the WorkChat program) | Semi-structured interview for students (sample question: What were the technical challenges you experienced when using the WorkChat software?) |
| Appropriateness (CRCs) | Intervention Appropriateness Measure (sample question: How well do you think WorkChat fits with students' goals for job training?) | Semi-structured interviews with CRCs (sample questions: How is WorkChat an appropriate fit for employment services?) |
| Feasibility (CRCs) | Intervention Expected Feasibility Survey (sample question: How well do you think WorkChat fits with students' goals for job training?) | Semi-structured interviews with CRCs (sample questions: What will help you the most when teaching WorkChat to students? |
| Feasibility (CRCs) | We will administratively monitor the time required to train CRCs to deliver WorkChat; fidelity of WorkChat delivery; fidelity checklist completion rates; CRC time allocated to WorkChat implementation | Semi-structured interviews with CRCs (sample questions: What was your experience completing the fidelity checks? What are barriers to WorkChat implementation? What are factors that help to successfully facilitate WorkChat implementation?) |
| Feasibility (student) | We will administratively monitor: adherence to training visits; reasons for missed visits; reasons for skipping different training components | Semi-structured interview for students (sample question: How can we make WorkChat easier to use for students in pre-employment transition services?) |
| Cost (MCTI administrators, University Implementation Team) | We will monitor the time CRCs, MCTI administrators, and University Team members spend on preparing the site to implement WorkChat. | Semi-structured interviews with MCTI administrators (sample question: How will cost be a major factor in continuing to use WorkChat at MCTI?) |