Table 2.

Evaluation framework methodology based on the RE-AIM and Kirkpatrick models—measures, data collection methods, timeline points

Evaluation model dimensions	Indicators	Corresponding research question	Outcome measures	Data collection methods	Timepoint of assessment
Effectiveness	Knowledge and perceived confidence in QI	Primary research question— To what extent does the MOOC improve learner’s knowledge and understanding of QI approaches and increasing perceived confidence in participating in QI initiatives?	Knowledge assessment—subjective/self-report and objective.	Survey items	Pre-MOOC and post-MOOC (immediate)
Reach	Learners’ characteristics	Secondary research question— What are the characteristics of the learners taking the MOOC?	Sociodemographic information—age, country, gender, language, education level, employment.	Survey items	Pre-MOOC
Reaction	Self-efficacy and motivation	Secondary research question— How did learners react to the course?	Reasons for doing the MOOC. Self-efficacy in learner’s ability to dedicate time and complete the course.	Survey items Qualitative interview data	Pre-MOOC and post-MOOC (immediate)
	Satisfaction and relevance		Satisfaction with learning experience and relevance to practice. How participants valued the course—strengths and weaknesses, areas of improvement. Feedback on course content, layout, format and design.
Learning	Collaborative learning	Secondary research question—How did the learners learn and how did they engage with other learners?	Attitudes and experiences of engaging with others on the course and asking for help. Collaborative learning—advantages and disadvantages.	Survey items	Pre-MOOC and post-MOOC (immediate and 3-month follow-up)
	Higher order learning		Perceptions of whether higher order learning was achieved during the course—apply new information to new situations, acquired new knowledge and understanding of QI.	Qualitative interview data
	Reflective and integrative learning		Connected their learning to problems that could be addressed by QI, better understanding of how a QI problem might look from another person perspective (eg, patient), learnt something that changed the way they understood a concept or idea, connected ideas from the course to prior knowledge and experience.
	Capability		The degree to which participants acquire the knowledge and confidence to engage in QI efforts based on their participation in the MOOC.
Maintenance (sustainable)	Long-term effects of the MOOC	Secondary research question— What evidence suggests that learners retained knowledge acquired from the course?	Learner’s confidence in their ability to design, implement, sustain QI activities.	Survey items Qualitative interview data	Post-MOOC (immediate and 3-month follow-up)
Behaviour	Postcourse practices in work environment and professional practice	Secondary research question—What evidence suggests that the MOOC increased participation in QI initiatives?	Perceived self-efficacy, motivation, confidence in initiating/implementing QI activities. Impact of the MOOC on work/practice and ability to influence others in QI. Barriers and facilitators to implementing QI projects.	Survey items Qualitative interview data	Post-MOOC (immediate and 3-month follow-up)

MOOC, massive open online course; QI, quality improvement.