Program Evaluation Models: Evaluating Processes and Outcomes in Graduate Medical Education

Rip Out Action Items

Program directors, designated institutional officials, and GME educators should:

1. Recognize that using program evaluation models yield insights that inform implementation or assess outcomes.

2. Select a model based on the question the evaluation seeks to answer (process and/or outcomes).

3. Plan your evaluation during the development phase and get input from stakeholder groups.

The Challenge

Program directors, designated intuitional officials (DIOs), and other graduate medical education (GME) educators strive to ensure the quality of their training programs. For example, faculty at annual program evaluation retreats often ask, “Is our training being implemented as we intended?” or “Is our training really working?” They may not recognize that these are 2 different types of questions; both may be answered differently depending on the program evaluation model used. Each model provides a different vantage point to inform the development and implementation of an educational initiative (process evaluation) or from which to judge the value or effectiveness of an educational initiative (outcome evaluation). Familiarity with commonly used models can optimize GME educators' ability to obtain actionable answers to a program's evaluation questions.

What Is Known

Two program evaluation models, appreciative inquiry and the logic model, have been successfully used to frame GME program evaluations (Table).

Table.

Key Features, Assumptions, Applications, and Pros and Cons of Evaluation Models

Appreciative Inquiry	Logic Model
Key feature: Enables program stakeholders to articulate what success looks and feels like, and to imagine possibilities for the future (alternative to strengths/weaknesses approach) using 4 steps. ○   Inquire (What is?) ○   Imagine (What might be?) ○   Innovate (What could be?) ○   Implement (How do we navigate change?) Assumption: Focusing on assets taps into the human potential for change and better informs program improvement than focusing on deficits or problems.	Key feature: Illustrates program development, implementation, and evaluation by using a single flow chart to display the relationship among program components: ○   Inputs (existing resources) ○   Activities (specific initiatives) ○   Outputs (program deliverables) ○   Outcomes (short and long term) Assumption: If resources are sufficient and activities are sound, then desired short-term and long-term outcomes should be achieved in a logical sequence.
Application of appreciative inquiry for process evaluation: Annual Program Evaluation section on graduate performance: Inquire (alumni interviews): “Describe an experience—postgraduation—when you realized you were prepared to practice autonomously.” Imagine/Innovate: “What do you wish the program had included, making it even better/more useful?” Implement: More and/or better curricular improvements compared to strengths and weaknesses of evaluation methods.	Application of logic model for outcome evaluation: New faculty development series: Inputs: buy-in from department leadership. Activities: initial pilot → implement 6 workshops. Outputs: attendance, evaluations forms. Outcomes: More faculty are (1) trying new teaching strategies on rounds; (2) giving feedback that trainees perceive as useful for improving clinical care; and (3) receiving higher rotation evaluation scores.
Pros (+) and Cons (-) + Focuses on assets not deficits. + Creates opportunity for people to be heard, to dream, and to act. - May be perceived as “rose-colored glasses.” - Familiarity with qualitative data is helpful.	Pros (+) and Cons (-) + Helpful visual “snapshot” of your program. + Flexible and adaptable in its use. - Can be unwieldy and cumbersome. - By focusing on outcomes, can overlook important processes.

How You Can Start TODAY

1. Consider. Sketch out how each program evaluation model “fits” with the program you are evaluating, and with the type of question you are asking. For example, fill a blank table comprised of columns labeled with inputs, activities, outputs, and outcomes to visualize how your program might align with the logic model. Or pilot some appreciative inquiry questions with program participants: Are responses to these questions likely to yield information that will be useful for program evaluation?

2. Begin systematically. Identify a small program or initiative whose implementation or effectiveness has not been evaluated and pick one of the models discussed. Convene a diverse group of stakeholders (eg, faculty, trainees, program coordinators) to discuss feasibility of evaluation and how evaluation data will be utilized. Outline your evaluation plan, then implement.

What You Can Do LONG TERM

1. Familiarize yourself with different program evaluation models. Utilize published resources in medical education and evaluation³ and attend evaluation-oriented workshops/meetings.

2. Get involved in the evaluation community. Take advantage of opportunities with program evaluators at your institution or through your national education organization(s).