Program Evaluation Through the Use of Logic Models

Abstract

Introduction:

In order to effectively design and evaluate the effectiveness of a new clinical program or intervention, pharmacists must be equipped with the skills and knowledge that are obtained by familiarity and use of a logic model. Currently, most pharmacy school curricula do not include logic model exercises to instill these necessary skills into the knowledge base of doctor of pharmacy students. This report provides understanding of how a logic model can be permanently implemented into pharmacy curricula in order to develop critical thinking skills that will allow students to become more well-rounded in their future practice of pharmacy.

Methods:

A 23-point questionnaire was developed by the principal investigator, primarily based on feedback from the student reflection papers and areas of interest with the author’s prior experience and use of logic models. They were distributed to assess the knowledge, attitudes, and perspectives of students enrolled or previously enrolled in the course.

Results:

Questionnaires were received from 128 students, representing approximately 32% of those provided the opportunity to participate. The majority of students (72.98%) viewed the potential benefits of learning about logic models favorably. Overall, 64.86% of students agreed that the experience gained through constructing their own logic model was an intellectually stimulating activity.

Conclusions:

The logic model is an effective tool that can be used to teach pharmacy students how planned program development would contribute to combating various public health issues.

Introduction

When designing a new intervention or clinical program, pharmacists must be familiar with how to design, implement, and evaluate the new venture in order to justify its effectiveness and impact. A logic model is one tool capable of doing this. More specifically, a logic model is a systematic and visual way to represent the relationship between the resources (e.g., human, financial, community) available to operate the program, the actions and activities required to implement the program, and the changes or results to be achieved (outcomes).¹

As of July 2020, there are 141 United States-based colleges and school of pharmacy with accredited professional degree programs, with only six programs located at a Historically Black College/University (HBCU).² Florida A&M University (FAMU) College of Pharmacy & Pharmaceutical Sciences is a mid-size HBCU with two campuses located in the southern region of the university, and is a fully accredited program through the Accreditation Council for Pharmacy Education (ACPE).² During the time of this study, the Introduction to Public Health Course was a required course for students to take in the fall semester of their second professional year (P2) of pharmacy school (since the completion of the study, the course has been moved to the fall of the third professional year [P3]). The pharmacy curricula focus heavily on medication therapy management but not as much on how to design, implement, and evaluate the effectiveness of the programs and services that they provide or may provide in the future. In order to familiarize students with the use of logic models, a didactic module was developed to provide students an opportunity to construct a logic model to maximize the understanding of its use. The learning experience was implemented as follows: (1) lectures explaining the history and use of logic models with a focus on community program development were given by the professor; (2) students were then assigned to groups in order to identify a community need and then to construct a logic model to describe its implementation and evaluation; and (3) students then assessed their views to ascertain if they felt the logic model tool would be useful in practice.

Introduction of logic models as a theoretical framework can be used to rationalize how a desired intervention and developed program can yield intended results and patient outcomes. Incorporating the use of logic models in the pharmacy curriculum will enable colleges of pharmacy to meet ACPE Standard 1 (Foundational Knowledge), Standard 2 (Essentials for Practice and Care), Standard 3 (Approach to Practice and Care), as well as Domain 2.3 (Health and Wellness Promoter), Domain 2.4 (Population-based Care), and Domain 3.4 (Interprofessional Collaboration) of the Center for Advancement of Pharmacy Education Outcomes. Using logic models allows students to gain a higher level of understanding, critical thinking, and program development skills to address public health concerns.³ The goal of this study was to assess the attitudes and feelings of pharmacy students towards the use of logic models as a tool for program development and utilization in pharmacy practice.

Methods

A new learning experience concerning logic models was included in a required two-credit hour public health course for P2 students. A lecture was provided that included the history and use of logic models in public health and its applicability to evaluating pharmacy programs, services, and interventions. A sample logic model (Fig. 1) was used for instructional purposes to provide the student an opportunity to see a community-based model recommended for use in actual practice.⁴ Since the course is also concerned with health disparities and population health issues, students were assigned to work in groups and tasked with designing a community-based program to address a potential health disparity within a community.

Fig. 1.

Sample logic model.⁴

In prior years, student feedback was collected regarding logic models through a reflection paper on their experience. A 23-point questionnaire was developed by the principal investigator, primarily based on feedback from the student reflection papers and areas of interest with the author’s prior experience and use of logic models. They were distributed to assess the knowledge, attitudes, and perspectives of students enrolled or previously enrolled in the course. The purpose of the questionnaire was to validate key topics in the use of logic models in program development by pharmacy students within various communities that face health disparities. The questionnaire collected demographic information and student responses through the use of a Likert scale (Table 1). The questionnaire was administered using an online survey platform, over a 30-day open period for students to respond to an email request for participation. Students who agreed to participate signed an informed consent agreement, and the overall project was approved by the FAMU Institutional Review Board. The inclusion criteria included students who were enrolled in their second, third, or fourth professional year for the 2016–2017 academic year, completed the Introduction to Public Health in Pharmacy course, and participated in the logic model program development group project. The 2016–2017 academic year was selected because there were over 350 eligible student participants for the study, with each participant responding anonymously to the questionnaire. The exclusion criteria included students who were enrolled in their first professional year for the 2016–2017 academic year. These students were excluded because they had not yet taken the Introduction to Public Health course. Additional exclusions included any student who had not taken the Introduction to Public Health in Pharmacy course, and any individuals who were not currently enrolled in the FAMU College of Pharmacy & Pharmaceutical Sciences during the 2016–2017 academic year.

Table 1.

Logic model questionnaire.

Pharmacy students’ perspectives on their knowledge about logic models:

• Do you have adequate knowledge about logic models for incorporation into pharmacy practice?

• Can you apply your knowledge about logic models to practice?

• Do you need more information about logic models?

• What are your preferred sources of information?

Pharmacy students’ attitude towards logic models:

• The utilization of logic models in developing a public health program will be an opportunity to help the community address a public health problem.

• Developing a public health program through the use of logic models will be a way to promote the practice of pharmacy.

• The utilization of the logic model in developing a public health program will increase collaboration with other healthcare professionals (e.g., physicians, nurses).

• Utilizing logic models in implementing public health programs will be an overwhelming responsibility.

• In the event that you need to create a program for your scope of practice as a pharmacist, how likely are you to use the logic model as a tool to plan, develop, implement, and evaluate your program?

Pharmacy students’ perspectives on their experience using logic models:

• The pros of the logic model include...

• The cons of the logic model include...

• It is the pharmacists’ responsibility to bring awareness to address public health issues in the community through program development and implementation via logic models.

• Overall, the experience gained through logic models has been intellectually stimulating.

• I have felt part of a network and/or team dedicated to developing programs that will address public health problems.

• Overall, the experience gained through use of logic models for program development in the public health area has been well coordinated.

The questionnaire distributed included student demographics (age, sex, race/ethnicity) and students’ professional year of study. The assessment questions reflected on the students’ personal knowledge as it pertained to the logic model, their perception of logic models, and their experience with using the logic model to develop a public health program. A pilot test of the questionnaire in the previous year is what led to the development and execution of the research study.

Results

Descriptive analyses were used to summarize demographic information as well as the knowledge and attitudes of the students regarding logic models.

Demographic information

Questionnaires were received from 128 students representing approximately 32% of those provided the opportunity to participate (questionnaire participation was not a mandatory component of the course). There were 66 P2, 60 P3, and two fourth-year student participants who completed the disseminated questionnaire. The majority of student respondents were female (75.78%), African American (53.91%), aged 18–24 years (53.91%), and located in Tallahassee, Florida (71.88%). The second largest groups of study respondents were identified as being Caucasian (22 participants), aged 25–34 years (49 participants), and were located in Crestview, Florida (the distance learning campus) (Table 2).

Table 2.

Basic demographics.

Characteristics	Number	%
Sex
Male	29	22.66%
Female	97	75.78%
Transgender	0	0%
Race/ethnicity
African American	69	53.91%
Caucasian	22	17.19%
Asian	10	7.81%
Latino/Hispanic	7	5.47%
Hawaiian Native/Pacific Islander	11	8.59%
Native American/Alaskan Native	1	0.78%
Other	11	8.59%
Age group (years)
18–24	69	53.91%
25–34	49	38.28%
35–44	6	4.69%
45–54	4	3.13%
Over 55	0	0%
Professional year classification
P2	66	51.56%
P3	60	46.88%
P4	2	1.56%
Assigned location for the 2016–2017 academic school year
Crestview	18	14.06%
Davie	7	5.47%
Jacksonville	5	3.91%
Orlando/Tampa	6	4.69%
Tallahassee	92	71.88%

P2 = second professional year; P3 = third professional year; P4 = fourth professional year.

Knowledge and attitudes about logic models

The majority of students (72.98%) viewed the potential benefits of learning about logic models favorably. These students believed that they learned new ideas, approaches, and skills that will assist them in implementing a community program once becoming a pharmacist. Moreover, 66.1% and 68% of students agreed respectively that use of this tool is a simple way to monitor program performance and to communicate progress (or lack of progress) in a program (Fig. 2). About 71% of students believed that the logic model provides accountability of a program and assures that program outcomes are being measured appropriately. When it came to utilization of logic models in the development of a future public health program, 79.5% of students agreed that it would provide an opportunity for pharmacists to help the community address a public health problem (Fig. 3). Furthermore, 75.2% of respondents agreed that developing a public health program would increase collaboration with other healthcare professionals for more patient-centered care.

Fig. 2.

Pros of the logic model.

Fig. 3.

The utilization of logic models in developing a public health program will be an opportunity to help the community address a public health problem.

The overwhelming majority of students agreed that it is the pharmacist’s responsibility to bring awareness to and address public health issues in the community through program development and implementation via logic models.

In contrast, 57.69% of students believed that logic models do not capture all qualities of a program. Likewise, 52.88% believed this model does not explain what supported or hindered the process of program development. Interestingly, 56% of students felt that utilizing logic models in implementing public health programs will be an overwhelming responsibility for pharmacists. Overall, 64.86% of students agreed that the experience gained through constructing their own logic model was an intellectually stimulating activity. More than half of the participants agreed that if they were to create a program in the pharmacy setting, they would use the logic model to plan, develop, implement, and evaluate it (Fig. 4).

Fig. 4.

How likely are you to use the logic model as a tool to plan, develop, implement, and evaluate your program?

Discussion

Pharmacy learners must be trained to become outcome oriented in order to successfully practice and administer programs effectively in today’s healthcare environment. Public health professionals are able to use logic models to evaluate program activities and ensure those activities are contributing to the success of accomplishing the overarching goals.⁵ Although the use of logic models in program development and evaluation is a common learning experience in public health curricula and practice, it is not an evaluative tool routinely taught in pharmacy programs. From a pharmacy student perspective, this is an important tool as it provides an opportunity to critically think through the design and implementation of a project and to identify the relationship between program objectives and eventual measurable outcomes. Logic models increase critical thinking skills by allowing the participant to prioritize focus and determine appropriate evaluation tools needed to assess, which will in turn allow for evidence-based recommendations to implement improvements to programs.⁵ Additional detailed descriptions of logic models, definitions, uses, and more explanative information can be found in the literature.^1,6–8 The incorporation of this tool into pharmacy curricula is important and contributes to the need to develop practitioners that are outcome driven in their approach to care and program implementation. This tool can be incorporated across pharmacy curricula in courses that involve disease state management, pharmacy management, and professional development. Although students evaluated the use of this model favorably, it was clear that it represented a different way of problem solving for them. The goal is to permanently include this tool as an important component of this course and to expand its use in other courses in order to supplement critical-thinking and decision-making experiences for students.

Conclusions

As pharmacy students graduate and pursue interprofessional roles, the importance of becoming outcome oriented must be fully realized. The decision-making process in health systems require that practitioners are outcome driven in their approach to care in order to justify their substantive roles to administrators and their place as major contributors to improving patient care. The logic model is an effective tool that can be used to teach pharmacy students how planned program development would contribute to combating various public health issues. Furthermore, the application of logic models can guide pharmacists on how to implement effective interventions in patient care and improve optimal health outcomes.