Development of an Office of Teaching, Learning and Assessment in a Pharmacy School

Abstract

This article describes the background, development, and assessment of an Office of Teaching, Learning and Assessment at the Harrison School of Pharmacy, Auburn University. This case study is intended to provide other colleges and schools with insights that may facilitate the relatively unproblematic development of similar programs.

Teaching has always been a fundamental mission of higher education, but it is only within the last 50 years that many universities, colleges, and schools have sought to enhance educational quality through the establishment of departments or services to support the excellence of teaching, learning, and assessment.^1,2 Rapid advances in technology, the assessment movement of the 1980s,³ and a call to redirect the academy's attention towards the scholarship of teaching,⁴ have caused universities to direct resources towards the enhancement of teaching, learning, and assessment so that by the mid-1990s about one third of colleges and universities had support centers and nearly all institutions had undertaken faculty development in these areas.⁵ Individual teaching and learning centers offer a great range of services ranging from lectures or workshops on educational philosophy, psychology, and innovation, to course planning, instructional methods, and assessment.^6,7

The trends in medical education are similar. In addition to the support services described above, offices of medical education and medical education research report significant involvement in curriculum and assessment administration, as well as the scholarship of education (e-mail correspondence, SIU School of Medicine).

ASSESSMENT AND EDUCATIONAL DEVELOPMENT

Pharmacy schools have not been able to respond comprehensively with formal support for their educational missions because they have limited resources. However, teaching, learning, and assessment have not been neglected. In response to the assessment movement that began in the 1980s, some individuals have either been reassigned or hired into assessment positions (often part-time) and almost half of the pharmacy schools surveyed in 2000 had assessment committees.⁸ In many schools, educationally inclined individuals have assumed an informal role in providing educational development for their colleagues and, in some, formal offices for education and assessment have been instituted.

Currently, many schools are beginning to consider more formal mechanisms and structures to undertake, assess, and assure the quality of teaching and learning. At the 2004 annual AACP meeting in Salt Lake City, more than 100 people attended a presentation by the Auburn University Harrison School of Pharmacy (AUHSOP) entitled “Development of an Office of Teaching, Learning and Assessment.” To generate dialogue, we asked participants to respond to the following questions:

• What is your current school's vision for assessment and the development of teaching and learning? What changes might be desirable? What needs to be done to achieve your vision?

• What barriers exist? How can they be overcome?

• Can your current human resources support your vision? What resources do you need?

• How do you promote and market your vision within your school and university?

In the following sections, the attendees' responses will provide context for a discussion of the AUHSOP's experiences in developing an Office of Teaching, Learning, and Assessment (OTLA). Using the Auburn experience as a case study, it is expected that the reader will gain insights about mission and purpose; development strategies; implementation challenges; criteria for quality and success; potential services and outcomes for the OTLA.

IMPLEMENTATION OF THE OTLA

Context

The 1997 ACPE Standards' emphasis on outcomes, and the implementation of an entry-level doctor of pharmacy degree, raised the bar on our School's accountability. The faculty culture (based on an outdated philosophy of academic freedom), curriculum development processes, curricular content, and teaching and learning methods would need to change to ensure that graduates could demonstrate prescribed knowledge, abilities, and attitudes upon graduation. The need for an assessment- and data-driven continuous improvement process also became apparent. However a comprehensive assessment program, incorporating a continuous improvement process, can generate massive amounts of data. Who would take responsibility for managing this information? How would it be possible to ensure timely and appropriate feedback?

We could not expect all of our faculty members to become experts in outcomes-centered education and assessment. However we were also wary of the problem of hiring educational experts or seeking the services of assessment consultants who were ignorant of the needs of the profession and so deeply engrained in education that they would find it difficult to communicate and serve the school effectively. The development of an OTLA was the AUHSOP's response to this challenge.

In September 1997, a new Associate Dean for Academic and Student Affairs was hired at the time that the entry-level PharmD program was initiated with administrative responsibility for curriculum, assessment, quality improvement, and faculty development, along with a variety of other academic and student affairs responsibilities. However, these activities were far more time consuming than what had initially been estimated and one Associate Dean could not possibly manage.

To deal with this, a Director of Assessment was hired in February 2001. The responsibilities assigned to this individual included assessment and continuous quality improvement, with the other responsibilities still assigned to the Associate Dean for Academic and Student Affairs. The Director of Assessment had significant experiences with accreditation processes, but did not have a background in pharmacy or another health profession. The need for a pharmacy presence resulted in a senior member of the Department of Pharmacy Practice being appointed in May 2001 to a 30% position as Senior Faculty Advisor for Assessment. In October 2001, the OTLA was created incorporating these individuals along with the Associate Dean, and the former Director of Assessment was named the director of the Office.

As it became apparent that the OTLA needed broader expertise and background in pharmacy education, the decision was made to replace the Director of Assessment with an Assistant Dean with a pharmacy background and to hire an Assistant Director with an education background. In August 2005, the Assistant Dean for Education assumed primary responsibility for directing continuous improvement of the quality of the academic program, teaching, and learning and was appointed as Associate Dean for Education. OTLA staffing now includes the Associate Dean for Education (and Director of the OTLA), an Assistant Director, and a support staff member. In addition, the AUHSOP has an OTLA Advisory Committee of faculty members that works closely with OTLA.

In addition to providing comprehensive assessment services, an OTLA could help AUHSOP achieve its vision by providing educational resources, theory, and best practices directed towards addressing current and emerging educational problems and issues, faculty development, continuous quality improvement support to our admissions and curriculum committees, and support for increasing the intensity of scholarship directed towards teaching.

In theory, an appropriately staffed OTLA could provide considerable educational support expertise for the School. Specifically, we wished the OTLA to:

• develop and direct an evidence-based comprehensive assessment program;

• manage the problem-based learning series in the third-professional year (P3), including facilitator training, case development, and scheduling;

• develop and administer a comprehensive performance assessment (Milestone Examination);

• provide educational research consultation and collaboration;

• develop and direct a comprehensive, outcomes-based faculty development program to improve teaching and assessment;

• develop and evaluate a faculty college to prepare each new faculty member for the challenges of pharmacy education;

• support our curriculum committee in curricular development; and;

• work with faculty to support and ensure the sustainability of educational change initiatives

Attendees at the AACP discussion shared similar expectations. In some cases, schools used university-level teaching and assessment offices to help them. Many revealed that these functions, especially assessment, were undertaken by a loose network of people in a somewhat haphazard and reactive way (ie, response to accreditation). There was consensus that it would be preferable for schools to mount concerted and ongoing efforts to assess and improve the quality of their educational programs.

This range of services requires a considerable array of abilities among the OTLA staff ^9,10:

• design and develop strategies that promote individual, pedagogical, curricular, and organizational growth;

• organize and implement programs, projects, and studies;

• plan and deliver oral presentations;

• design and conduct research about teaching and learning;

• assess and evaluate needs, curriculum, instruction, and outcomes;

• produce written publications, both for local consumption and for submission to peer-reviewed journals;

• establish and maintain collegial and consulting relationships;

• manage data; and

• understand, support, and facilitate consensus about the school's direction and philosophy.

In addition, there was a requirement for content knowledge of health professions education, in particular pharmacy, learning theory, and best practices, faculty development, clinical reasoning, teamwork communications, evidence-based practice, change management, test construction and psychometrics, and organizational psychology.

CRITERIA FOR SUCCESS

What enables an OTLA to be effective? Experience at the AUHSOP and other professional schools¹¹ suggest the following are crucial:

Credibility.

The OTLA staff must have and continue to build credibility with faculty, students, and administrators. We believed that the appropriate persons would either be credentialed in education (at either the master's or doctoral level) or be a health professions faculty member with developed expertise. While expertise in teaching and assessment is essential, the hiring of educational specialists can have some pitfalls. Coming from a different academic culture and speaking a different language (“education-speak”) can make integration difficult. On the other hand, hiring a faculty member into the position can also be problematic because their educational credentials can be questioned. Credibility with faculty needs to be developed. In most cases, this is done by participating in teaching themselves (“walking the talk”) and by undertaking scholarship in assessment, curricular development, and teaching. Credibility with the students is also important. For assessments of courses and teaching to be successful, they must believe that their comments (1) are read, (2) are kept confidential, and (3) can make a difference.

Value added.

The OTLA must be seen to contribute to the mission of the institution through its services to students, faculty members, the school, the university, the community, and the profession. The dean's positive assessment of the office is important but, because leadership changes, it is important that the faculty as a whole perceive the value of the educational research and development undertaken by the OTLA. It is imperative that the office measures the outcomes of its own initiatives and communicates them to the faculty.

Critical mass of educational scholars.

With only 2 staff members, an OTLA by itself is not enough to sustain a focus on the scholarship of teaching and assessment. The office must begin to develop faculty associates. In addition to viewing their own teaching as scholarship, these individuals can play a role in the development activities of the office.

Collegiality.

OTLA members must be seen as unconditionally supportive of faculty success. An important factor is in this perception is power. OTLA personnel must not have authority or power over the individuals they are to serve. Thus situated, OTLA staff members are in an excellent position to help faculty members develop collegial networks within the school. By their willing presence and high visibility, the OTLA staff can bring people together in collaboration concerning teaching, learning, and assessment.

Visibility.

It is important for the OTLA to be visible within the school, institution, and profession. Staff members must establish regular, consistent, and quality communication with faculty members through tools such as assessment reports at faculty meetings, weekly teaching tips, monthly faculty development workshops, and active contribution on committees. They can increase their visibility in the university by working with existing (and well-respected) centers/offices on campus.

Innovation.

The OTLA must be seen as being innovative, serving to support internal and external changes. To be successful in this role, the staff must also be seen as objective. This perception can be cultivated by members not aligning with a particular department.

Systematicity.

Many assessment efforts are incomplete. Studies are undertaken but the data are not evaluated; outcomes are reported, but nothing is done to create change based on the report. Therefore, an OTLA must be systematic in assuring that every step of the assessment cycle is complete, well documented, and implemented so that continuing quality improvement is a reality.

CHALLENGES

Four major challenges in implementing an OTLA were identified by Auburn and the conference meeting attendees:

Generating demand for educational expertise.

Demand for expertise in teaching, learning, and assessment expertise varies from institution to institution. At Auburn, the initial drive originated at the administrative level, but in many schools, faculty members bemoan a lack of administrative interest and support in establishing an OTLA. Many schools identify a general need to encourage buy-in from all faculty members and all departments and to develop a culture of assessment and continuous quality improvement among all faculty members. Strategies identified included: creating coalitions to generate a stronger voice for education; making appointments only to people with proper credentials, experience, and expertise; creating a shared vision; capitalizing on others' (published) success; and seeking synergy with university-level teaching offices.

Departmental divisions.

Many schools' departmental structure cannot accommodate people whose focus is on educational support and scholarship. In some departments, it would be difficult to generate sufficient respect for education-focused scholarship to make it feasible to hire someone into a faculty position. Conversely, if faculty members shift their priorities in order to be hired into staff positions, it can severely limit their promotional opportunities. Therefore, in many cases, the institution of a formal teaching and assessment office will require some restructuring. At Auburn, it was decided to initially hire the OTLA staff as nontenure-track contractual faculty members without departmental affiliations. This OTLA structure was selected by the AUHSOP's Executive Team to facilitate better articulation of the Office with the School's Mission, Vision, and Strategic Plan. Since much of the required curriculum, assessment, and faculty development work was focused around the School's PharmD program (an interdepartmental program), it was thought that having OTLA located in the Dean's Office rather than within a particular academic department would be more advantageous. Contractual nontenure-track positions were initially used because the School's administration was uncertain as to the types of personnel that would be successful. It was also felt that the initial issues of program development and service would consume a great deal of the time of OTLA professional staff and that hiring individuals into tenure-track positions would carry responsibilities for scholarly productivity that would not be achievable. A disadvantage of the OTLA structure is that it may be viewed by some faculty members as an arm of administration to carry out a specific agenda. This may result in decreased credibility with some faculty members, and may deter some individuals from applying for such positions.

Overcoming resistance to change.

Change can be threatening. A formal OTLA would require faculty members to go public about what was once their private business, subjecting them to scrutiny and perhaps the need to make major changes in the way they undertake their teaching assignments. Ways in which resistance can be handled include: starting with some easy and highly public wins; showing faculty how the OTLA services can make their lives easier; setting realistic goals and timelines; beginning incrementally; involving faculty members from the very beginning and maintaining continuing dialogue; making best use of accreditation pressures (negative accreditation results can be helpful!); and using the OTLA to orient and mentor new faculty members. The most critical people in this process are the department chairs, because it is these people who are empowered to put in force the initiatives begun by the OTLA. It is for this reason that it is critical that the director of the OTLA have access to, or be a member of, the executive committee.

Resources.

Limited resources were identified as a major barrier by all participants in the AACP discussion. In addition to staff, an OTLA needs an operating budget and infrastructure: space, computer support, secretarial and administrative support (eg, for meeting coordination, data entry, mailings, scheduling, etc). At Auburn it was clear that we would need to make support for OTLA a priority within AUHSOP's budget. Initially it was unclear what the expense would entail; however, it became clear that the workload necessary to do a credible job exceeded expectations of those initially assigned the responsibilities. Justification of the expense has become clearer over time in that we are not asking faculty members who are not specializing in curricular design and assessment to become experts, allowing them to continue with their other mission-based responsibilities. The overall productivity of OTLA and timeliness of curricular and assessment initiatives have increased the faculty's ability to address these areas in a timely manner. Annual direct costs of the assessment activities at AUHSOP exceed $225,000.

Finding Personnel

Identifying personnel with the appropriate understanding of education principles who also have a sufficient understanding of health professions education has been a major challenge. Schools and colleges of pharmacy wanting to establish an OTLA or to hire individuals to help with assessment, curriculum, or faculty development should consider recruitment from the following sources.

Other pharmacy schools.

This can be difficult as there are few qualified individuals currently working in pharmacy schools.

Medical school offices of education.

Medical schools are a particularly attractive source for qualified individuals, but these individuals may be hard to recruit unless an offer is particularly attractive.

Colleges of education and university-level centers for teaching and learning.

It may be possible to hire good individuals from other education-related departments in colleges and universities.

Promotion of faculty members from within the school.

This must be undertaken with care. The competencies required are not the same as those for a faculty member and accepting such a position would bring significant changes in all aspects of job and career. This is usually not a good place for a faculty member who is trying to hold on for a few years prior to retirement. Faculty members who are interested in entering this area should be encouraged to do so if they can consider the work as scholarship. Given the available pharmacy-oriented manpower resources, engaging interested faculty members in a formal training program is advisable.

AUHSOP OTLA: CURRENT STATUS

Although the OTLA is involved at many levels at AUHSOP, our efforts are focused in 5 primary areas: assessment and evaluation, faculty development, curriculum development, program administration, and scholarship.

Assessment and Evaluation

Assessment and evaluation are the most central functions of the OTLA. The goal is to support continuous improvement in student learning and the quality of all aspects of the program. The OTLA distinguishes assessment and evaluation as follows¹²:

Assessment: The process of measuring and analyzing a performance or product to provide feedback to improve future performance or products.

Evaluation: The process of measuring the quality of a performance, work product, or use of a process against a set of standards to make a judgment or determination if, or to what level, the standards have been met.

The OTLA is attempting to move faculty members away from an evaluation paradigm and towards assessment. For example, faculty members are asked to use classroom assessment techniques more frequently. Assessment techniques such as the “Muddiest Point” and “One-Minute Paper” and the submission of drafts of SOAP notes enable students to receive feedback on learning performance that is not linked to grades. As another example, faculty members are encouraged to seek regular feedback from students about their teaching during the semester. The result is that student evaluations are less of a surprise and more positive. By focusing more on feedback mechanisms, faculty members become collaborators in learning and facilitators of growth. When the focus is not on assigning grades but instead on providing feedback for improvement, students begin to see faculty members less as adversaries and more as allies.

Of course, evaluation is necessary, and the OTLA intends to assume increasing responsibility for this function. Potential approaches include the administration of a progress test^13,14 twice per semester that is not linked to instruction, short answer pharmacotherapy reasoning examinations,¹⁵ objective “key-feature”examinations,^16-19 and multistation Objective Structured Clinical Examinations (OSCEs).²⁰ While the OTLA currently manages an annual 9-station OSCE that is administered to students in the P2 to P4 years (the Milestone Assessment in which each station assesses the ability to solve health and drug-related problems and communicate professionally with standardized patients), other assessment methods need to be employed to have sufficient validity and reliability to be used for summative purposes.

Assessment of Teaching

Ongoing assessment is essential to ensure all faculty members are comfortable in the classroom and are creating an atmosphere conducive to learning for all students. Since the Faculty College (see below for description) was introduced in 2002, each of the new faculty members has been assessed at least once during his/her first year of teaching. Many have been assessed several times. We believe the key to successful assessment involves 3 steps:

• (1) ;Meet with the teacher before the lesson and ascertain if there are any particular areas of teaching that he/she would like OTLA to focus on in order to provide suggestions on how to improve this particular area. In addition, understanding the major learning objective(s) of the lesson is very important in making an informed assessment of whether the students met the objective(s).

• (2) ;Observe all aspects of the teacher/learner experience objectively, looking for strengths and areas for improvement. The preparedness of the learners must be taken into account as well as the abilities of the teacher.

• (3) ;Provide timely, honest feedback focusing on how the teacher can improve his/her teaching and make the learning experience more effective for the students.

During the first 4 years of the OTLA's existence, several dozen assessments were performed and feedback was provided. In most cases, the OTLA observed their suggestions for improvement being implemented in second and subsequent assessments of teachers. In many regards, the program has been very successful, but we can improve greatly by broadening the scope of our assessment program. To date, every assessment has been done at the request of the individual faculty member. We need to institutionalize the program to ensure that every faculty member involved in teaching is assessed periodically in order to assure a quality learning environment for all students in all courses. This requires the support of each department head in order to be successful.

Course and Teacher Evaluations

Auburn University requires that we ask certain questions of our students for every class taught at the school. Even though the University refers to their evaluation as “Course Evaluations” all of the questions relate directly to the teacher and evaluate only very superficial instructional techniques. For this reason, 4 years ago, OTLA began to do separate course evaluations and teacher evaluations for each course taught at the AUHSOP. We now get usable feedback from the students on the course and each teacher who provides instruction in the course. We include the University questions to meet our requirement, but have added several other evidence-based questions that better assess the teacher/learner experience. In addition, a copy of the evaluation instrument is sent to each teacher before the evaluation so they can make suggestions as to the addition and deletion of certain items. This allows us to ask the questions that each teacher feels are important to receive feedback about.

Outcomes Evaluation

The evaluation of curricular outcomes must be undertaken at several levels. For example, the Milestone Examination provides information about knowledge and skill acquisition. The analysis of Board Examination pass rates provides information about knowledge upon graduation. However, it is only through long-term tracking of our graduates' competence, professional activities, and career histories that we can get a sense of the overall success of our program. Currently, the OTLA administers an outcomes-centered alumni survey every 3 years that includes a demographics section, information about their current practice, and their assessment of how well the program prepared them to meet our intended outcomes as well as the challenges of their current position.

Faculty Development

OTLA's first step in a robust faculty development program was the creation of the Faculty College. Begun in 2002, the Faculty College is a 4-week program intended to prepare each newly hired faculty member for success in pharmacy education. After participating, every new faculty member should feel more comfortable and be more effective in the classroom, regardless of how much or how little teaching experience he/she may have had previously.

One of the unforeseen advantages of the Faculty College has been the acceptance of OTLA and our vision by participation of up to 9 new faculty members each year. Having completed 4 sessions of the Faculty College, we have a nucleus of over 20 faculty members who have bought into what OTLA is trying to do and spread the word among their colleagues.

We are now expanding the Faculty College beyond the initial 4-week program. Through mentoring, peer assessment, monthly workshops, teaching tips e-mailed to all faculty members, and assessment updates at each faculty meeting, we hope to involve more and more faculty members in an ongoing process. Recognizing that faculty development is a process and not a product, OTLA's goal is to fully integrate faculty development, curriculum development, and assessment into one process that focuses on outcomes-based education centered on problem-based learning. The School is supporting this effort by designating time for faculty development in a way that class and clinic schedules do not interfere.

Curriculum Design

OTLA facilitated the adoption of a DACUM (“Developing a Curriculum”) process to develop our revised outcomes-based curriculum that was implemented beginning fall semester 2005.²¹ DACUM is an approach to curricular development that is based in occupational analysis. After identifying the responsibilities and tasks that must be performed by a practitioner, it establishes a “backwards-designed” curriculum that provides students with adequate opportunities to develop competence in undertaking these responsibilities and tasks. As a result of the DACUM process, the AUHSOP curriculum is being completely restructured “from the bottom up,” in true outcomes-based fashion, with all phases and courses being reorganized and reconceptualized. In the new curriculum, the fundamental disciplines are being integrated with practice-centered problems with the goal of creating a knowledge paradigm that is uniquely pharmacy's and explicitly linked to pharmaceutical care.

Course Administration

The P3 year of the AUHSOP curriculum is completely problem-based. One of the responsibilities of the OTLA is to administer this course and ensure consistency of the cases and learning process given that both faculty and hired facilitators manage the student learning teams.

Education as Scholarship

Auburn University and the Harrison School of Pharmacy support scholarship, broadly defined. OTLA staff members are engaged individually in the scholarship of integration and teaching, and they promote the educational scholarship of faculty members through assistance and collaboration in grant writing; collaboration in educational research; and promotion of classroom research. It is the goal of AUHSOP to continue to develop the scholarship mission.

LESSONS LEARNED

Get the Right People the First Time

When developing a new office to support teaching, learning and assessment it is critical to have a clear understanding of the performance criteria against which applicants will be measured. Although working knowledge of psychometrics is useful, it is not nearly as important as knowledge of the process of continuous quality improvement. While a pharmacy background is not essential, a good understanding of the unique requirements of professional education is critical. Perhaps most importantly, if OTLA staff members are to be able to collaborate effectively with faculty members, academic credibility, strong interpersonal skills, and documented success in leading sustainable change are probably the most important characteristics to seek.

There is a high cost to choosing the wrong personnel. New assessment initiatives naturally encounter mistrust, and when it becomes clear to faculty members that the wrong people are hired, they become much more resistant to change.

Begin With Clear Goals and Criteria for Success

In the development of many teaching, learning, and assessment initiatives, it is often not clear how success should look. As a result, what one often gets from teaching, learning, and assessment offices is a menu of offerings that varies from one year to the next…and no discernable outcomes. The solution to this problem is to define criteria for success. To cite 2 examples: successful assessment of any particular process or outcome should result in documented improvements; and successful faculty development should result in improved student learning or enhanced faculty success.

Be Systematic and Strategic

The enhancement of teaching, learning, and assessment can consume a lot of faculty time and effort and most colleges and schools of pharmacy do not have sufficient resources to do everything they desire. As a result, it is critical to use a systematic planning process and prioritize needs. Too often, assessment offices measure so many things that all available resources are spent in measurement and nothing changes. Appropriate planning limits attention to a handful of critical issues and ensures that there are enough resources to translate assessment into robust interventions that lead to sustainable change.

FUTURE INITIATIVES

Over the last year, the OTLA has facilitated a novel self-study process that has resulted in the development of criteria for performance in every major quality area in all the programs offered by the School. It is now the responsibility of the OTLA to successfully facilitate its reflective assessment process “full-circle” to result in sustainable improvements relative to the criteria. The development of the comprehensive assessment program will be described in a subsequent manuscript.

As a result of reflection on the issues and problems pharmacy schools face in implementing offices of teaching, learning, and assessment, we have become convinced that there is a clear need for personnel who are trained appropriately in modern pharmacy education. Therefore, we are planning to develop a master of science degree program in professional education in cooperation with Auburn's College of Education.

CONCLUSION

Given the increasing necessity for accountability in providing quality professional education, pharmacy schools need to develop educational support systems and educational scholarship as a career track for a defined subset of “academic pharmacists.” In creating an Office of Teaching, Learning and Assessment, we have taken a first step. The professionals who staff our Office provide a wide variety of evidence-based educational support services. However, we need to go much further. The issue is that the evidence upon which we base the vast majority of our recommendations comes from disciplines that are different from our own. In some cases, the differences may not be significant. However, pharmacy is a unique profession, with distinct competencies. Pharmacy education has opportunities and limitations peculiar to our context. In some significant ways, we have the need to develop our own theories and methods. This is a requirement that cannot be met by one or even a handful of “academic pharmacy” departments. Globally applicable theories and tools will require a critical mass of contributors; hopefully, a few at each institution.

In some ways, we are fortunate as a discipline to be starting this kind of initiative so late in the game. We have available to us the experiences of other disciplines, notably academic medicine and academic education departments. Hopefully, we can learn from their successes and mistakes, and as a result, have fewer growing pains.