Incorporating ACPE Standards in a Holistic Approach to School Operations and Accreditation

Abstract

Auburn University Harrison School of Pharmacy (AUHSOP) implemented a comprehensive, holistic process of continuous quality improvement (CQI) for its program using the Accreditation Council for Pharmacy Education's (ACPE) Accreditation Standards as the foundation. The process served as a way to increase the impact of the accreditation standards through continuous monitoring and programmatic improvements. Internally generated standards were also included in the CQI process. Design of the program included defining the program, establishing quality criteria, developing a table of measures that provided evaluation metrics, and assessing the program annually. Each year, faculty members, staff, students and other constituents assess the program for strengths, areas for improvement and insights. A CQI report is created and is made available to constituents, including ACPE through the Assessment and Management System (AAMS). Areas for improvement and new goals are then incorporated into the school's strategic plan and committee assignments for the year. Some of the suggestions included making the program appraisal more meaningful and on a continuous, ongoing basis compared to doing an appraisal for accreditation purposes at 8-year intervals. The process has increased constituent awareness of all aspects of the program and prompted numerous improvements since its adoption. A culture of assessment resulted from the process.

INTRODUCTION

Continuous quality improvement (CQI) is a systematic process that identifies, describes, and analyzes program strengths and areas for improvement and then tests and implements solutions, leading to organizational learning.¹ It consists of an ongoing cycle of collecting data and using it to make decisions that incrementally improve program processes so as to better achieve intended outcomes.^2,3 Continuous, small improvements yield fewer unanticipated problems and better results more quickly and sustainably when compared with the more typical program evaluation approaches that occur once every few years.. Another benefit is that it engenders a “CQI frame of mind,” providing a strong foundation for the development of an assessment culture that embraces continuous improvement, innovation and long-term planning.⁴

Deming’s work on CQI has had a well-documented and profound impact on industry, including organized health care.⁵ Many institutions of higher education have embraced CQI since Deming’s approach was introduced.⁶ In fact, a search of the ERIC Clearinghouse on Higher Education revealed that over 37,000 articles have been published by the Academy regarding implementation and use of CQI.

Education in the health professions has generally lagged behind general education’s embrace of CQI as evidenced by the paucity of publications from the primary health care disciplines. Generally, most CQI activities reported address how CQI principles have been used to improve specific courses or segments of a curriculum (eg, experiential learning).

Medical and nursing educators have published on the use of CQI approaches to address program improvements and criteria of their respective accreditation standards. The Commission on Collegiate Nursing Education (CCNE) requires an interim report documenting continued improvement activities and continued compliance with CCNE standards and key elements. These reports reflect the process of maintaining excellence in the education program.^7-9 The Liaison Committee on Medical Education requires internal CQI to ensure effective monitoring of the medical education program’s compliance with accreditation requirements. However, most assessed criteria are operational in nature (eg, grades reported on time, student awareness of mistreatment policies). The American Association of Medical Colleges publishes benchmark data on criteria.¹⁰ Medical educators have suggested that CQI approaches should be more holistic. Research on the effectiveness of CQI to effect change in medical education is also needed.^11,12

In pharmacy, there is an upsurge of interest in the use of a CQI approach driven by our accreditation imperative.^13-15 ACPE’s accreditation standards for the doctor of pharmacy degree and its associated guidance documents call for educational programs to exceed minimum requirements through initiatives designed to ensure CQI.^16,17

Although some pharmacy programs have used a CQI approach to improve specific courses and/or curricular segments, historically, pharmacy colleges and schools adopt the more typical program assessment strategy that is done in large scale every four or more years.¹³ Typically, they begin self-studies 12 to 18 months ahead of a scheduled accreditation site visit. Because the work is primarily completed for achievement of accreditation, and because a lot of uncoordinated changes may happen all at once, unanticipated problems can occur and improvements may not be sustained. Moreover, educators may delay proposed changes until the next progress report or accreditation cycle is due.

In 2006, the Auburn University Harrison School of Pharmacy (AUHSOP) adopted an alternative to the intermittent assessment associated with accreditation self-study that incorporated the ACPE accreditation standards into a holistic CQI program. This program includes all aspects of the school’s operations and supports the future preparation of the school’s accreditation self-study. The accreditation standards provided a systematic framework to examine and report data related to most aspects of our program’s mission.¹² All aspects of the program were included: research, graduate and postgraduate education, residency/fellowship programs, and entrepreneurial activities. Borrowing from Baldrige’s themes for organizational excellence, our holistic approach was designed to be aspirational (Table 1).¹⁸

Table 1.

Themes Necessary to Achieve Excellence

This paper describes the process that incorporated all these aspects to yield a holistic CQI approach that drove all areas of the program: strategic planning, committee work, administrative task assignments, and accreditation preparation. The authors’ decided to share this approach following requests from others about the program, perhaps stimulated by the latest accreditation standards.

Design and Implementation of the CQI System

Collins and Apple’s four-step program CQI design process was used to establish the program at AUHSOP. The steps were defining the program, establishing quality criteria for the program, constructing a table of measures or assessment matrix, and assessing the program annually.¹⁹

The first step of the procedure, defining the program, was important because it forced stakeholders to reach a consensus on what the program was and what it was not. It began by revisiting why the program existed (vision), what it does (mission), how it operates (values), and what constrains it (context). Next, the key internal and external stakeholders were listed, and their interests identified. This step was critical in enabling the school to understand and consider perspectives on its performance. This information facilitated the design of reports to these stakeholders. The stakeholders identified were people who had authority over the program such as funders, policymakers, regulators, and advisory boards, people who had direct responsibility for the program such as administrators, faculty and staff, people who were the intended beneficiaries of the program such as students and practitioners, their families, and their communities, and people who might be harmed or disadvantaged by the program.

Clarifying the scope of the program was an important aspect of design. By clarifying what we were and what we were not, what we did and what we did not do, misunderstandings among stakeholders were reduced. Moreover, clarifying the scope reduced the chances of unanticipated problems during program assessment. An example of “scope” is the degree to which research and scholarship are emphasized.

Educators completed the definition of the program when its assets, systems, inputs, input sources, processes, outputs, outcomes, and current and future goals were identified. These were not necessarily assessed, but their identification pointed to potential issues such as bottlenecks or constraints.

The primary goal of the program assessment was to enhance the education quality. To take a theory-based approach, we evaluated the program “as a whole,” and then we questioned how program quality would arise. The first step of this phase was analytical: identifying quality areas or components of the program that we hypothesized had a significant effect on quality. The ACPE Standards were used as a starting point with categories added according to the mission of the school. Next, the assessment team considered each quality area in turn, asking the question: if this quality area (curriculum, for example) was to be viewed as being excellent or of high quality by our stakeholders, what characteristics must it possess? Once a list of quality areas was identified, we then prioritized them. The objective was to identify 3-8 characteristics (criteria) most important to that quality area. As an example, in the quality area “teaching and learning methods,” we prioritized an emphasis on thinking, student accountability, ability-based outcomes, and pedagogically competent faculty.

Writing the performance criterion-statements was the most challenging step of the procedure. Each statement was intended to enable relevant stakeholders to envision a quality performance. For example, we used the following statement to describe the criterion “pedagogically competent:” Faculty consciously and systematically develop their understanding of, and skill in, motivating student pharmacists. They seek new ways to engage students actively and thoughtfully in systematically organized, diverse, and well-designed classroom activities, which result in enduring understanding and an ongoing use of new knowledge and skills. Assessment and appropriate technologies are incorporated into classroom and independent learning experiences.

Once the preliminary criteria were drafted, an assessment was conducted to test their applicability. The assessment used a strengths, improvements, insights (SII) approach. First, “strengths” identified which component of the program’s performance was praiseworthy. A strength statement addressed what attribute or action was valuable, why it was important, and how to reproduce it. Second, “areas for improvement” identified instances of insufficient quality, and hypothesized changes that could be made (between this assessment and the next) to improve performance. This clearly distinguished the issues that caused problems and delineated changes that could be implemented to resolve these difficulties. The process required careful documentation to continually strengthen support for the needed changes. Third, “insights” identified any new and significant understanding gained concerning the performance area, including proposed changes to performance criteria. This assessment allowed the workgroup to identify deficiencies in the criteria list, so that the group could strengthen the criterion statements.

To determine which criteria to assess, the workgroup constructed a table of measures. It was important that we choose this list judiciously in order to use valid and reliable measures. In the process, some criteria were set aside – at least temporarily – ensuring the feasibility of the assessment system as a whole. Criteria selected for formal assessment and evaluation were included in the table of measures and tracked and reported in the annual assessment report. Appendix 1 shows the table of measures. The table of measures included when the assessment would be made, how it would be undertaken, to whom the results would be reported, and how the completed cycle of CQI would be verified. The criteria were defined as a preamble to the table of measures, and a determination was made whether they were to be informally or formally assessed according to the assessment and evaluation plan. Each accreditation standard had a distinct table of measures.

During the final stage, the group assessed the program as planned in the table of measures. At the end of the assessment cycle, the group wrote a report assessed the process and reviewed the outcomes. Based on this review, the assessment program (and associated assumptions) was adjusted. This is a full-circle assessment using measures of quality to document and improve performance.

The preliminary design of the program assessment system was undertaken during a retreat where critical stakeholders were invited – administrators, faculty, students, staff, and practitioners/preceptors. Logistically speaking it was difficult to include all stakeholders, but it was critical to facilitate communication and to include their perspective in the development process. This was done during three stages of the process: when program criteria were identified, when the program underwent the preliminary assessment, and after the assessment system was designed.

The assessment plan was used annually at a faculty retreat to make ongoing program improvements. Progress on previously identified areas for improvement was reviewed and documented and new areas for improvement strengths and insights were noted. The assessment process often stimulated ideas or revealed additional issues tangential to the standard under consideration and to previously identified areas for improvement. Often, these ideas led to other initiatives and outcomes of strategic importance to stakeholders. Finally, a school committee edited the data and faculty finalized and approved a report.

The administration used the final report for several tasks: annual adjustment of the school’s comprehensive strategic plan, formulation of committee charges, and specification of administrative task assignments. Reports from school committees (eg, strategic planning, assessment) were considered at the next yearly CQI retreat. Areas of improvement had to be prioritized to prevent the organization from becoming overwhelmed. Likewise, constituents valued celebrating progress toward improvements.

The CQI data were integrated into the operational system and acted upon each year. The CQI report was saved to the AAMS and access was given to ACPE. These documents served as the basis for preparation of ACPE accreditation site visit documents. ACPE reviewed the documents, and during the last accreditation site visit, the school received a commendation for the CQI process.

DISCUSSION

The CQI process based on the ACPE standards has numerous strengths. For our program, it seemed that time spent in CQI deliberations served to educate members about the program. Discrepancies in participants’ understanding of quality were useful in identifying the critical factors contributing to problems and dysfunction. It was an accountable approach for improving our program since it took a broad range of stakeholders into account. Because we considered the perspectives of diverse stakeholders, we largely avoided “group think.” The collaborative participation helped create a knowledgeable workforce that participated in decision making and program improvements. Results assisted leadership in maintaining a comprehensive strategic plan and developing committees and their assignments. Use of the CQI reports helped in the preparation of accreditation self-study documents. Subjectively, the time required to prepare accreditation documents appears to have been shortened and that most faculty were more comfortable in supporting reports to ACPE site visit teams. The program motivated us to achieve our aspirational goals.

Supplementing quality criteria derived from the literature or from institutional benchmarks added the benefit of additional validity and reliability. However, participating constituents continued to identify most of the criteria collaboratively, which increased their investment in the process. Because faculty members focus in their areas of interest, they sometimes lack a holistic understanding of all areas of the program and the progress being made. Engaging in the CQI process as described has helped with communication and respect for the work of colleagues. Explicitly incorporating CQI results into standard operational procedures also made the CQI efforts relevant (eg, strategic planning, committee development, administrative policy).

One threat to sustainability, once the program was developed, was ensuring faculty fully implemented the assessment system. Explicit and documented assessment tended not to be habitual for most. Full implementation included ensuring all the assessments were undertaken, interpretations were accurate, actions were taken, results were monitored, documentation was completed, and appropriate reports were shared with stakeholders. The individual or body charged with assessment (for example, the assessment committee) had to be vigilant and assertive over the first few cycles of its use. Occasionally, the processes for completing the annual assessment were modified (ie, annual retreat priorities precluded extensive discussion), but the key was to ensure that the program was assessed in some manner.

As the curriculum has continued to change, measures of ability-based outcomes and other knowledge-based measures have been incorporated into the program’s assessment making the process more meaningful. Inclusion of data from the American Association of Colleges of Pharmacy questionnaires administered to our constituents and Entrustable Professional Activities are examples of useful information.²⁰ There will always be more measures than a program is able to manage, and some measures for which an objective mechanism to collect data is not apparent. Over time, some items in the table of measures have been discarded, and others have changed as new priorities have arisen and additional data sources have become available.

As in other disciplines with accreditation requirements, new standards can cause disruption in the CQI process.⁷ However, for the most part, the 2016 standards reflected a reorganization and shift in emphasis rather than wholesale changes in expectations, so we were able to reassign areas for improvement and restructure our CQI report with minimal interference. Major curriculum changes may also be disruptive since curricular structure and coursework may be in flux for extended periods. However, the changes should be indicated by the results of ongoing CQI efforts and the existing quality criteria should continue to be useful in monitoring efforts. Ultimately, given its comprehensive nature, the system must remain agile and relatively simple to implement. One way to keep stakeholders engaged is to ask them to focus on the most problematic areas and move to others as performance and outcomes improve in the previous areas. Resistance from faculty and staff can develop unless a school’s administration effectively uses and supports their work. Preparation of an accreditation self-study has drawn heavily from the CQI reports and has been less time consuming.

CONCLUSION

Preparing for accreditation does not necessarily allow a school to understand itself or to achieve excellence. However, if an assessment system is constructed with a holistic approach to all program operations based on collaboration around shared values and standards for excellence, and if the data are collected, integrated and analyzed collaboratively as well, the system becomes meaningful and useful for subsequent actions. Properly implemented, a well-constructed CQI system can stimulate the development of an assessment culture ensuring a school’s ongoing self-assessment against best education practices. Our process raised the bar for our program, allowing us to develop and sustain high performance expectations. The process also makes our strategic plan more dynamic and contributes aspirational direction. But most importantly, this holistic approach brought assessment into the center of our academic conversations – we feel we achieved an assessment culture.

Appendix 1. Standard 9 – The Goal of the Curriculum 1 (ACPE Standards 2007)