A Modified Delphi Process to Define Competencies for Assessment Leads Supporting a Doctor of Pharmacy Program

Abstract

Objective. To define the competencies for individuals designated as assessment leads in colleges and schools of pharmacy.

Methods. Twenty-three assessment experts in pharmacy participated in a modified Delphi process to describe competencies for an assessment lead, defined as the individual responsible for curricular assessment and assessment-related to doctor of pharmacy program accreditation. Round 1 asked open-ended questions about knowledge, skills, and attitudes. Round 2 grouped responses for comment and rating for consensus, which was prospectively set at 80%.

Results. Twelve competencies were defined and grouped into 3 areas: Context for Assessment, Managing the Process of Assessment, and Leadership of Assessment Activities. In order to verify the panel’s work, assessment competencies from other disciplines were reviewed and compared.

Conclusions. The competencies describe roles for assessment professionals as experts, managers, and leaders of assessment processes. They can be used by assessment professionals in self-assessing areas for professional development and by administrators in selecting, developing, and supporting designated leads.

INTRODUCTION

The role of assessment in pharmacy professional education is growing. This growth is driven not only by accreditation requirements, but also the desire to provide competitive educational programs that produce competent practitioners. Assessment can help evaluate the best teaching strategies to reach today’s learners and provides a data-driven approach to determining optimal use of scarce resources. When used to its fullest capacity, and when coupled with a strong culture and good knowledge of organizational change, assessment can transform curricula.¹

Recognition of the importance of assessment has produced an increase in educators interested in and dedicated to the assessment of student learning and programmatic outcomes. Professionals tasked with assessment roles are seeking guidance and opportunities for networking and learning as evidenced by the growth of the American Association of Colleges of Pharmacy (AACP) Special Interest Group (SIG) for Assessment established in 2010. Membership in this SIG has increased from 219 pharmacy educators when it originally formed, to 767 members in 2015, a nearly fourfold increase in only five years.² In addition, the academy is responding with scholarly assistance to help assessment professionals establish grounding in the art and science of assessment. In particular, three recent publications, a formative assessment toolkit,³ a paper on integrating the new Center for the Advancement of Pharmacy Education (CAPE) outcomes into pharmacy curricula,⁴ and a paper on assessing the CAPE outcomes⁵ have all provided guidance for accomplishing assessment work in pharmacy colleges and schools.

While fostering broad engagement and collaboration in assessment is encouraged,⁶ and providing tools and guidance is a necessity, the roles and responsibilities of the assessment person are critical considerations. Despite growing experience with assessment, colleges/schools face significant challenges, including advancing assessment of the affective domain, integrating assessment with continuing quality improvement practices, and building a culture of assessment. In addition, the increased emphasis on assessment in the Accreditation Council for Pharmacy Education’s Standards 2016⁷ must also be considered. The current expectations for and complexity of assessment necessitates the identification of an assessment lead in a doctor of pharmacy (PharmD) program. Yet, as the technical and leadership needs for assessment have continued to mature, it may be challenging to determine the expertise needed for new assessment leads and the professional development priorities for existing assessment professionals.

The aim of this work was to draw together a group of experienced pharmacy assessment professionals to identify and obtain consensus on assessment-related competencies for individuals designated as the assessment lead in a college or school. These competencies will be useful to assessment professionals in self-assessing areas for professional development and to administrators in selecting, developing and supporting designated leads.

METHODS

A two-round modified Delphi process was used to obtain opinions from assessment experts on the competencies needed in an assessment lead in a college or school of pharmacy. The Delphi process uses multiple rounds of structured, anonymous data collection to draw together the collective wisdom of a group of experts.⁸ In the initial round, expert panelists generally respond to open-response questions.⁹ This input is then collated and presented back to the group for review and refinement.^10,11 Statements defined from early rounds can be rated by the panelists to determine consensus in later rounds.^9,12

Several characteristics of the Delphi process encourage quality participation and input. In contrast to discussion, the Delphi process allows each panelist an equal opportunity to present ideas.¹³ With its anonymity, it encourages expression that is free from group pressure.¹⁴ With the feeding back of results through successive rounds, it also allows for opinions to be heard in a nonadversarial manner.⁸ Views can be retracted, altered, or added after seeing others responses and giving the issue further consideration.¹⁴ In addition, the Delphi process allows for obtaining opinion without physically bringing experts together.⁸

The Delphi process has been used to define competencies in medical education^15,16 and pharmacy education.^17,18 In addition, it has been used to define competencies for student affairs professionals,¹⁹ and teachers.^20,21

The Delphi process relies on the identification and use of experts within the discipline.⁹ However, the optimal number of Delphi panelists is not agreed upon in the literature ^12,14,22 and studies show a wide variation in panel size.⁹ In determining panel size, the availability of time and money has been acknowledged as important and influential in decision-making.⁹ Hasson, Keeney and McKenna discuss the ramifications of panel size and caution that larger sizes result in greater generation of data, which influences the amount of analysis and the potential for analysis difficulties.⁸ Ten to 15 subjects has been suggested as sufficient when the subjects are homogenous.¹⁰ Recent Delphi processes in pharmacy education have sought minimum panel sizes of 20.^23,24 Considering the pool of available experts and its breadth in terms of experiences, a panel size of 20-30 was deemed appropriate.

To identify experts who could serve as participants, an invitation email with direction to an online form was circulated to members of the AACP Assessment SIG. Participants were asked to indicate their assessment-related experiences and panelists were selected from the respondent group. Criteria for selection included: college or university assessment committee membership; a publication or award in assessment; and provided university-level assessment consultation or chaired or led a self-study of a college or school’s PharmD program, or served on an ACPE site visit team for a college or school’s self-study, providing assessment expertise, or served as an officer for an assessment-related special interest group within a professional organization. Points were awarded based on the total number of activities reported. Those with the most points in these categories were selected to participate. An email was delivered inviting their participation in the Delphi process and providing information on the study.

In round 1, open-ended questions were used to gather input on the desired competencies for curricular assessment and assessment related to PharmD program accreditation. Panelists were asked to comment on the necessary knowledge and skills, as well as the necessary attitudes, values, beliefs, and behaviors. Recognizing that an individual leading assessment may hold various titles within a college or school of pharmacy (eg, assistant or associate dean, director, coordinator, or assessment committee chair), panelists were asked to focus on the competencies needed for the point person or lead, who may or may not execute each assessment initiative on their own.

Panelist comments were reviewed by two members of the research team and performance-based competency statements were drafted. A report was generated with the competency statements and the direct quotes from panelists supporting each statement. Because of the structure of the questions, the panelists’ comments tended toward more knowledge-based language; however, the performance-based competencies were fully vetted by the participants in the subsequent rounds.

The round 1 report was returned to participants. Panelists were asked to indicate their level of agreement with each competency statement using a 5-point Likert rating system (ie, strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). A consensus level was set prospectively as there is no agreed upon consensus level in the literature.^9,22 Keeney et al argues that the importance of the topic can guide consensus level, suggesting that 100% consensus may be desirable for life or death issues, while 51% may be appropriate for preferences.²² For this study, the consensus level was set at 80% of panelists agreeing or strongly agreeing with a competency statement. In addition to rating the competency statements, panelists were specifically asked to comment on areas where they disagreed with a competency statement.

The competencies from round 2 were sent to participants with the instructions: “If desired, please feel free to comment on the competency or to suggest additions or modifications to the descriptors/additional detail.” Responses provided additional context and background on the panelists’ perspectives and were used by the investigators to expand competency descriptors. In preparing the data for presentation, the descriptors were modified for readability from the direct quotes from panelists in order to give some direction in understanding the competency. However, they were not vetted or voted upon by the panelists and should not be interpreted as subcompetencies.

For all rounds, panelist responses were collected via the Web-based survey software program Qualtrics (Qualtrics Labs Inc., Provo, UT). This study was determined exempt by the University of Minnesota’s institutional review board.

As a means of verifying the panel’s work, assessment competencies from other disciplines were sought. Two sets of assessment-specific competencies were identified and reviewed: the Essential Competencies for Program Evaluators (ECPE)²⁵ and the Assessment Skills and Knowledge for Student Affairs Practitioners and Scholars (ASK).²⁶ These standards were compared to the competencies defined by the Delphi panel.

RESULTS

The initial request sent to the AACP Assessment SIG identified 115 potential experts based on their self-reported assessment-related activities and expertise. Thirty-one of these individuals were invited to participate in the study, 23 agreed to become study panelists, and 21 panelists completed round 2. Eighteen panelists participated in the optional comment period.

Twelve competency statements were defined. To facilitate communication and discussion, competencies were grouped by the authors into 3 areas: Context for Assessment (Table 1), Managing the Process of Assessment (Table 2), and Leadership of Assessment Activities (Table 3).

Table 1.

Defined Assessment Competencies for Area 1: Context of Assessment

Table 2.

Defined Assessment Competencies for Area 2: Managing the Process of Assessment

Table 3.

Defined Assessment Competencies for Area 3: Leadership of Assessment Activities

Competencies from related fields acknowledge the importance of the Context of Assessment (area 1). The ASK standards highlight benchmarking and the politics of assessment,²⁶ while the ECPE Standards require situational analysis to become familiar with a program, its stakeholders, and its position within the larger organization.²⁵ The competencies from the pharmacy Delphi panelists expand on these topics and categorize context-related competencies into the broader areas of higher education; the university, school, or college; accreditation; and the field of pharmacy.

Competencies from related fields also focus heavily on Managing the Process of Assessment (area 2). The competencies for program evaluators (ECPE) includes categories for systematic inquiry, such as “design, data collection, analysis, interpretation, and reporting” and project management or “the nuts and bolts of conducting an evaluation, such as budgeting, coordinating resources, and supervising procedures.”²⁵ Similarly, the competencies for student affairs practitioners (ASK) contain standards for assessment design, articulating learning and development outcomes, selection of data collection and management methods, assessment instruments, surveys used for assessment purposes, interviews and focus groups used for assessment purposes, assessment methods, program review and evaluation, and effective reporting and use of results.²⁶ Pharmacy Delphi panelists emphasized elements of managing the process of assessment, such as managing the resources for assessment, guiding the process, and selecting the tools. Additionally, the pharmacy panelists specifically called out the role of assessment professionals in contributing to curricular improvement and participating in scholarly activities.

The competencies from pharmacy Delphi panelists differ from competencies in related fields in two ways: emphasis on leadership and definition of professional principles. The pharmacy Delphi panelists placed greater emphasis on Leadership of Assessment Activities (area 3). Competencies from related fields included some leadership-related elements, such as interpersonal competence (ie, “people skills, communication, negotiation, conflict, collaboration, and cross-cultural skills”)²⁵ and educating others about assessment methods and outcomes.²⁶

Competencies from related fields include a standard for assessment ethics²⁶ and a category for professional practice.²⁵ These professional principles competencies address institutional review boards, the Family Educational Rights and Privacy Act (FERPA), confidentiality, integrity, and honesty.^25,26 The absence of a professional principles competency is a potential oversight in the work of the pharmacy Delphi panelists.

In addition, the global competencies developed in this study lack the detail found in other related competencies work. For example, competency 7: “identifies, selects, develops and utilizes assessment tools,” translates into three separate ASK competencies (ie, selection of data collection and management methods, assessment instruments, and surveys used for assessment purposes). Furthermore, each ASK competency has up to 7 ability statements.²⁶

DISCUSSION

In this study, 21 pharmacy assessment experts completed a two-round Delphi process resulting in the definition of 12 competency statements for an assessment lead. The competencies focus on the knowledge, skills, and attitudes needed to lead assessment initiatives for a college or school, including curricular assessment and assessment related to PharmD program accreditation. The competencies were examined in comparison to other competencies that have been defined for assessment professionals in other areas of higher education.

In Context for Assessment (area 1), “monitors higher education assessment-related trends” (competency 1), “works within the university and college/school to accomplish assessment activities” (competency 2), and “interprets accreditation-related standards and process” (competency 3) are competencies that would be anticipated as needed for an assessment lead. These competencies likely appear in job descriptions and guide development efforts. However, “values the practice of pharmacy and pharmacy education” (competency 4) may require more intentional examination and action. In open commenting, Delphi participants shared that many assessment leads may not have a pharmacy background. As a result, they may not come to their positions with an appreciation of the current practice challenges graduates are being educated to address or the unique needs of pharmacy (or health professions) education. As nonpharmacy individuals enter colleges and schools as assessment leads, explicit consideration should be given to the pharmacy-related knowledge and experiences that will help them accomplish their work. Similarly, pharmacy-trained individuals who transition into assessment roles may need to gain assessment-specific knowledge and experience to accomplish their work.

Managing the Process of Assessment (area 2) addresses the management competencies necessary to run assessment activities efficiently and effectively, including planning, budgeting, organizing, and staffing. In particular, competency 5 addresses the management of resources (human and fiscal) that are associated with assessment. While some assessment leads have direct responsibilities for hiring of personnel and management of budgets, others have only indirect authority. Being a good steward of resources is critical for a lead whether or not they possess formal authority. For example, the assessment lead may serve an important role in recruiting faculty members to assessment committees or recommending the purchase of tools (eg, software, simulators).

Leadership of Assessment Activities (area 3) is a recognition of the assessment lead’s role in “the process of influencing an organized group toward accomplishing their goals.”²⁷ Hoey and Bailey investigated the changing roles in institutional assessment to determine if the scope of skills necessary is broadening from a focus on technical expertise to include aspects of leadership and strategic planning.²⁸ Leadership has been described as setting direction, motivating action, and aligning people.²⁹ Strong leaders have been shown to use five practices: Model the Way, Inspire a Shared Vision, Challenge the Process, Enable Others to Act, and Encourage the Heart.³⁰ In other words, strong leaders have the ability to: create standards of excellence, set an example for others to follow, look for innovative ways to improve the organization, envision the future of what the organization can become, experiment and take risks, foster collaboration, build spirited teams, recognize contributions, and celebrate accomplishments.³⁰ These practices and behaviors, which can be developed, are important to the work of the assessment lead.

The three areas of competency suggest roles for assessment professionals as experts, managers, and leaders. Hiring or internally recruiting an assessment lead that has all of the expertise needed may not be possible. To that end, encouraging assessment professionals to determine their needs relative to these competencies, identifying areas for knowledge or skill expansion, is important. Similar to pharmacists engaging in continuing professional development,^31,32 assessment leads are required to take responsibility for their own professional growth and colleges and schools must support them in this process.

While having a strong assessment lead is important, colleges and schools must also deliberately work to identify collaborations, develop teams, and build a culture of assessment in order to advance assessment efforts. The role of the assessment lead is not to “do assessment” for the organization, but to recruit faculty members and help to ensure that team members are all heading in the same direction. It is essential for the lead to involve and encourage others, while also providing expertise in assessment theories, principles, methods, analytical techniques, and reporting strategies.

Collaborations are available in many locations. Working within the university to accomplish assessment activities (competency 2) will enhance connections to local experts. The importance of broad participation in assessment across an institution has been chronicled by three papers examining contemporary issues in assessment in American higher education: Schuh examined the role of student affairs,³³ Volkwein described the role of institutional research,³⁴ and Gilchrist and Oakleaf detailed the role of librarians.³⁵ These papers emphasize that successful assessment programs involve collaboration across the institution and are not simply the role and responsibility of one individual. In leading the assessment efforts (competency 11), the lead needs to engage faculty members, staff, students and stakeholders in the work of assessment, and to motivate meaningful contributions. Recognizing the need for and importance of these skills, Duncan-Hewitt and colleagues³⁶ listed credibility and collegiality as criteria for success in establishing an office of teaching, learning, and assessment. To aid in engaging others, Hutchings provides recommendations for strengthening the connection between faculty members and the enterprise of assessment.³⁷

Assessment is an asset in transforming curricula.¹ Each assessment initiative can benefit from a team of individuals that is helping to define the focus, debate options, analyze data, and make recommendations. In serving as a resource to others on assessment-related theories and principles (competency 12), the lead must facilitate assessment at a variety of levels (eg, student learning, programmatic). In order to be successful, a culture of assessment is needed. One of the keys to a culture of assessment is leaders who are actively committed to supporting the efforts of assessment.³⁸ In addition, this culture can be accomplished by recognizing and supporting faculty members who are champions of assessment, allocating resources to support assessment efforts, supporting faculty innovation and inquiry, and providing release time for the work to be conducted. Additional methods for creating a culture of assessment have recently been summarized.⁶

This study focused on responses from assessment experts in pharmacy. Responses may have been influenced by the panelists’ current job descriptions or by the panelists’ concerns about feasibility given resource constraints. Research that includes perspectives from pharmacy faculty members, administrators, and/or assessment experts from other health professions could yield new insights related to these competency areas. Given that assessment requires the active engagement of many, these competencies could be built upon to describe the desired competencies of assessment committee chairs, assessment committee members, and others tasked with assessment-related responsibilities.

When compared against assessment-related competencies from outside the profession, several opportunities for enhancement were noted. The competencies derived from this research were not as detailed as those from other groups. Further detail may be needed to better support assessment professionals in pharmacy. In addition, defining levels of competency (ie, basic, intermediate advanced) may make the competencies more useful in the professional development of assessment leads.

Several areas within the competencies may require further discussion, debate, and refinement. For instance, competencies related to the professional practice of assessment (eg, ethics, confidentiality) might be further detailed. In addition, competencies related to influence could be elaborated, such as competencies needed for: the scholarship of assessment, advancing assessment practices within the academy or supporting broad scale curricular change. Information on pharmacy assessment professionals’ abilities relative to these competencies could be useful in framing programming and guiding the development of support tools.

CONCLUSIONS

With multi-faceted roles as experts, managers, and leaders, assessment leads must continually work to advance their skills. The competencies defined in this research could be used by assessment leads in self-assessing areas for professional development. While not intended as a job description, an awareness of these competencies can help guide administrators as they identify individuals to serve as assessment leads. Future research is needed to elaborate on the competencies and determine assessment leads’ abilities relative to the competencies.