Assessing the Co-Curriculum by Mapping Student Organization Involvement to Curricular Outcomes Using Mixed Methods

Abstract

Objective. To use institution-specific curricular outcomes as a framework to map skill development opportunities available through cocurricular involvement in pharmacy student organizations.

Methods. Participants completed a modified Extracurricular Involvement Inventory individually to measure the intensity of their involvement in each student organization. Participants also completed the Co-Curriculum Outcomes Assessment Mapping Survey (COAMS) instrument as a group to indicate what skills (ie, curricular outcomes) they developed through involvement in a student organization and student organization activities, programs, and events, and to provide examples of these skills. Data sources were triangulated to map skill development opportunities in the co-curriculum to curricular outcomes.

Results. The COAMS identified all curricular outcomes as skills students have the opportunity to develop through student organization involvement in the co-curriculum. Communication was the most common skill identified. Other common skills included professionalism and ethical behavior, collaboration and influence, and in-depth knowledge and proficient skills. A co-curriculum heat map was used to illustrate the degree to which students reported these skills were emphasized through student organization involvement in the co-curriculum.

Conclusion. Evaluation of activities in the context of curricular outcomes can provide a more comprehensive understanding of how the co-curriculum complements the curriculum, thereby complying with accreditation expectations. Cocurricular mapping provides valuable information regarding student skill development opportunities to multiple stakeholders (eg, students, faculty, curriculum leadership). This process can be applied to diverse programs, adapted to measure institution-specific experiences, and measure various constructs of interest.

INTRODUCTION

In an era of evolving healthcare systems and patient care needs, new emphasis has been placed on the importance of health professions students’ personal and professional skill development in addition to knowledge acquisition.¹ Many students use cocurricular activities to develop personal and professional skills (eg, communication, collaboration, problem-solving, professionalism).^1,2 Cocurricular activities, such as service learning, student organizations, leadership development, and community outreach, are activities already embedded within most schools and colleges of pharmacy.^3-5 Recognizing the value these experiences have on the student learning experience, the Accreditation Council for Pharmacy Education (ACPE) Standards 2016 outlines the expectation that schools and colleges of pharmacy implement a co-curriculum, ensure all students (including student leaders and non-leaders) participate in cocurricular activities, evaluate student learning in these activities, and demonstrate how the co-curriculum augments the curriculum.⁶ As pharmacy programs work to comply with this accreditation expectation, a critical gap in the literature exists on processes and tools for programmatic assessment of the co-curriculum to demonstrate how it complements the curriculum.

Recognizing the impact of student involvement within the co-curriculum for both student leaders and non-leaders, recent efforts have begun investigating cocurricular activities for all students.^7,8 Hoffman and colleagues described the implementation of a cocurricular professional development program to promote involvement in professional education, patient care service, legislative advocacy, professional service/leadership, and health care-related community service/philanthropy.⁷ Vos and colleagues illustrated a method for integrating required cocurricular activities and quantifying cocurricular hours.⁸ While these efforts begin to highlight the diversity of cocurricular activities available to students, a gap remains in understanding the skills gained through such experiences, which is a critical step in demonstrating how the co-curriculum complements the curriculum. Patel and colleagues noted that most curricula are inundated with numerous student-led cocurricular activities,⁴ suggesting that faculty and students believe these to be value-added experiences. However, empirical studies supporting such anecdotal reflections and observations are notably absent. Thus, more information is needed to identify learning and skill development opportunities in cocurricular activities for all pharmacy students.

Transitioning from a non-compulsory, voluntary culture of cocurricular activities to a new ACPE expectation requires intentional planning, implementation, and evaluation. As schools and colleges of pharmacy begin planning, implementing, and evaluating their cocurricular programs, more information is needed to identify successful efforts and best practices for structuring the co-curriculum as well as processes and tools for assessment. Most schools and colleges of pharmacy report using a hybrid co-curriculum model, consisting of pre-specified cocurricular activities and a list of cocurricular activities that fit a category.⁹ While schools and colleges of pharmacy reported confidence in their ability to satisfy the ACPE co-curriculum requirement, many schools and colleges reported assessment of the co-curriculum as a challenge and/or area of concern.⁹

Evaluating cocurricular activities in the context of curricular outcomes provides opportunities to identify how the co-curriculum complements the curriculum. By aligning these two experiences, schools and colleges of pharmacy can develop a more comprehensive understanding of the holistic student learning experience.¹⁰ In spring 2016, the UNC Eshelman School of Pharmacy piloted a mixed methods approach to evaluate skills Doctor of Pharmacy (PharmD) students have the opportunity to gain through student organization involvement within the co-curriculum.¹⁰ Embracing Astin’s theory that student involvement requires energy investment in objects of interest, occurs along a continuum, and has both quantitative and qualitative features,¹¹ our approach focused on the evaluation of cocurricular activities (ie, objects of interest) for both the student leader and student member (ie, involvement continuum) and measured the degree of student involvement in the co-curriculum (ie, quantitative and qualitative features) through document reviews, student involvement surveys, and focus groups/interviews.¹⁰

While this approach provided several advantages, a notable limitation included the resource intensive nature of conducting student focus groups and faculty advisor interviews and analyzing the qualitative transcripts for the three pilot student organizations.¹⁰ Additionally, ACPE Standards 2016 state the importance of both curricular and cocurricular experiences in advancing students’ professional skill development in the affective domain-related expectations of Standards 3 and 4 (eg, problem solving, communication, professionalism).⁶ Recognizing the diverse structure and activities of the co-curriculum and curriculum across the academy, more information is needed to establish best practices in cocurricular evaluation and identify processes/tools for illustrating the complementary nature of a program’s curriculum and co-curriculum.⁹ The purpose of this study was to outline a process for using institution-specific curricular outcomes as a framework to map skill development opportunities available through pharmacy student organization involvement in the co-curriculum, and describe a co-curriculum assessment tool to demonstrate how the co-curriculum complements the curriculum, as expected by ACPE, through the mapping of curricular outcomes.

METHODS

In an effort to expand our programmatic assessment efforts of the co-curriculum to include all student organizations at the school, modifications were made to our pilot methodology to reduce limitations and improve efficiencies. A mixed methods approach¹² involving student involvement inventories and a study survey instrument (ie, Co-Curriculum Outcomes Assessment Mapping Survey [COAMS] instrument) was used in our expansion efforts to assess the co-curriculum. This study was considered exempt from further review by the Institutional Review Board at the University of North Carolina at Chapel Hill.

All student organizations were invited to participate in the spring 2017 initiative to identify skills pharmacy students have the opportunity to gain through involvement in student organizations within the co-curriculum at the UNC Eshelman School of Pharmacy. A diverse representation of student leaders and student members, as well as class years, for each student organization was desired to best capture the knowledge and awareness of skill development opportunities among students with varying levels of involvement within the student organization. The spring 2017 initiative originated with a pilot sample of student organizations (n=3) to refine methodology, modify the COAMS instrument as needed, evaluate the utility of study findings, and determine the successfulness of this approach in demonstrating the complementary nature of the co-curriculum to the curriculum.

Members of the three pilot student organizations engaged in a one-hour session with the research team. Prior to the session, leaders from the three student organizations submitted a list of all the signature or hallmark activities, programs, and events of their organization. Additionally, the student leaders were instructed to bring a laptop for the student organization representatives to use as a group to complete the COAMS instrument during the working lunch session. Upon arrival, each student independently completed the modified Extracurricular Involvement Inventory (EII)¹³ and received an overview of the session prior to completing the COAMS instrument.

The EII is a tool that measures a student’s involvement in organized activities and student organizations, both quantitively and qualitatively,¹³ thus embracing Astin’s theory that student involvement occurs along a continuum involving both quantitative and qualitative features.¹¹ A student’s total EII score is the product of the scaled quantity index (ie, students self-reported hours spent involved in organized activities and student organizations) and the sum of the quality index (eg, meeting participation, program, or activity attendance).¹³ This tool has been previously used in pharmacy to describe the types of student organizations students participate in and their level of involvement.^3,10

During the session overview, each student organization group received a list of their previously submitted signature hallmark activities, programs, and events; the PharmD program core competency definitions; and ACPE Standards 2016 Appendix 1 to use, as needed, while completing the COAMS instrument via Qualtrics (Provo, UT). Because the School’s PharmD Program Core Competencies (ie, curricular outcomes) are mapped to ACPE Standards 3 and 4 (Table 1), students were not provided a list of these affective domains. Instead, they were given a copy of ACPE Standards 2016 Appendix 1 in an effort to better understand the content areas reinforced through the co-curriculum. Students were instructed that the focus of the session would be on skill development opportunities available for all members of the student organization. They were also provided an overview of the PharmD Program Core Competencies, ACPE Standards 2016 Appendix 1 content areas, and the COAMS instrument that they would complete together as a student organization group. The COAMS instrument consisted of three sections: Section 1, Activity, Program, or Event: Outcomes and Core Competencies; Section 2, Student Organization: Outcomes and Core Competencies; and Section 3, Student Organization: ACPE Content Areas. The inclusion of co-curriculum outcomes assessment at both the activity level and the student organization level was intentional. Tasking student groups with first reflecting on skill development opportunities at various activities, programs, or events encouraged them to think holistically about the various skill development opportunities available within the student organization. Students were encouraged to use this discussion and reflection to identify the outcomes and core competencies available at the student organization level. Students were also instructed that the research team would be available throughout the lunch session to assist, as needed, with understanding and applying the program outcomes and Appendix 1 content areas as they completed the COAMS instrument.

Table 1.

Core Competencies of a Pharmacy School Program Mapped to Accreditation Standards Established by the Accreditation Council for Pharmacy Education

The COAMS instrument first requires student organization groups to provide the number of hallmark or signature events to be reported. Students were instructed that hallmark or signature events were defined as those annual programs, activities, or events that were available to all members of the student organization that embody the core values and promote the mission and vision of the student organization. Additionally, students were instructed to initially report a maximum of five hallmark or signature events, and if sufficient time remained, they could report additional hallmark or signature events at the end. The instrument then guided students to COAMS Section 1, Activity, Program, or Event: Outcomes and Core Competencies, where it prompted students to identify to what extent they agreed that each activity, program, or event emphasized the PharmD program core competencies using a scale of 1=strongly disagree, 2=somewhat disagree, 3=somewhat agree, and 4=strongly agree. Students were asked to provide specific examples of how a general member gained up to three of the selected skills (ie, core competencies) identified as somewhat agree or strongly agree in a free response section following each reported signature or hallmark event.

After completing section 1, students proceeded to COAMS Section 2, Student Organization: Outcomes and Core Competencies where they were prompted to identify to what extent they agreed the student organization emphasized the PharmD program core competencies using the same scale. Students then proceeded to Section 3, Student Organization: ACPE Content Areas, where they were prompted to report which ACPE Standards 2016, Appendix 1, content areas a typical member had the opportunity to learn or apply through participation in the student organization. Similar to section 1, students were asked in section 3 to provide specific examples of how a general member learned or applied each of the selected content areas in a free response section. After completing the COAMS, the students were given an opportunity to report additional signature or hallmark events held by their organization if time permitted.

The pilot effort provided valuable information that resulted in modifications to the COAMS instrument prior to expanding the study to include all student organizations. The number of signature or hallmark events students were initially asked to report on the COAMS instrument was reduced from five to three, as there was insufficient time for the student groups to report on five events and complete the remainder of the COAMS instrument. Additionally, the four-point response scale ranging from strongly disagree to strongly agree was changed to rarely, sometimes, frequently, and always, as students noted they were hesitant to select responses of somewhat disagree or strongly disagree for the core competencies as they related to skill development opportunities within the student organization. The COAMS instrument stems were modified accordingly to align with the updated scale anchors. The three student organizations who participated in the pilot reviewed their COAMS instrument submissions and revised their responses, as needed, to align with the modified scale. No other modifications were made to the COAMS instrument in preparation for expanding the survey to the remaining student organizations.

Student leaders from the remaining 15 organizations were asked to submit a list of their groups hallmark or signature events. However, the number of events was limited to a maximum of five. This was a modification from the pilot study, in which student organization leaders provided a list of all the organization’s hallmark or signature events in advance of the session. In the pilot, participants often spent significant time reviewing the list and identifying those student organization programs, activities, or events to report on. In the expanded study, if student participants felt there were other programs, activities, or events that better represented their student organization’s hallmark or signature activities, they could report on them during the lunch session. As in the pilot study, student participants in the expanded study completed the modified EII individually and the COAMS instrument as a group.

At the start of the session, each student participant completed a modified EII. The EII inventory scoring was calculated using the formula outlined by Winston and Massaro: (scaled quantity dimension x sum of quality dimensions).¹³ The five items assessing the quality dimension are scored as three points for each “very often” response, 2 points for each “often” response, 1 point for each “occasional,” and 0 points for each “never” and all other responses.¹³ The quantity dimension is measured as 0 points for 0 hours/week, 1 point for 1-8 hours/week, 2 points for 9-16 hours/week, and so on for each 8-hour interval.¹³ Excel was used to analyze all EII scores. The higher a student’s EII score, the higher the student’s level of involvement.¹³ This instrument was originally designed to measure a student’s involvement in all extracurricular activities; however, for the purposes of this study, the EII instrument was modified to measure a student’s involvement only in the student organization of interest to align with previous work involving this inventory.¹⁰

During the lunch session, each group of students representing an organization was tasked with providing specific examples of how a general member (not an officer) was able to gain those skills (ie, PharmD program core competencies) reported as frequently or always emphasized within the student organization’s program, activity, or event reported in section 1 of the COAMS. Two research investigators independently coded all qualitative student responses to triangulate student identified skills that were frequently or always emphasized. The PharmD program core competency definitions provided to student organization groups during the working lunch session (Table 1) served as the codebook. Upon completion of the independent review, the primary investigator collated the two independent coding efforts and identified areas of disagreement. The two research investigators met to discuss the areas of disagreement and arrive at consensus for final coding of all qualitative responses in section 1.

A research auditor reviewed a random sample of three student organizations and agreed with 83.3% (20 of 24) of all sample codes. Agreement discrepancy was noted for four codes, which the two research investigators reviewed and reached consensus. Three of these codes retained the original coding established during the consensus meeting of the two research investigators. One code, Critical Thinking and Problem Solving, was changed to match the auditor’s coding. All Critical Thinking and Problem Solving codes were then reviewed by the primary investigator. One code was identified for additional review by the two research investigators, who agreed to code the statement as Critical Thinking and Problem Solving, which was a change from the original consensus. No other Critical Thinking and Problem Solving codes were identified as warranting review or change. All coding was completed using Microsoft Word.

RESULTS

Findings indicated diverse representation of leadership status (ie, student leader vs. student member) among student participants representing each student organization in the co-curriculum. All 18 student organizations were represented by at least one participant (student leader or student member) in the study, and 13 (72.2%) student organizations were represented by both student leaders and student members. Eleven (61.1%) student organizations were represented by students from at least two program years. Mean (SD) hours that participants reported being involved in the student organization ranged from 1.0 (0.0) to 12.6 (17.5) hours per week. The median modified EII score for each student organization ranged from 5 to 28, indicating diverse levels of student involvement (Appendix A).

Results from COAMS Section 1, Activity, Program, or Event: Outcomes and Core Competencies, and Section 2, Student Organization: Outcomes and Core Competencies, were compared. All skills identified by students as “frequently” or “always” emphasized at student organization activities, programs, and events in COAMS Section 1 were triangulated with COAMS Section 1 researcher coding of supporting qualitative comments. All these skills were also identified as being “always” emphasized at the student organization level in COAMS section 2. Remaining skills identified by students as “frequently” or “always” emphasized in section 1 that triangulated with qualitative coding by the research team were identified in multiple hallmark events submitted by the student organization. These findings were used to map skill development opportunities (ie, core competencies) to each student organization (Figure 1).

Figure 1.

Skill Development Opportunities Mapped to UNC Eshelman School of Pharmacy Student Organizations

All of the school’s nine core competencies were identified as skill development opportunities available through student organization involvement within the co-curriculum (Figure 1). Communication was the most common skill, identified in 94.4% (n=17) of student organizations. Other common skills included professionalism and ethical behavior, collaboration and influence, and in-depth knowledge and proficient skills, which were identified in 83.3% (n=15), 77.8% (n=14), and 72.2% (n=13) of student organizations respectively. Accessing and analyzing information was the least common skill, identified in 27.8% (n=5) of student organizations. Adaptability, initiative, and curiosity, and inquisitiveness were each identified in 38.9% (n=7) of student organizations, while critical thinking and problem solving was identified in 44.4% (n=8) of student organizations (Figure 1).

Findings from COAMS Section 2, Student Organization: Outcomes and Core Competencies, were used to develop a co-curriculum heat map of the skill development opportunities available through student organization involvement (Figure 2). This heat map illustrates the degree to which students reported these skills (ie, curricular outcomes) were emphasized through student organization involvement within the co-curriculum. Communication was reported as always emphasized by 83.3% (n=15) of student organizations; professionalism and ethical behavior were reported as always emphasized by 66.7% (n=12) of student organizations; and accessing and analyzing information was reported as sometimes or rarely emphasized by 61.1% (n=11) of student organizations.

Figure 2.

Heat Map⁺ of Skills Emphasized Through Student Organization Involvement in the Co-Curriculum of the UNC Eshelman School of Pharmacy

⁺The heat map above is coded as Black Solid (Always – 4), Dark Grey Diagonal (Frequently – 3), Light Grey Dots (Sometimes – 2), and White Solid (Rarely – 1). Black solid indicates those skills students identified as always emphasized within the student organization, while white solid indicates those skills students identified as rarely emphasized within the student organization.

DISCUSSION

While previous work investigated skills gained through student leadership,^14-18 more information is needed regarding skill development opportunities available for all students through cocurricular involvement. Further, a literature gap remains in terms of processes and tools for programmatic assessment of the co-curriculum that can demonstrate how the co-curriculum complements the curriculum. This study is one of the first to describe a process and tools for assessing the co-curriculum in the context of curriculum outcomes. Further, this study is also one of the first to evaluate the skill development opportunities available for all pharmacy students through student organization involvement in the co-curriculum. The described co-curriculum assessment process (Table 2), modified EII tool, and COAMS instrument can easily be translated to other schools and colleges of pharmacy to assess their unique co-curriculum in the context of their unique curricular outcomes, and demonstrate how their co-curriculum complements their curriculum to comply with the co-curriculum expectations as outlined in ACPE Standards 2016.

Table 2.

Twelve Best Practices for Implementing a Process for Assessing and Mapping the Co-Curriculum in a School of Pharmacy

Building on our previous efforts,¹⁰ this study outlines a more efficient process for evaluating student organization involvement in the co-curriculum in the context of curricular outcomes. More specifically, this mixed methods approach outlines a less resource-intensive process through the use of a survey instrument rather than focus groups and interviews. In our previous pilot effort, three 1-hour focus groups/interviews were held for each student organization.¹⁰ Recognizing the significant time required to conduct three hours of focus groups and interviews for each student organizations and the resources required to analyze the resulting qualitative transcripts, a more efficient process was needed as we expanded our study from three organizations to 18. Using a mixed methods survey instrument improved efficiencies as it required only one hour of data collection and reduced the time required for data analysis. Additionally, multiple student organizations could participate in the same working session, which reduced the contact hours required.

This study approach also enhanced the quality of data collected through the use of student members and student leaders rather than faculty advisors; identification of student organization hallmark programs, activities, and events in advance of the study session; and qualitative coding of open-ended comments by the research team to identify supporting statements of student perceptions. By evaluating cocurricular activities in the context of curricular outcomes, a more comprehensive understanding of how the co-curriculum complements and augments the curriculum can be understood. Our findings suggest that this co-curriculum assessment process was effective at illustrating how the co-curriculum complements the curriculum, as evidenced by the mapping of student organizations to curricular outcomes (ie, core competencies). The process outlined here can easily be applied to co-curricula at other schools of pharmacy to assess their unique co-curriculum and demonstrate how it complements their program’s curriculum and curricular outcomes of interest (eg, accreditation standards, program outcomes) (Table 2).

Our mixed methods approach also provided opportunities for data triangulation.¹² By reviewing students’ specific examples of how they developed skills through involvement in their student organization’s programs, activities, and events, the research team was able to reduce the biases and limitations inherent in student self-report. Qualitative coding by the research team allowed for data source triangulation and thus reduced the biases of student self-reported benefits. Additionally, it allowed for identification of skills not selected by the student group as always or frequently emphasized within the student organization. For example, student organizations I and J both indicated that critical thinking and problem solving were sometimes emphasized for all students within the student organization (Figure 2). However, researcher review and coding of the qualitative comments identified critical thinking and problem-solving as consistently emphasized for all students through multiple hallmark events described. Thus, critical thinking and problem solving skills were mapped to student organizations I and J (Figure 1). Notably, this qualitative review focused specifically on whether or not the skill development opportunity existed, not the extent to which the skill was emphasized within the student organization. This objective evaluation allowed the research team to validate student responses and summarize these findings in the form of a co-curriculum map (Figure 1), as demonstrated in this example with critical thinking and problem solving for student organizations I and J.

A comprehensive picture of skill development opportunities that exist in both the curriculum and co-curriculum can provide valuable information for multiple stakeholders.¹⁰ The co-curriculum assessment process described facilitated development of a co-curriculum map that provided a visual summary of the skill development opportunities available through student organization involvement in the co-curriculum (Figure 1). By evaluating these skill development opportunities in the context of curricular outcomes, a more comprehensive understanding of how the co-curriculum complements the curriculum can be understood. Expanding this co-curriculum map to include curriculum components (ie, course, course activities, assessments) could provide a holistic understanding of the full student learning experience across both the curriculum and co-curriculum.

While a co-curriculum map can identify if and where curricular outcomes are addressed in the co-curriculum, production of a co-curriculum heat map can quickly summarize what skills are emphasized and to what degree (Figure 2). This more detailed information may be valuable for students, faculty members, student affairs professionals, and curriculum and assessment administrators. The variety and volume of cocurricular activities available can at times be overwhelming for students, specifically first-year students new to the pharmacy program.¹⁹ Encouraging students to be reflective, a co-curriculum heat map can assist them with identifying opportunities in which they can further develop skills of interest, encouraging them to take ownership of their learning experience and exercise self-awareness skills. In the spirit of promoting collaboration rather than competition, consideration should be given to disseminating a modified co-curriculum heat map that summarizes the top three to five skills emphasized within each student organization in the co-curriculum. The heat map can also equip faculty members and student affairs professionals with information to guide students who need additional exposure and/or practice in specific skill development areas to cocurricular activities that emphasize these skills. For curriculum and assessment leadership, a co-curriculum heat map can provide a more comprehensive picture of cocurricular activities (eg, service learning, student organizations, leadership development, and community outreach) in which students can develop and practice key skills of interest.

Use of a mixed methods approach provided opportunities to gather insight into students’ cocurricular involvement as well as programmatic assessment of the co-curriculum. Modified EII data findings show that the intensity with which students are involved in student organizations within the co-curriculum varies. This is expected given the varying foci, initiatives, and priorities of the different student organizations. Capturing this information can provide valuable information to students and faculty members as they identify optimal cocurricular activities to enhance student learning and skill development. For example, students who have other commitments and factors to consider (eg, family, part-time or full-time work, lengthy commutes) may have limited time available to commit to cocurricular activities.² The EII data can be utilized in conjunction with co-curriculum maps and/or heat maps to identify cocurricular activities that do not typically require extensive time but could still meet a student’s needs. For example, participants in student organization D reported being involved with the organization for a mean (SD) of 1.0(0.0) hour/week compared to participants in student organization R who reported being involved in their organization for a mean of 3.0(1.7) hours/week. The co-curriculum map (Figure 1) associates five skills with participation in student organization D compared to only two skills associated with participation in student organization R. The co-curriculum heat map (Figure 2) identifies six skills as always or frequently emphasized in organization D compared to only three skills emphasized in organization R. Thus, a student who is interested in both student organizations but has limited time to devote to them might prefer to engage with student organization D. Supplementing EII information with the co-curriculum map and/or heat map can aid students and faculty members in identifying valuable cocurricular opportunities that accommodate students’ available time and meets their skill development needs and interests.

Although this study describes a process and introduces a tool for mapping skills developed through cocurricular involvement in pharmacy student organizations in the context of curricular outcomes, its implementation is limited to a single institution. While the co-curriculum assessment data findings are specific to our institution, the process and tools described can be easily translated to the programmatic assessment efforts of other schools’ and colleges’ of pharmacy co-curricula to demonstrate how their unique co-curriculum complements the curriculum (Table 2). Implementation challenges encountered that should be considered in future efforts included multiple scheduling conflicts with various student organization events and academic calendars (eg, examinations, differing class schedules) resulted in recruitment challenges. Participation by a minimum of one student leader and two general members was requested of each student organization; however, all student organization members who wished to participate were permitted to do so. While the majority of student organizations were represented by both student leaders and general members, future efforts should continue to strive for representation from both student leaders and general members to best represent the student organization and the skill development opportunities available. While participant self-selection is a noted limitation to study methodologies that solicit volunteers, the modified EII scores found in this study ranged for most organizations. This finding suggests that study participants had varying levels of involvement within the student organization and thus did not only represent students highly involved in the organization.

Using the formula provided by Winston and Massaro in the original EII, a student’s total score is calculated by the product of (scaled quantity dimension) x (sum of quality dimensions.¹³ Because the EII was modified to only measure a student’s involvement in the organization of interest, the majority of participants (n=49, 92.5%) scored 1 point (ie, 1-8 hours/week) on the quantity dimension, thus reducing the instrument’s ability to differentiate students’ quantitative involvement in student organizations. While this was a noted limitation in our previous efforts,¹⁰ the original calculation was retained for consistency among cocurricular assessment efforts. While limitations exist in self-reporting one’s time use with or without time-logging tools,²⁰ an opportunity exists to consider using the EII to evaluate the intensity of pharmacy student involvement in the co-curriculum as a whole (ie, across all student organizations and cocurricular activities).

While student participation in the one-hour lunch session improved efficiencies and reduced resources compared to the 2016 focus groups and interviews,¹⁰ the brief time limited students’ ability to effectively complete the COAMS survey instrument. Specifically, most student groups had to rush through Section 3, Student Organization: ACPE Content Areas, and several groups were unable to provide qualitative responses to support and explain their content area selections. Consequently, ACPE content area data were excluded from analysis because of the reduced data reliability resulting from the low response rate. Future efforts may consider extending the lunch session to allow additional time for students to complete the study instrument.

The process and tools described in this paper can facilitate pharmacy schools’ programmatic assessment of the co-curriculum, evaluate alignment between the curriculum and co-curriculum, and demonstrate how the co-curriculum complements the curriculum. Recognizing that curricula and co-curricula are unique to each school, this study outlines a process that pharmacy programs can translate and apply to their own institution for assessment and mapping of the co-curriculum (Table 2). More specifically, we suggest determining the curricular outcome(s) of interest that would best position the institution to demonstrate how the co-curriculum complements and augments the curriculum. These outcomes may include program outcomes, accreditation standards, or other constructs of interest. Identifying cocurricular activities (eg, service learning, student organizations, community outreach) and selecting co-curriculum assessment tool(s) or instrument(s) are important next steps.

In our efforts, a mixed methods approach using the modified EII tool and the COAMS instrument was effective at assessing skills gained through student involvement in the co-curriculum in the context of curricular outcomes. It is important for schools to determine available resources to facilitate programmatic assessment of the co-curriculum, including time and personnel. Transitioning from our focus group pilot approach¹⁰ to this mixed methods approach reduced our resource needs for data collection and analysis and improved efficiencies. When evaluating available resources, it is also important to establish an implementation plan, including needed faculty and staff support, student participation, timelines, and tools/instruments. If engaging students, we suggest recruiting appropriate student representation (eg, student leaders, student members, students from various class years) for participation in a 60- to 90-minute working session, giving consideration to the ability of students to determine if they have acquired or had the opportunity to acquire skills within a particular organization (and, if not, what other types of data or participants might be needed to help with the assessment). Acquiring a list of cocurricular activities prior to the session can improve efficiencies to ensure sufficient time is available to: provide an overview of the session; educate participants on the assessment tool(s), curricular outcome(s) of interest, and focus of the session; and provide adequate time for students to complete the instrument(s). Available resources should be considered when determining data collection and analysis methods (eg, quantitative analysis, qualitative coding, data triangulation) and establishing a dissemination plan. Finally, we suggest summarizing findings in a desired output (eg, co-curriculum map, co-curriculum heat map) for each identified stakeholder (eg, students, faculty, curriculum and assessment leadership) and determining the frequency in which co-curriculum assessment is needed to inform data-driven, continuous quality improvement (Table 2 ). While UNC completed this co-curriculum assessment for all student organizations in the co-curriculum, we are currently determining our sustainability plan, including frequency in which each organization in the co-curriculum will be re-evaluated (eg, every 3 years).

CONCLUSION

This study is one of the first to describe a process for identifying and mapping skills students have the opportunity to gain through student organization involvement in the co-curriculum. This process facilitates programmatic assessment of the co-curriculum and is effective at demonstrating how the co-curriculum complements the co-curriculum by mapping to curricular outcomes. Recognizing that schools and colleges of pharmacy across the country have varying curricular and cocurricular experiences unique to their own institution, identification of successful processes for assessing the co-curriculum are needed to better understand this student learning experience.⁹ The process outlined here (Table 2) can be applied to a variety of programs and adapted to measure institution specific experiences as well as various constructs of interest (eg, accreditation standards, program outcomes). Cocurricular mapping, including co-curriculum heat maps, can provide valuable information on student skill development opportunities to multiple stakeholders.

Appendix A.

Doctor of Pharmacy Students’ Responses on a Modified Extracurricular Involvement Inventory Regarding Student Organizations