A Systematic Review of Assessment Tools Measuring Interprofessional Education Outcomes Relevant to Pharmacy Education

Abstract

Objective. To identify and describe the available quantitative tools that assess interprofessional education (IPE) relevant to pharmacy education.

Methods. A systematic approach was used to identify quantitative IPE assessment tools relevant to pharmacy education. The search strategy included the National Center for Interprofessional Practice and Education Resource Exchange (Nexus) website, a systematic search of the literature, and a manual search of journals deemed likely to include relevant tools.

Results. The search identified a total of 44 tools from the Nexus website, 158 abstracts from the systematic literature search, and 570 abstracts from the manual search. A total of 36 assessment tools met the criteria to be included in the summary, and their application to IPE relevant to pharmacy education was discussed.

Conclusion. Each of the tools has advantages and disadvantages. No single comprehensive tool exists to fulfill assessment needs. However, numerous tools are available that can be mapped to IPE-related accreditation standards for pharmacy education.

INTRODUCTION

The World Health Organization recognizes interprofessional education (IPE) when students from two or more professions learn about, from, and with each other to enable effective collaboration and improve health outcomes.¹ Because of the increasing recognition of the value of interprofessional collaborative practice, national competencies have been developed in the United States to facilitate the delivery of IPE within an academic curriculum.² Training future health care providers for interprofessional collaborative practice (ICP) is an important step in achieving the “Triple Aim” of improved patient care and experiences, population health, and reduced costs.³

While approaches to IPE have expanded, assessment in this area continues to develop, and further research in this area is warranted.² Assessment approaches for IPE are varied, and best practices have not been identified. Recent reports from the Institute of Medicine have called for formative and summative assessment of IPE using qualitative and quantitative methods.^4,5 Also, Barr’s modified Kirkpatrick’s hierarchy of assessment is recommended as a framework for IPE.⁶ Using this framework, experts recommend the IPE field include higher order assessments such as changes in behaviors, care delivery, patient satisfaction, and patient outcomes.^7,8 Specific quantitative measurement tools for assessing IPE are available in the literature and continue to expand.^9,10 However, pharmacy educators are challenged with increased emphasis on IPE delivery and assessment as stated in the Center for the Advancement of Pharmacy Education (CAPE) outcomes¹¹ and the Accreditation Council for Pharmacy Education (ACPE) national standards.¹² The CAPE outcomes discuss preparing pharmacy students to be collaborators on the health care team. In addition, the ACPE standards have an increased focus on IPE and specifically require documentation of student assessment outcomes in the Pre-Advanced Pharmacy Practice Experience (APPE) and APPE curriculum. A primer for implementing IPE within pharmacy curricula suggested that each school develop an overall assessment plan; however, an extensive review of specific assessment strategies and tools was not the sole focus.¹³ A recent review focusing on assessment of interprofessional teamwork in medical education is useful.¹⁴ However, literature summarizing IPE assessment strategies that apply to pharmacy education is lacking. The purpose of this review is to identify and briefly describe the quantitative tools available that assess IPE as it pertains to pharmacy education.

METHODS

A systematic approach was used to identify quantitative IPE assessment tools relevant to pharmacy education, beginning with a review of the tools available on the National Center for Interprofessional Practice and Education Resource Exchange (Nexus) website.⁹ This widely recognized source for high-quality measurement instruments included tools initially identified from a curated selection process¹⁵ that used a comprehensive, systematic review in 2012.¹⁶ Several more tools were subsequently added through a peer review process. All tools posted on the Nexus website up to September 2015 were analyzed for inclusion in the initial review. Following acceptance of this manuscript by the Journal, but prior to publication, the Nexus website updated the assessment and evaluation collection in January 2017. A second review of the website was conducted to include any updated or additional tools in August 2017.

To identify additional tools published since 2012, a systematic search of the literature was conducted with the assistance of health sciences librarians adapting the previous search strategy¹⁶ for years 2012-2014. Search terms included interprofessional, interdisciplinary, interoccupation, interinstitution, interdepartment, interorganization, multiprofessional, multidisciplinary, multi-occupation, multi-institution, multi-organization, education, practice, instrument, questionnaire, survey, scale, team, and collaboration. Abstracts were reviewed. Finally, a manual search of journals deemed likely to include relevant assessment tools was conducted (January 2005-June 2015) using the following terms: interprofessional, interdisciplinary, multidisciplinary, evaluation, and assessment. Table 1 lists the journals used in the manual search.

Table 1.

Journals Included in Manual Search

Assessment tools were included if the publication included a copy of the tool and if it assessed knowledge, skills, behaviors, and/or attitudes consistent with the Interprofessional Education Collaborative (IPEC) domains² (values, knowledge of roles, communication, teamwork) or CAPE 2013 outcomes¹¹ related to interprofessional collaboration. Thus, included tools measured the following: attitudes toward/willingness to work in interprofessional teams, professional identity in relation to other health professionals, quality of team interactions/function (how well the team members work together), and/or efficacy of team interactions (patient outcomes). Tools were excluded if they measured a specific learning session only or had limited or no ability to assess students in other activities or curricula. Tools also were excluded if they did not include pharmacists or student pharmacists as part of the health care team assessed, unless they had direct applicability to the role of pharmacists on health care teams and could be potentially applied to pharmacy students (ie, did not include language that precluded pharmacists or student pharmacists or assessed a practice situation in which pharmacists would not typically participate).

All assessment tools included were categorized by level of evaluation using the Kirkpatrick outcomes model⁶ and inclusion of pharmacists/student pharmacists (Figure 1). Any information reported about reliability/validity also was included. To aid end users, the tools were secondarily categorized by IPEC domain (values, roles, communication, teamwork)² and ACPE Standards 2016¹² related to interprofessional education (Standards 11.1-11.3 and 25.6). All of the categorization was completed by one author. If there were questions, the group of authors discussed them and reached a consensus.

Figure 1.

Kirkpatrick Assessment Model

RESULTS

The search strategy identified 44 tools from the Nexus website upon initial review and an additional two tools and two updates from the second review, 158 abstracts from the systematic literature search (117 from MEDLINE, one from ERIC, 40 from CINAHL), and 570 abstracts from the manual search of relevant journals (Figure 2). Thirty-six assessment tools met the criteria to be included in the summary, 29 from the Nexus website, two from the literature search, and three from the manual search of relevant journals (Table 2).

Figure 2.

Diagram of Search Strategy and Yielded Results

Table 2.

Quantitative Tools to Assess Interprofessional Education Applicable for Pharmacy Education

Concerning categorization by Kirkpatrick assessment levels, eight tools were found to assess reaction, 11 tools assessed modification of attitudes/perceptions, one tool assessed acquisition of knowledge or skills, 19 tools assessed behavior change, and three tools assessed change in organizational practice. The length of the tools ranged from five to 59 items. Fifteen tools were designed to assess an individual member of a team; 15 tools were designed to assess the team as a whole, and four tools could be used to assess an individual or a team. The majority of tools (58%) did not include a pharmacist or student pharmacist in the validity or reliability testing of the tool. The authors provided examples of how to use some of the selected tools for assessment within the pharmacy curricula (Table 3). These are examples for pharmacy educators, but this is not intended to serve as “best in class” recommendations.

Table 3.

Examples of Selected Assessment Tools and Their Application to Pharmacy Education

DISCUSSION

Recent publications affirm that the emphasis on interprofessional education is going to be long-standing.^2,3,5,12 As colleges and schools of pharmacy develop IPE opportunities for students, the need for quality assessment measures is clear. A review like this one, the first to identify published IPE assessment tools relevant to pharmacy, would serve as a starting point for developing a quantitative assessment and resource plan. However, there is also a need for qualitative assessment of IPE activities. In fact, mixed methods evaluations, both quantitative and qualitative, are recommended; specifically, if the evaluation questions incorporate the examination of both what and why.^4-7 And while a fairly robust number of quantitative tools have been identified, when evaluating them for placement into Kirkpatrick’s outcomes, there is a lack of tools evaluating the top two levels: change in organizational practice and benefits to patients/clients.⁸

When developing an assessment plan, one must consider curricular mapping of quantitative and qualitative measures to educational standards for the profession in addition to the IPEC core competencies.² It is also important for those developing the plan to be mindful of overassessment and its potential negative impact on the ability to obtain robust evaluation results.¹³ While Table 2 lists and categorizes the available quantitative tools, below is a more detailed summary of the applicability of certain select tools (presented according to the Kirkpatrick evaluation framework⁶) to provide additional context.

Level 1: Reaction

These tools assess the readiness for students or health care providers to engage in interprofessional activities. Tools evaluating this level are most important to include early in the IPE implementation process. However, once IPE becomes an established component of the educational culture, the utility of these tools becomes less apparent. It should be noted that the updates to the Nexus website limited the inclusion of these types of tools and increased the emphasis of tools measuring skills, behaviors, and outcomes. Among the tools in the reaction level of evaluation, the two TeamSTEPPS tools (T-TAQ and T-TPQ) stand out in that they measure reaction when taken individually and can be used as stand-alone tools without full implementation of the TeamSTEPPS training. If the activity has exposed the students to core components of the TeamSTEPPS training, both the T-TPQ and T-TAQ tools,^23,24 can be used to evaluate “team-wide” if TeamSTEPPS training has affected teams overall. An additional measure of reaction is the learner’s opinions of the IPE program or activity. The We Learn Interprofessional Program Assessment Scale²⁵ was developed to collect feedback useful for interprofessional educators designing IPE and may be useful in the didactic portion of the pharmacy curriculum.

Level 2: Modification of Attitudes and Perceptions

The tools in this category assess attitudes toward the values and/or use of interprofessional teams in education and/or patient care. Among all of the tools that assess this level of evaluation, the Attitudes Toward Health Care Team Scale (ATHCT) stands out as it has been evaluated in both pharmacists²⁷ and pharmacy students.²⁸ While its explicit use as a team or individual assessment is not clearly noted, it appears that it could be used in either way. It is a useful tool in measuring learners’ perceptions of interprofessional collaboration as it relates to patient care. The General Role Perception Questionnaire (GRPQ) assesses an individual’s views of the roles of other health professions³⁰ and can be used to measure the change in role perception with repeated use. The Medicine Use Processes Matrix (MUPM) is a tool that evaluates an individual’s understanding of the medication use process in primary care,³¹ specifically the perceptions of the roles involved in this process. A drawback to this tool is that it is based in primary care and cannot be applied outside of primary care practice, thus limiting its utility for pharmacists learning and practicing in other settings.

Two additional tools evaluating this level are the Scale of Attitudes Toward Physician-Pharmacist Collaboration (SATP2C)³² and Student Perceptions of Physician-Pharmacist Interprofessional Clinical Education (SPICE-2).³³ These measures of attitudes may be most useful in a pre/post design to evaluate a particular IPE activity. Both of these tools were designed to specifically include pharmacy and medical students; there are pros and cons to these tools because there are only two professions learners involved in the development and validation process. It should be noted that the SPICE-2 tool was modified and validated in additional learners from other professions by revising the wording of the original questions and changing the name of the tool to Student Perceptions of Interprofessional Clinical Education Revised (SPICE-R).³⁴

Level 3: Behavioral Change

The tools in this category assess quality of interactions or perceptions of roles of each member of the patient care team. It is clear that much attention has been dedicated to this evaluation level over the past decade, as the majority of tools identified for this review, are at this level. While several of the tools have been developed to assess individuals, the majority have been developed to assess teams and evaluate overall team behavior.

There are two tools designed to assess attainment of IPE competencies. The IPEC Competency Survey Instrument can be used in assessing IPE program design and outcomes; it is based on the competencies adopted by organizations in the United States.⁴⁴ However, the tool was designed as a self-assessment; it is unlikely that self-assessment of competencies alone, will be sufficient to fulfill ACPE accreditation standards. The second available tool is the Interprofessional Collaborative Competency Attainment Survey (ICCAS), which is also a self-assessment of IPE competencies.⁴² It evaluates the Canadian competencies of interprofessional care and was intended to be used in a post-pre/post-design. While neither are observational tools, both can be used to improve learners’ self-awareness of the competencies and identify areas for improvement. Further, both tools would likely be useful during a range of IPE experiences: didactic, simulation or APPEs in the pharmacy curriculum.

The Team Skills Scale (TSS) is a self-assessment of skills required to work effectively on an interprofessional geriatric patient care team.⁵⁵ The TSS is commonly used in conjunction with the ATHCT.^27,28 While not specific to the duration of being a team member and lacking student reflection on a specific team interaction, it may serve as a useful tool during APPE or longitudinal pre-APPE IPE activities focusing on the geriatric patient population.

A few tools have been developed to assess individual learners working as part of an interprofessional team. They were developed to use an external observer including faculty, preceptors, interprofessional preceptors for a 360-evaluation, and peers. The Interprofessional Collaborator Assessment Rubric (ICAR) includes items developed to align with the Canadian competencies of interprofessional care.⁴⁰ The reliability of the tool improves with repeated observations. A modified version of the ICAR with the same constructs, but condensed number of items is also available.⁴¹ The Individual Teamwork Observation and Feedback Tool (iTOFT) was developed with a limited number of items to emphasize the feasibility of use in clinical settings.³⁹ All of these tools may be useful to assess students during APPEs in the pharmacy curriculum.

Twelve tools addressing Kirkpatrick’s hierarchy level 3 (behavioral change) and described as tools assessing teams as a whole were identified. These tools can be used on either a longitudinal basis or for a one-time IPE event and be completed by an external observer or be administered in the workplace. Relative to the tools identified to be used in a longitudinal manner, the Collaborative Practice Assessment Tool (CPAT) is geared toward a team working together in a patient care setting for a long period of time.³⁶ It would not be beneficial and is not recommended for assessing brief team interactions. Similarly, the Team Decision Making Questionnaire (TDMQ) is most applicable to long-term teams in a clinical practice.⁵⁴ It can be used in conjunction with a tool measuring team dynamics to identify areas for team development and improvement. And, while its description notes the tool as assessing interprofessional team quality, the items appear more applicable to the individual member and how the team decision-making process specifically helps them. The Interprofessional Model of Patient Care (IPMPC) is a combination of scales (modification of the Effort-Reward Imbalance and Safety Attitudes questionnaires) measuring interprofessional team relationships. It is best used before and after an intervention with an existing team to compare improvement over time.⁴⁵ This tool must be used twice to measure a difference, and it is not a tool to use before an experience with a new, naïve team; it is to be administered to existing teams. From the standpoint of applying this to pharmacy education, the question is whether this tool would be able to detect differences during the common four to eight weeks of an APPE rotation given that these are typical “de novo” teams.

Contrary to the tools evaluating behavior change longitudinally and among existing teams, other tools are available to evaluate behavior change resulting from a single IPE experience and may have more applicability in the pharmacy curriculum. The first of these three tools is the Collaboration and Satisfaction About Care Decisions (CSACD).^37,38 Designed to be used after a single patient encounter, this tool may be considered when assessing interactions in an interprofessional team objective structured clinical examination (OSCE) or for a specific encounter during APPEs. The other single experience tools to measure behavior change are unique in their use and focus. Specifically, the Interprofessional Collaboration (IPC) scale is designed for individuals to assess team members from other professions.⁴³ If used, professional labels may need to be altered depending on the role of the evaluator and evaluatee. For teams with more than two professions, multiple versions of the evaluation would need to be completed as the evaluator completes one evaluation for each member of the different professions on the team. Finally, the Safety Organizing Scale (SOS) tool educates health care providers about elements of safety culture behaviors with a goal to improve safety outcomes.⁵² Tool items do not specify the application to an interprofessional team.

Two additional tools evaluating behavior change are completed by external observers as opposed to the participants themselves and can be used for newly formed or established teams. The first tool, the Performance Assessment for Communication and Teamwork (PACT), is a useful behavioral observation tool, particularly when using the Novice version.^48,49 It could perceivably be used to evaluate teams during patient care APPEs or a simulation. It is best to evaluate synchronous teams during a specific patient encounter. The second tool completed by an external observer is the Observed Interprofessional Collaboration (OIPC).⁴⁷ This tool is applied when observing team meetings where a team discusses multiple patients and is not making immediate patient-care decisions at the point of patient care. This tool can also be useful on APPE rotations or during simulations that involve the transition of care or discharge planning meetings.

Another tool measuring behavior change is the Assessment of Interprofessional Team Collaboration Scale (AITCS) that is used to evaluate a team in the workplace environment.³⁵ While designed to be implemented in a workplace setting as opposed to an educational setting, it was developed in response to the limitations of existing tools; specifically, those lacking the incorporation of the patient as a team member. While other tools exist that measure behavior change, they are either not available for free use (Relational Coordination Scale^50,51 and Team Climate Inventory⁵³) or only available from the author directly (Interdisciplinary Team Performance Scale⁴⁶).

Level 4. Change in Organizational Practice

To date, the highest level of Kirkpatrick’s hierarchy that appears to be evaluated among the available IPE assessment tools is level 4, change in organizational practice. The purpose of these tools is to assess interprofessional collaboration at an organizational level. Three tools addressing this are the Index for Interdisciplinary Collaboration (IIC),⁵⁶ the Interprofessional Socialization and Valuing Scale (ISVS),⁵⁷ and the Healthcare Team Vitality Instrument (HTVI).⁵⁸ While the IIC and ISVS can be used for either individual (to evaluate perceptions of collaboration) or team assessment, the HTVI is intended only for team assessment. Specifically, the use of either the IIC or ISVS can help to identify if an intervention results in changes in the perception of the team’s view of its collaboration.

Limitations

Limitations to this systematic review are threefold. First, tools were included only if they were perceived to be relevant to pharmacy education, which excludes numerous tools that can be used to assess interprofessional teams. For example, a recent review of IPE assessments applicable to medical education includes additional tools that could be used.¹⁴ The Communication and Teamwork Skills (CATS)⁵⁹ and Teamwork Mini Clinical Evaluation Exercise (T-MEX)⁶⁰ tools recommended in this medical education review are important to note, since the ACPE standards emphasize including prescribers in IPE, and it is likely many schools will have medical students involved in interprofessional activities.

Second, only quantitative tools were included in this review. Qualitative tools or methods and mixed methods (qualitative plus quantitative) can also contribute important information to an assessment plan in addition to quantitative data alone.^5,8 For example, Harris and colleagues outline an analytical strategy using qualitative coding techniques that could be used to assess interprofessional case discussions in practice.⁶¹ However, qualitative methods are time consuming to evaluate with large numbers of students, and thus are likely to have limited reach.

Third, the systematic review process used may not have identified all tools available and applicable to pharmacy education. In fact, the authors collectively know of a few additional tools that were not uncovered in this review process. These tools include the Team Performance Scale (TPS),⁶² with reliability testing completed in medical⁶² and pharmacy⁶³ students using team-based learning, the McMaster-Ottawa Scale⁶⁴ that has been used in team observed structured clinical encounter (TOSCE) performance evaluation, and the KidSIM Team Performance Scale⁶⁵ used to measure interprofessional student team behaviors after experiencing an acute-care simulation curriculum. These tools may have application for use in pharmacy education.

Finally, the Interprofessional Collaborative Organization Map and Preparedness Assessment (IP-COMPASS),⁶⁶ Team Development Measure (TDM),⁶⁷ and Assessment for Collaborative Environments (ACE-15)⁶⁸ would likely not be useful for assessment of pharmacy learners, but could be beneficial to measure the clinical learning environment and culture for IPE in APPE settings.

CONCLUSION

Thirty-six quantitative assessment tools have been made available to assess interprofessional education for teams that include or are applicable to the pharmacists or pharmacy students. Of the available tools, the majority assesses the Kirkpatrick level 3 (behavioral change). Each of the tools summarized presents advantages and disadvantages. At this time, no one comprehensive tool exists to fulfill assessment needs for appropriately assessing interprofessional education competencies as a whole. However, numerous tools are available that can be mapped to all IPE-related accreditation standards in ACPE Standards 2016. When developing an assessment plan, we recommend the use of a cadre of quantitative tools in addition to reflective writing and/or qualitative tools to assess student progression and development of interprofessional collaborator competency. We recommend development of future tools that require external observation of individual and team behavior, change in organizational practice, and patient benefits that are intended to be used for both short-term and newly formed teams of learners. In addition, performing additional validation of tools that did not include pharmacy learners and comparing the utility of various tools in pharmacy education settings could be considered for future research.