Development and Validation of the Student Perceptions of Physician-Pharmacist Interprofessional Clinical Education (SPICE) Instrument

Abstract

Objectives. To describe the development and validation of an instrument designed to assess student perceptions of physician-pharmacist interprofessional clinical education (SPICE).

Methods. Faculty members from pharmacy and medical schools developed items for the instrument, and 179 medical and pharmacy students completed the scale. Psychometric properties, including reliability and construct validity, were assessed using confirmatory factor analysis.

Results. The final instrument consisted of 10 items with 3 subscales measuring student perceptions of interprofessional teamwork and team-based practice, roles/responsibilities for collaborative practice, and patient outcomes from collaborative practice. Validity and reliability of the instrument were demonstrated.

Conclusion. The SPICE instrument demonstrated promise as a valid and reliable measure of pharmacy and medical student perceptions of interprofessional clinical education. SPICE may serve as a useful instrument for educational researchers in assessing the impact of interprofessional educational experiences.

INTRODUCTION

The use of interdisciplinary care teams has been promoted to improve health and safety outcomes for patients.^1-5 Several studies have characterized benefits associated with team-based healthcare delivery.^6-15 Specific examples include greater blood pressure control when care is delivered by a physician-pharmacist team,^8,12 increased patient satisfaction,¹⁴ decreased healthcare costs,¹⁴ and improved quality of care.¹⁵ Despite this progress in the patient-care arena, consecutive Cochrane reviews performed within the last 10 years have neither consistently nor comprehensively established the benefits of the interdisciplinary model as it relates to healthcare education.^16,17 Authors of the most recent of these reviews cite the small number of studies, the heterogeneity of interventions, and methodological limitations as principal limitations and call for more rigorously designed studies to “provide better evidence of the impact of interprofessional education on professional practice and healthcare outcomes.”¹⁷

To provide evidence of the impact of interprofessional education within healthcare disciplines, there have been efforts to produce, validate, and use measurement scales.^18-21 Specific instruments include the Readiness for Interprofessional Learning Scale (RIPLS),^18,21 the Interdisciplinary Education Perception Scale (IEPS),²¹ the Attitudes Toward Health Care Teams Scale (ATHCTS),²¹ and the Scale of Attitudes Toward Physician-Pharmacist Collaboration (SATP²C).^19,20 Since the reliability and validity of measurement instruments are contextually constrained, further development, refinement, and validation of measurement scales are warranted.²²

This study entailed the development and validation of an instrument intended to measure student perceptions of physician-pharmacist interprofessional clinical education (SPICE). By formally establishing the psychometric properties of the instrument, research findings from its use in clinical education settings will be strengthened. Further, the development and validation of a measurement instrument can aid in assessing the impact of interprofessional education initiatives on students’ perspectives regarding collaborative endeavors.

Physician and pharmacist faculty members practicing at the Texas Tech University Health Sciences Center (TTUHSC) Center for Family Medicine in Amarillo established a collaborative practice clinic in August 2011 to deliver Medicare’s Annual Wellness Visit (AWV) benefit.^23,24 Prompted in part by the TTUHSC Quality Enhancement Plan for regional reaccreditation, which was designed to address perceived deficiencies in interprofessional teamwork,²⁵ these faculty members decided to include medical and pharmacy students in the AWV clinic to increase their exposure to interprofessional education. The AWV team investigated use of the aforementioned scales to assess changing perceptions of students toward interprofessional education following exposure to the AWV clinic but ultimately decided that a customized instrument was more desirable. The rationale for this decision was twofold. First, only 1 of the previously validated instruments was designed specifically for physician-pharmacist teamwork (SATP²C); physician-pharmacist teamwork was the delivery model for the AWV clinic. Additionally, in order to contribute maximally to the aforementioned Quality Enhancement Plan, the AWV team intended to generate more data specific to patient outcomes and educational preparedness for interprofessional education. SPICE was created out of this desire.

METHODS

A structured process was followed to develop and refine the SPICE instrument. First, a pool of questions (items) was created using a 5-point Likert-type response system on which 5=strongly agree, 4=agree, 3=neutral, 2=disagree, and 1=strongly disagree. Items were either derived from existing instruments or developed independently by the researchers. Content validity of the pool was established by means of expert review prior to pilot testing in a small sample of students from the population of interest. No changes to the instrument were made based on results of this pilot test. The instrument was then administered to a larger, more representative sample of medical and pharmacy students. Confirmatory factor analysis (CFA) was performed on data generated during this test to assess the validity and reliability of the instrument and to make refinements to the instrument, leading to accepted standards of validity and reliability.

The SPICE instrument was designed for use with students in medical and pharmacy schools. After receiving approval from the TTUHSC institutional review board, pilot testing of the initial instrument was conducted with a relatively small sample of 27 students from the TTUHSC Schools of Medicine and Pharmacy at a single campus (ie, Amarillo). A paper copy of the instrument was distributed to third-year medical and fourth-year pharmacy students participating in the interprofessional AWV clinic in 2012. Third-year medical and fourth-year pharmacy students were selected based on curricular requirements of practice experiences in family medicine (medical students) and ambulatory care (pharmacy students).

Following pilot testing of the SPICE instrument, third-year medical and fourth-year pharmacy students from multiple campuses within the TTUHSC system (ie, Abilene, Amarillo, Dallas, Lubbock, and Midland/Odessa) were asked to complete the anonymous, paper-based 20-item instrument. The final sample was comprised of 179 students, providing an adequate sample size for instrument validation.

For initial pilot testing of the instrument, analysis of its validity followed a process entailing some subjectivity. The pool of items selected for inclusion in the instrument was based upon the perspectives of experts (faculty within TTUHSC Schools of Medicine and Pharmacy). Items for the instrument were generated by 2 of the investigators, and then reviewed by faculty co-investigators in medicine and pharmacy. Item wording and ordering was developed iteratively through dialogue among investigators until consensus was achieved, leading to the development of the initial instrument. This preliminary process was focused on establishing the content validity of the instrument.

CFA was used to establish construct validity.²⁶ The CFA process entailed a priori model specification (defining factor structures characterizing the relationships of latent and observed variables) followed by testing to see if sample data fit the model, thereby confirming the model. To determine if a specified model achieves goodness-of-fit, a variety of statistical tests and indices were used.^18,26 A chi-square test was used to test the null hypothesis that the covariance matrix for the specified model equals the data covariance matrix. Given that the goal of this test is to fail to reject the null hypothesis, a non-significant finding (p>0.05) was desired. Because this measure has some limitations, the ratio of chi-square to degrees of freedom (df) was an alternative measure. A common standard is that a good model will produce chi-square/df <2. Another goodness-of-fit measure was the comparative fit index (CFI). Goodness-of-fit is demonstrated when CFI>0.95. Root mean square error of approximation (RMSEA) is a lack of fit indicator, so goodness-of-fit was indicated when RMSEA <0.06. Generally, several measures were generated to establish goodness-of-fit. If the proposed model achieved goodness-of-fit, then the relationships of variables (both latent and observed) within the model were reviewed. Parameter estimates, such as correlation coefficients and regression weights, were reviewed to determine the relationships of variables (both latent and observed) and estimates of variance.

Following the pilot, the SPICE instrument was tested with a large sample of students, producing a final sample size of 179. CFA was conducted using AMOS (IBM SPSS AMOS 21, Chicago, IL). The originally proposed structure of the instrument was evaluated for model fit. To determine if instrument refinement was needed, model fit indices were explored. Alternative factor structures were evaluated, and deletion of items that did not contribute to the measurement model was explored. Following review of alternative models, a final refinement to the instrument was crafted. Confirmatory factor analysis was subsequently used to characterize and assess the structure of the final, refined SPICE instrument. In addition to using CFA to assess the SPICE instrument’s validity and reliability, Cronbach alpha and composite reliability were calculated to assess instrument reliability. For all tests of significance, the a priori level of significance was alpha=0.05.

RESULTS

The initial instrument developed to measure student perceptions of physician-pharmacist interprofessional clinical education was comprised of 20 items, which were created specifically to measure attributes of students’ perceptions of collaborative clinical education. Three constructs were specified: Interprofessional Teamwork and Team-Based Practice (5-item scale), Roles/Responsibilities for Collaborative Practice (7-item scale), and Education-Related Perceptions of Interprofessional Learning (8-item scale). Of the 20 items included in the instrument, 5 were grounded in the SATP²C scale.¹⁹ Of the 5 SATP²C items, 2 were modified for nuanced contextualization and 3 were used directly. The SATP²C instrument’s psychometric properties have been assessed and reliability and validity of the scale documented.¹⁹ Three of the 20 items in the SPICE instrument were drawn from an instrument previously developed (but not formally validated) by a member of the research team. Of these items, 1 was unchanged from its original form and 2 were modified. The remaining 12 items were crafted by the researchers to ensure that the SPICE instrument had a sufficient number of items per measured construct.

In the first step of assessing the reliability of the instrument, a preliminary sample from 27 medical and pharmacy students was collected. For the 20-item instrument, Cronbach alpha was 0.812, indicating appropriate internal consistency reliability. For each item, Cronbach alpha was calculated with the item removed from the scale. Of the 20 items, only 1 was found to increase the reliability of the instrument if removed. Given that removal of this single item would only marginally increase reliability, however, all 20 items in the instrument were retained for further testing.

To assess the validity of the initial 20-item, 3-factor instrument, a CFA using maximum likelihood estimation was conducted for the final sample (n=179). The sample included broad representation by academic discipline, year in academic program, and gender (Table 1). Responses were received from students on 5 campuses. Most students were white (60%) or Asian (24%).

Table 1.

Sample Characteristics of the Study Student Population (N=179)^a

Standardized estimates for the CFA model are provided in Table 2. A review of the CFA model produced a variety of concerns related to the validity of the structure. First, the 3 proposed factors were significantly highly correlated (>0.90) with each other, suggesting that they may not have measured unique constructs. Given that the model may have been misspecified, review and revision of the proposed model structure were warranted. Second, several of the item loadings were low (<0.30), and some were negative, suggesting that some of the items provided little contribution to the instrument, potentially leading to model misspecification. Lastly, the overall model fit attributes did not meet common standards (eg, CFI>0.95, RMSEA<0.06). Accordingly, review of the CFA for the initially proposed, 3-factor model led to the conclusions that: (1) items should be removed from the instrument, (2) the factor structure should be revised, and (3) a CFA of the revised structure should be conducted to better assess the factors that the instrument was capturing.

Table 2.

Confirmatory Factor Analysis Results for Initial 3-Factor, 20-Item Structure

The original 20-item scale was revised, guided by theoretical and statistical principles. From a statistical perspective, items were considered for removal from the instrument based on low or negative loadings. From the theoretical perspective, items considered to be potentially contextually biased (eg, items for which bias might be present among students from a particular academic discipline) or of questionable generalizability were considered for removal from the instrument. Review and refinement of the factor structure were then conducted. Based on a rigorous review grounded in these guiding principles, the factor structure was revised and 10 items were removed from the instrument, yielding a more parsimonious model. The 3 factors comprising the revised structure were identified as: Interprofessional Teamwork and Team-Based Practice (6-item scale), Roles/Responsibilities for Collaborative Practice (2-item scale), and Patient Outcomes from Collaborative Practice (2-item scale). The resulting 10-item instrument, with the item numbers from the original 20-item instrument retained, is provided in Table 3.

Table 3.

Items in the Final Student Perceptions of Physician Pharmacist Interprofessional Clinical Education Instrument^a,b

Given the questionable factor structure of the initial model (Table 2), a CFA was conducted for the revised 3-factor, 10-item model. Prior to conducting the CFA, data were reviewed to assure appropriateness of the analysis. Standardized loadings for the revised CFA model are provided in Table 4. Model fit characteristics demonstrated appropriateness of the revised model. The model chi-square test was not significant (p=0.183), and the ratio of chi-square to degrees of freedom was 1.220, demonstrating the specified model’s goodness of fit. Additional fit indices (CFI=0.987, RMSEA=0.036) provide further evidence of goodness of fit.

Table 4.

Confirmatory Factor Analysis Results for Final 3-Factor, 10-Item Instrument Structure

To characterize construct validity, convergent and discriminant validity were reviewed. All standardized regression weights in the revised model were significant, with only 1 regression weight less than 0.60 (Table 4). Lower bounds of item reliabilities (ie, squared multiple correlations for each item) are also represented. The reliabilities represented the proportion of item variance explained by the factor (latent variable). The relatively high item loadings (ie, regression weights) and the absence of modification indices, which suggest item cross-loadings, provided support for convergent and discriminant validity, respectively. Further, the positive correlations of the factors were consistent with the overall nomological framework of student perceptions of physician-pharmacist interprofessional clinical education. The 3 factors comprising the model collectively contributed to the overarching measure of student perceptions, thereby supporting the notion of convergent validity. Though the 3 factors were correlated, 2 of the 3 correlations (0.31, 0.36) were small, and the third was moderate. The modest correlations suggested that the 3 factors were measuring somewhat unique constructs, thereby providing additional support for discriminant validity. The combined assessments of convergent and discriminant validity collectively provided evidence of construct validity.

Composite reliabilities for each of the 3 factors (subscales) were derived. For Interprofessional Teamwork and Team-Based Practice, the reliability was 0.851; for Roles/Responsibilities for Collaborative Practice, it was 0.582; and for Patient Outcomes from Collaborative Practice, it was 0.726. These values generally demonstrated acceptable reliability for the instrument, though reliability for Roles/Responsibilities for Collaborative Practice was marginally below the recommended standard of 0.60.²⁷ For the 2 factors that were associated with only 2 items, and in particular for Roles/Responsibilities for Collaborative Practice, expanding the number of items might have enhanced construct validity. To further assess internal consistency reliability of the revised SPICE instrument, Cronbach alpha was calculated for the instrument (alpha=0.837).

To further establish the validity of SPICE, the instrument was readministered to 46 students in the sample following an interprofessional education intervention. These students had a significant gain in perception scores on all 3 factors (Interprofessional Teamwork and Team-Based Practice, p=0.003; Roles/Responsibilities for Collaborative Practice, p<0.001; Patient Outcomes from Collaborative Practice, p<0.001). Further, within each factor, standard errors were roughly equivalent between pre- and post-interprofessional clinical education experience. These findings provided evidence of the stability and sensitivity of the instrument.

DISCUSSION

The initial development of the SPICE instrument focused on crafting an instrument specifically to assess the perceptions of medical and pharmacy students engaged in interprofessional clinical education. The instrument was to be used to characterize students’ perceptions before and after an interprofessional clinical education experience. The goal was to determine if perceptions changed following the interprofessional experience. Some of the items in the initial instrument were introduced to identify differences by academic discipline. For example, medical and pharmacy students were anticipated to have differing views about their respective qualifications and responsibilities. Following the refinement process, however, the character of the final SPICE instrument changed, focusing less on identifying differences in perception by academic discipline and more on commonalities between academic disciplines, with an emphasis on cooperation. Items that might have been subject to bias (eg, “Medical and pharmacy school are equally rigorous”) were excluded from the final instrument. Additionally, the constructs were revised to include a factor that focused on perceptions regarding the impact of interprofessional practice on patient outcomes (ie, patient outcomes from collaborative practice). As a result, the final SPICE instrument consisted of 2 factors that focused inwardly on student experiences and 1 factor that focused outwardly on patient outcomes.

The SPICE instrument was initially developed for use in assessing change in students’ perceptions during a predefined educational intervention. However, broader applicability and utility of the instrument should be considered when determining its merits within the domain of other validated instruments. In 2009, the American College of Clinical Pharmacy (ACCP) published a White Paper documenting the principles and application of interprofessional education.²¹ The ACCP report identified validated instruments that have been used to assess interprofessional education. Specific instruments reviewed included the RIPLS, the IEPS, and the ATHCTS. Although the validated SATP²C was not documented in the ACCP report, items from this instrument were used in developing the SPICE instrument.^19,20

While SPICE, the SATP²C, and the scales referenced in the 2009 ACCP report measure student perceptions regarding interprofessional education, there are important differences. SPICE is comprised of 3 subscales and 10 items, with each item allowing a 5-point response (ie, strongly disagree, disagree, neutral, agree, strongly agree). Analysis of the RIPLS instrument suggests that it is comprised of 3 or 4 subscales (Teamwork & Collaboration, Professional Identity, Negative Professional Identity, Roles & Responsibilities) and nineteen 5-point items.^18,21 The IEPS consists of 4 subscales (Competence/Autonomy, Perceived Need for Cooperation, Perception of Actual Cooperation, Understanding Others’ Values) and eighteen 6-point items.^21,28 The ATHCTS consists of 2 subscales (Quality of Care/Process, Physician Centrality) and twenty-one 6-point items.^21,29 The SATP²C is comprised of 3 subscales (Responsibility & Accountability, Shared Authority, Interdisciplinary Education) and sixteen 4-point items.^19,20 Because each instrument can be used to measure students’ perceptions regarding interprofessional education, there may be some overlap among the instruments. Although each of the instruments can be useful in a variety of settings, SPICE differs from the others in that it has a unique subscale that deals explicitly with perceptions regarding the association of team-based healthcare delivery with patient outcomes. Other instruments contain items associated with patient outcomes, but no instruments appear to have a subscale specifically focused on patient outcomes. SPICE may be an attractive measurement tool when there is an intentional focus on students’ perceptions related to patient outcomes. Further, the SPICE instrument has fewer items than any of the 4 established instruments, making it easier to administer and likely less time-consuming for respondents.

Methods for validating the instruments differ. Reliabilities have been published for all of the instruments discussed, but differing analytical methods have been used for characterizing their factor structures. Exploratory factor analysis (EFA) has been used to determine the number of factors and the factor loadings for some of the instruments,¹⁹ whereas CFA was used with SPICE and others.¹⁸ In general, EFA allows for determination of the numbers of factors measured by an instrument, and CFA allows for confirmation of the factor structure. CFA on the 3-factor structure of the SPICE instrument yielded clearly acceptable goodness-of-fit measures and item reliabilities. Additionally, overall instrument reliability generally met standards of acceptability. Though EFA and CFA are useful tools, CFA is based upon a priori model specification and, therefore, may be less susceptible than EFA to chance variations in the data. Further, with CFA it is possible to specify a simple model structure that does not allow items to cross-load to multiple factors. When selecting from alternative instruments for use, methods used to assess each instrument’s validity merit consideration.

Future research assessing results from the use of multiple instruments on the same population may add clarity to the distinctions among the instruments. At present, it is possible only to compare the results of validation studies of instruments on nonequivalent populations. In the future, further studies with larger samples and involving multiple institutions and student populations would be beneficial to better establish the validity of the SPICE instrument. The instrument’s reliability may be further improved if the 2 factors comprised of only 2 items were expanded. A 12-item instrument, with each factor comprised of 3 or more items, might yield improvements while retaining the unique focus on patient outcomes and a relatively small number of items. Though future revisions of SPICE may be pursued, the instrument depicted in Table 3 demonstrated validity.

While the SPICE instrument has demonstrated validity and reliability, there are limitations. First, the instrument’s validity and reliability were determined in part by the environment, setting, and population. Validity and reliability of the revised 10-item instrument were demonstrated only in third-year medical and fourth-year pharmacy students. Additionally, demographic data were self-reported and not verified. Findings from this review should be confirmed by application of the SPICE instrument in other settings and with other student populations. Replication of the findings observed in this study may confirm the instrument’s validity.

CONCLUSIONS

This study demonstrated the validity and reliability of the SPICE instrument, which was designed to measure student perceptions of physician-pharmacist interprofessional clinical education. The instrument may be used by researchers as a means of assessing the impact of interprofessional education experiences on students’ perceptions. SPICE was originally developed for use in a specific educational experience. Given the formal assessment of its psychometric properties, as documented herein, SPICE may be attractive for broader use among educational researchers within medical and pharmacy colleges and schools. Further research should be conducted to determine if the SPICE instrument can be revised for broader applicability to students in other healthcare disciplines. Also, further revisions to the instrument, including adding items to the 2 subscales comprised of only 2 items, may improve the validity and usefulness of the instrument.