Establishing the Content Validity of a Student Pharmacist Patient Counseling Competency Assessment in Oncology

Abstract

Objective. The goal of this project was to establish content validity and describe internal consistency of a patient counseling competency assessment instrument used to evaluate student pharmacists practicing in an oncology setting.

Methods. The study involved a modified e-Delphi panel of oncology clinical pharmacy specialists, clinical pharmacy generalists, and oncology pharmacy residents. Iterative rounds of the e-Delphi process were conducted until consensus was reached on most instrument items. Consensus was defined as agreement by at least 75% of participants that an item was or was not important.

Results. The modified e-Delphi process included three rounds of responses from 13 panelists and resulted in a 35-item instrument with consensus reached on 33/35 (94%) of the items. All participants indicated that the assessment result options allowed them to indicate the student’s level of competency either extremely well or very well.

Conclusion. A modified e-Delphi method was used to validate a reliable instrument for the assessment of student pharmacist counseling abilities in an oncology setting. Similar methodology should be considered during the development of student assessment tools, especially for high-impact student pharmacist activities such as chemotherapeutic medication counseling.

INTRODUCTION

Student pharmacists are important resources that allow for an extension of the clinical pharmacist workforce. In 2011, the Pharmacy Practice Model Initiative (PPMI) of the American Society of Health-System Pharmacists called for increased involvement of student pharmacists in drug therapy management services. ^{1, 2} Also, for new pharmacy graduates, educating patients regarding the appropriate use of medications is a core entrustable professional activity (EPA). ³ Student pharmacists have carried out increasingly complex functions, moving from observers to integral members of the care team. Delgado and colleagues reported on student pharmacists as part of a layered learning model in an acute care setting, which included patient counseling at discharge and for targeted disease states. ⁴ The student pharmacists allowed for pharmacy services to be expanded to patients who would not otherwise interact with pharmacists. During experiential education, student pharmacists not only gain valuable experience but also provide important clinical services to patients.

Student pharmacists completing advanced pharmacy practice experiences (APPEs) at The Johns Hopkins Hospital (JHH) are expected to provide patient counseling. Preceptors ensure each student pharmacist is sufficiently prepared and able to counsel patients as early as possible in the rotation. To promote autonomy, student pharmacists at JHH undergo patient counseling competency assessments, which cover topics relevant to their APPE (eg, internal medicine topics such as diabetes management, opioids and naloxone, and proper use of inhalers and oncology topics such as prophylactic antimicrobials and immunosuppressants for hematopoietic stem cell transplantation [HSCT]). Using a role-playing approach, the preceptor determines whether the student is competent to counsel independently, requires supervision while counseling, or must repeat the assessment.

While our institution has developed instruments to evaluate student counseling competency for targeted topics, their use is currently not supported by validity evidence. In general, the literature lacks published work that describes validity evidence of instruments used to assess student pharmacists’ readiness to provide patient counseling in oncology. Some validity evidence exists for instruments used in other areas of pharmacy: Lin and colleagues established validity evidence for an instrument used to evaluate pharmacists providing counseling on herbal and dietary supplements. ⁵ After initial creation of the instrument, pharmacists determined that it demonstrated favorable reliability and construct validity. In another study, Zhou and colleagues assessed the validity and reliability of preceptor assessments of student pharmacists’ performance on clinical rotations. ⁶ While these works are important and provide a useful framework, they are not specific to the oncology setting.

At JHH, existing institution-specific competency assessment instruments for patient counseling have been used for several years. While they are perceived as valuable among preceptors, there has not yet been a formal evaluation of the validity, content, or structure of the assessment instruments. As student pharmacists are a critical part of the pharmacy workforce and allow the pharmacy team to counsel more patients, it is important to ensure that the assessment instruments accurately reflect intended content and have the support of preceptors.

The goal of this project was to validate the instrument to ensure the delivery of high-quality, effective, and relevant patient counseling. A secondary goal was to develop a framework to validate assessment instruments for other topics in oncology and internal medicine. The main objective was to evaluate the content validity of the assessment instrument through an e-Delphi panel of experts involved in conducting the counseling competency assessments.

METHODS

To evaluate the content validity of the assessment instrument, a modified e-Delphi panel process was conducted. Participants were 13 oncology clinical pharmacy specialists, generalists, and residents at JHH. This group included all known individuals within the oncology practice area of the institution who had both content expertise and experience with assessing student pharmacists for counseling activities. The project was approved by the Johns Hopkins Medicine Institutional Review Board.

The Delphi process is a method of systematically determining consensus among experts; the process consists of several rounds of surveys or consultations. ⁷ The use of the e-Delphi process (referring to an electronic version) has been described ⁸ and applied in several health care fields. ^9-11 Advantages of e-Delphi over traditional Delphi include the ability for panelists to maintain anonymity, the convenience of collecting responses for investigators and panelists, and the ability to include panelists in locations that may not be convenient or feasibly visited by the investigator. ¹² Several Delphi panels, in the traditional and modified form, have been conducted in academic and experiential pharmacy practice settings and have focused on patient counseling, definition of competencies, and curriculum design. ^{7, 14-18}

The topic for which the instrument was assessed was “immunosuppressants for patients undergoing HSCT.” The study began with distributing an electronic survey (Qualtrics International Inc) to all panelists for e-Delphi round 1 (N = 13). Basic nonidentifying descriptive data were captured, including role/job title and years of experience. In the main portion of the survey, panelists were asked to evaluate each item of the assessment instrument in use at the practice site. Panelists were asked to rate the level of importance of each item on a five-point Likert scale and to suggest whether existing items should be removed or new items should be added. The estimated response time for the survey was six to 10 minutes. Panelists were asked to answer the questionnaire within 15 business days, at which point weekly email reminders were sent for an additional two weeks.

The expert panel conducted three instrument evaluation rounds. After each round, responses were summarized, and the next round began with delivery of the summarized, anonymous responses to each panelist. Each panelist then reviewed the summarized data, compared them to their own responses, and responded to the survey in the next round. In round 1, free-text response questions were included after each survey section asking the participant whether they would add any new items or modify existing items. In round 2, panelists were presented with a summary of the free-text responses and were asked whether they categorically agreed or disagreed with the proposed change. If consensus was reached, the changes were made and incorporated into the round 3 survey. After rounds 1 and 2, if consensus was reached that an item was unimportant, the item was removed in the subsequent survey. The final revised instrument was shared with all panelists.

The a priori definition of consensus for this analysis was at least 75% of panelists having similar observations. For example, at least 75% of participants needed to report that an item was “extremely important” or “very important” (scores of five and four on the Likert scale, respectively) for consensus to be reached that the item was important. Conversely, at least 75% of participants needed to report an item was “slightly important” or “not at all important” for consensus to be reached that the item was not important. Reliability of responses in each round was described by Cronbach alpha, which has previously been applied in Delphi studies. ^{18, 19} Aggregate data were downloaded from Qualtrics and stored on a secure drive hosted by Johns Hopkins. Statistical analyses were conducted using Stata/IC 13 (StataCorp LLC).

RESULTS

Participants (N=13) included five current clinical specialists (38%), three current PGY2 oncology pharmacy residents (23%), three current pharmacist generalists (23%), and two clinical specialists formerly employed by the institution (15%). The response rate for round 1 was 92% (n=12). The average duration of the pharmacists’ experience was seven years (range one to 16), and the average duration of oncology experience was six years (range one to 15). Most participants had previous experience administering the counseling assessment on hematopoietic cell transplantation (HCT) (58%) or other topics (25%).

In round 1, consensus was reached for 29 of 32 items; the overall Cronbach alpha was 0.87. The three nonconsensus items were reevaluated in round 2, after participants reviewed the summary results. A total of 21 suggestions for changes to the proposed instrument items were submitted.

In round 2, consensus was reached for 26 of 32 items, with 69% participation (n = 9). Of the six items that did not reach consensus, five items were removed and one revised based on qualitative feedback. Additional questions were added seeking consensus on the addition, removal, or change of items from round 1, based on the descriptive comments received. Consensus was reached for 16 of 21 proposed changes, which were incorporated into the round 3 survey. In this iteration the Cronbach alpha was 0.93. Of the four comments submitted in round 2, none resulted in substantive changes to the survey items.

The round 3 survey had a 100% participation rate (n=13) and demonstrated consensus in all but two items regarding the goal serum concentration ranges for tacrolimus and sirolimus, which on further review, was deemed to be important inclusions, so the items remained in the final assessment instrument (Table 1). The modified e-Delphi process resulted in a 35-item instrument (Figure 1) with consensus reached on 33/35 (94%) of the items, and a Cronbach alpha of 0.95. All participants indicated that the assessment result options allowed them to indicate the student’s level of competency either extremely well or very well.

Table 1.

Summary of Responses to Round 3 of an e-Delphi Panel Conducted to Establish the Content Validity of a Patient Counseling Competency Assessment in Oncology for Pharmacy Students

Figure 1.

Final items included in the competency assessment for hematopoietic stem cell transplant immunosuppression. This instrument is to be used by a preceptor to assess a student’s counseling competency. The preceptor may use the checkboxes to track the student’s progress and note areas for improvement at the end of the assessment. Abbreviations: GVHD: graft-versus-host disease; MMF: mycophenolate mofetil.

DISCUSSION

Student pharmacists can undertake complex tasks including patient counseling. While standardized instruments have been used to evaluate the competency of student pharmacists to counsel patients at our institution for several years, the validity and internal consistency of the instruments had not been evaluated, nor were previously validated tools available in the literature. We believe that applying a consistent approach to the training and assessment of student pharmacists is important to ensuring the delivery of consistent, high-quality patient education. This report demonstrates the feasibility of conducting a modified e-Delphi process to establish content validity and reliability of an instrument used to assess student pharmacist counseling competency.

Content validity was established by iterative rounds of consensus decision-making, as outlined by others. Internal consistency was evaluated by calculating Cronbach alpha. The final assessment instrument yielded an alpha of 0.95, ranging from 0.69 to 0.94 among sections of the instrument, indicating very good internal consistency.

The clinical content for this effort was immunosuppressive agents used for prophylaxis of graft rejection and graft-versus-host disease for patients undergoing allogeneic hematopoietic cell transplantation at our institution. This topic was chosen because of its overall complexity, including multiple medications, compared with other existing topics. Thus, it may not be readily generalizable to other institutions. However, it serves as a proof of concept for those using similar approaches to determining whether student pharmacists can competently provide patient counseling. While most participants had experience using the instrument, several years elapsed between original implementation and this work and some staff had been replaced, which we believe allowed for objective participation. We also expect that previous experience assessing students with the instrument provided valuable context and subject matter expertise in which participants considered necessary changes.

The modified e-Delphi method was straightforward in its design and application, using survey software and email communication. Yet, because the modified e-Delphi process may be prohibitively labor intensive to apply it to all topics on which student pharmacists may be assessed, it may be most useful for establishing instrument validity for complex topics, or those for which subjective controversy exists. Then, the institution-specific framework can be employed to revise assessments of other, more straightforward topics. The high participation rate for our surveys indicates that stakeholders had a clear interest in ensuring the appropriateness of the instrument. However, it may become burdensome to repeat the process several times. Students are currently assessed on topics chosen at the discretion of their preceptor, based on the needs of patients during the rotation. This creates an opportunity for specific groups of preceptors to focus on topics that may be most relevant to their practice area.

The revised instrument is now used for assessing student pharmacists, and future directions for this research will include applying the modified e-Delphi process to other assessment instruments for student pharmacists at our institution. By establishing validity of one specialized instrument, additional instruments may undergo a similar process such that pharmacy preceptors can be confident that student pharmacists are evaluated on appropriate criteria. With proven consensus and a demonstrated high internal consistency of the instrument, future studies may then determine the effectiveness of patient counseling conducted by student pharmacists and may evaluate students’ performance with patients as compared with the assessment outcomes from the initial role play with the preceptor.

CONCLUSION

A modified e-Delphi method was employed to establish content validity and internal consistency of a frequently used assessment instrument for student pharmacists participating in patient counseling in oncology. While the end result is an improved assessment instrument for counseling patients on immunosuppression for HCT, this work also serves as a proof of concept and feasibility study. The modified e-Delphi method may be used to establish content validity for assessment instruments in other areas of patient counseling, including content areas that may be complex or controversial.