Integration of an Internet-based Medical Chart Into a Pharmacotherapy Lecture Series

Abstract

Objectives

To integrate an Internet-based medical chart (IMC) system into a pharmacotherapy course to facilitate evaluation and feedback processes, foster development of written documentation skills, and prepare pharmacy students for future changes in electronic medical documentation systems.

Design

An IMC system was introduced into a pharmacotherapy course for third-professional year pharmacy students and 4 “finish the SOAP note” activities were added to the curriculum. Students' performance on the SOAP notes were assessed by a team of evaluators. At the end of the semester, students and evaluators completed separate 6-item survey instruments concerning the usefulness of the IMC system in meeting the course objectives.

Assessment

Students' performance on documentation activities improved over the course of the semester: 87% of the students avoided repeating previous mistakes by their final documentation activity. The vast majority of the students and evaluators found the system easy to use and the activities helpful.

Conclusion

The development, implementation, and initial expansion of the IMC system across both laboratory and pharmacotherapy courses was a success. Continued integration into clinical coursework is planned and will further expand opportunities for applied learning experiences to prepare students for their experiential program and beyond.

INTRODUCTION

The Internet-based Medical Chart (IMC) is an integrated system that serves as both a simulated electronic medical record and an evaluation system for students' documentation of simulated patient care activities.¹ The IMC system was developed because attempts to find a commercially available electronic medical record system that would meet our educational needs proved difficult. Commercially available products lacked an integrated, efficient scoring and feedback mechanism. Students could enter notes from simulated patient encounters into the electronic chart, but to be evaluated, the notes would have to be printed out on paper and sent to evaluators, which created administrative inefficiencies. Furthermore, commercially available products lacked the ability to function in simulated time, which was necessary for students to complete multiple follow-up visits with the same simulated patient. Use of a commercially available product may have also resulted in students being more concerned about how and what to document in the specific system rather than how and what to document for the patient. While traditional Internet-based educational support systems (ie, WebCT and Blackboard) could be considered for submission of work electronically, they lack the capability to function as a simulated electronic medical record. Thus, the IMC system was developed to meet the following objectives: (1) serve as a medical chart for documentation of multiple simulated encounters for a given simulated patient; (2) manage the dates of note entries to reflect simulated elapsed time between entries rather than actual elapsed time; (3) foster development of documentation skills and clinical decision making; (4) optimize the quality and timeliness of feedback to students; and (5) minimize the administrative workload of documentation and feedback activities. The IMC system was created to allow users to complete the following distinct but interrelated medical chart tasks: student review of previous student and simulated provider SOAP (subjective, objective, assessment, and plan) note entries prior to conducting a simulated patient visit; student entry of SOAP notes documenting interaction with a simulated patient; student revision of SOAP notes when student's notes are unsatisfactory; student completion of “Finish the SOAP Note” exercises (the student is given the subjective and objective data but must complete the assessment and plan portions of the SOAP note).

The IMC system requires both a server running Macromedia ColdFusion (Adobe Systems Incorporated, San Jose, Calif) that houses the dynamic web pages and a server-side database (Microsoft Access) to manage the simulated cases and archive student entries and evaluators' feedback. Students, evaluators, and course coordinators can work with the IMC system using any computer with an Internet connection. Appendix 1 provides an example of a student's view of a simulated chart in the system. The dynamic web pages provide customized views of the electronic medical record and evaluation system based on user requests and the conditional programming of the database and web pages.

The IMC system was originally developed to support longitudinal simulated patient care in the second-professional year of the University of Minnesota College of Pharmacy teaching laboratory course sequence (ie, Pharmaceutical Care Skills IV).¹ Its integration into this laboratory course both provided students with the opportunity to improve their documentation skills and helped identify areas that commonly challenged students as they worked to develop their documentation and clinical thinking skills. One of the conclusions of the original work was that the IMC system would need to be incorporated longitudinally into the students' coursework beyond the Pharmaceutical Care Skills IV course to provide ample opportunities to practice and develop their skills.

DESIGN

Pharmacotherapy III is the third course in a 4-course pharmacotherapy sequence (20 credits total). This course is offered in the fall semester of the third-professional year (one semester after the Pharmaceutical Care Skills IV course). Pharmacotherapy III is organized into 4 modules of approximately equal length, encompassing the areas of geriatric, psychiatric, neurologic, and pulmonary pharmacotherapy. At this point in the curriculum, students have already completed 50% of the pharmacotherapy sequence and are therefore ready to evaluate more complicated patients with a variety of therapeutic issues. Pharmacotherapy III was a predominantly lecture-based course with a few case-based learning opportunities built into the modules (historically these cases did not cross content modules and did not hold students individually accountable for their clinical thought process). Considering Miller's taxonomy, the course focused on “knows” and “knows how” levels of learning and assessment, but did little to build in the “shows how” levels.² Consequently, faculty members from the Pharmacotherapy III course elected to add simulated patient cases to the course content using the IMC system's “finish the SOAP note” feature, an activity that provides students with the subjective and objective components of a note and asks them to complete the assessment and plan for a simulated patient case.¹

Four “finish the SOAP note” activities were added to the Pharmacotherapy III course. One SOAP note activity occurred in each of the 4 modules of the course. Each SOAP note focused on the disease state covered in that module, but also included conditions and problems related to previous pharmacotherapy coursework.

The core objectives of the SOAP note exercises were the same regardless of whether they occurred during the laboratory course or the pharmacotherapy course. However, with the addition of the IMC system, specific objectives beyond those for the laboratory were added for the Pharmacotherapy III course:

• (1) Build learning opportunities in Pharmacotherapy III through cases that resemble “real-life” scenarios, including patients with multiple medical problems, patients with multiple medications, and patients with medication-induced problems;

• (2) Build an active-learning strategy that holds individual students accountable for their clinical decision-making; and

• (3) Build an active-learning strategy that includes writing as the form of communication to allow for inclusion of brief rationale and not simply the selection of a correct response

The disease states and number of problems covered by the 4 activities are shown in Table 1. The number of comorbid conditions was greater than students had been previously exposed to in the teaching laboratory. Simulated patients in these exercises had multiple problems (including medication-induced problems) and were taking multiple medications. Students were given approximately 1 week to complete each note exercise. Each student note received one rating for the submitted assessment and a separate rating for the submitted plan using the rubric developed in the teaching laboratory and previously published.¹ Because the Pharmacotherapy III course was traditionally point-based grading, the ratings were then converted to a numerical score with 100% of the available points for an excellent (EX) rating, 85% for a satisfactory (SA) rating, and 70% for a needs improvement (NI) rating. One additional category, markedly lacking detail (0%), was reserved for students whose notes suggested they put little or no effort into the exercise. Evaluators were instructed to provide comments explaining ratings. Students typically received grades and feedback within 1 week of completing the activity. Note the review and evaluation workload was divided equally among 9 evaluators. For consistency, the same evaluator reviewed all 4 of a student's notes.

Table 1.

Simulated Patient Care Activities Performed by Pharmacy Students and Documented Using an Internet-based Medical Chart

^*Second time student working with same simulated patient

^†Third time student working with same simulated patient Complexity = types of conditions and number of problems

CAD = coronary artery disease; HTN = hypertension; GERD = gastroesophageal reflux disease

Similar to the teaching laboratory, evaluators in Pharmacotherapy III accessed the scoring rubric and students' work using the IMC system's evaluation pages. Prior to the start of the semester, evaluators were provided with an e-mail that contained all the information necessary to access the system to evaluate student submissions. Evaluators met as a group before grading the first activity and also met following each SOAP note activity. These meetings focused on reviewing and discussing (1) any issues that arose, (2) approaches to ensuring an answer key that clearly addressed the most common responses that students submitted, (3) appropriate use of the grading rubric, and (4) acceptable approaches to grading notes that did not perfectly match the answer keys. In addition, grading or other issues encountered during the semester were addressed via e-mail.

At the beginning of the semester, students were given a brief instruction sheet on how to access the IMC system. Since students had previously used the IMC system in the teaching laboratory, they required only minimal additional training. Because the students were using the system through Internet connections that were largely off campus, any technical issues were handled predominantly via electronic communications.

Students were surveyed as part of the usual end-of-course evaluation process. As with other evaluated components of the course, students could choose whether or not to participate in the survey. The survey instrument contained 6 statements that students were asked to assess on a 5-point scale ranging from strongly disagree to strongly agree (Table 2).

Table 2.

Pharmacy Students Responses Regarding Use of an Internet-based Medical Chart System to Complete Simulated Patient Care Activities in a Pharmacotherapy Course

^*Responses were rated using a Likert scale on which 1 = strongly disagree and 5 = strongly agree

SOAP = subjective, objective, assessment, and plan

A separate survey instrument was distributed to the 9 evaluators from Pharmacotherapy III. The survey instrument contained 6 questions corresponding to the questions the students were asked (Table 3). As with the students' survey instrument, evaluators were asked to indicate the extent to which they agreed with each of the statements provided on a 5-point scale ranging from strongly disagree to strongly agree. Additionally, evaluators were asked to estimate the amount of time spent on grading each set of notes and to describe specific benefits and challenges of incorporating this active-learning strategy.

Table 3.

Survey Responses of Evaluators Regarding Use of an Internet-based Medical Chart System to Evaluate SOAP Notes Submitted by Students Enrolled in a Pharmacotherapy Course

SOAP = subjective, objective, assessment, and plan

^*Responses were rated using a Likert scale on which 1 = strongly disagree and 5 = strongly agree

RESULTS

A total of 628 student notes were entered into the IMC system by 158 students in the pharmacotherapy course. Student performance on the SOAP note activities is summarized in Table 4. On the first note, approximately half of the students received at least one NI rating on the assessment or plan, while about a fifth received at least 1 EX rating and no NI ratings. As students progressed through the remaining 3 notes in the pharmacotherapy course, the percentage of students receiving no NI ratings and at least one EX rating increased and the percentage receiving one or more NI ratings decreased. By the end of the pharmacotherapy course, approximately a fourth of students received any NI ratings on their notes and more than half received at least one EX rating.

Table 4.

Overall Student Performance on SOAP Notes^* in a Pharmacotherapy Course

SOAP = subjective, objective, assessment, and plan; NI = needs improvement; EX = excellent; SA = satisfactory

^*Each student was responsible for submitting his/her own note for each activity

^† p < 0.001 vs activity 1 using paired t test statistic

^‡ p < 0.001 vs activity 1 using McNemar test statistic for paired categorical data

Two hundred fifty-nine (41.2%) of the 628 SOAP notes received at least one NI rating. Evaluator comments were reviewed to identify the students' error(s)/omission(s) that resulted in the assignment of NI ratings. Students made 573 errors/omissions resulting in 331 NI ratings on 259 notes. Only 3 types of errors occurred more than 5% of the time, and those are shown in Table 5.

Table 5.

Students' Errors/Omissions Contributing to a Needs Improvement (NI) Rating^*

^*There were a total of 573 errors/omissions on 331 components rated as NI in 259 notes receiving at least 1 NI. Mean 1.73 errors/omissions per NI rating

Table 6 summarizes the number of students who received NI ratings on the assessment and plan section of any of the first 3 notes and then reports the number and percent of these students who avoided making the same specific error on all subsequent notes and on the last note of the semester. Approximately two thirds of students avoided making the same mistake twice. The percentage of students repeating the same mistakes by the end of the semester was low, with only 5 (4.7%) and 11 (10.7%) students repeating their previous mistakes in the assessment and plan sections, respectively. Overall, only 12.8% of students (16 of the 125) who made one or more errors resulting in an NI rating repeated their mistake at the end of the semester.

Table 6.

Evidence of Pharmacy Students Learning From Their Mistakes as a Result of Using an Internet-based Medical Chart System to Review Errors Made on SOAP Notes for Simulated Patient Care Activities (N = 158^*)

^*Total number of students was 158. All notes were “finish the note” activities. The background and subjective/objective information was provided to the students

A total of 124 evaluations were returned out of 158 students for a response rate of 78%. The questions and the respective students' responses are shown in Table 2. The students found the system easy to use. The vast majority of students agreed or strongly agreed that the exercises facilitated through the IMC system improved their pharmacotherapy knowledge (86.3%) and documentation skills (78.2%), which is consistent with the student performance data and the experience from the initial use of the IMC system.¹ The lowest ratings from students came on the timeliness of feedback and usefulness of feedback, with 46.0% and 43.5% agreeing or strongly agreeing that it was timely and useful, respectively. Several students who disagreed or strongly disagreed with the item regarding timeliness and feedback voiced concerns that the feedback for the fourth note was interrupted by the Thanksgiving holiday.

All evaluators for the course completed the survey instrument. On average, evaluators indicated that they spent 3.0 (+1.0) hours evaluating approximately 20 notes (approximately 9 minutes per graded note). Evaluator ratings and distribution of ratings for each of the questions are shown in Table 3.

Overall, evaluators found the system easy to use and felt the exercises fostered improvement in both pharmacotherapy knowledge and documentation skills. Only 1 response of disagree or strongly disagree was assigned for any of the evaluation questions. The respondent in that instance assigned a disagree rating for ease of use (this evaluator's comments on the survey instrument suggested that the concern was with the rubric's structure and not the technology itself). Other than this single rating, evaluators overwhelmingly indicated that the exercises were useful and all but one either agreed or strongly agreed to serve as an evaluator in the future.

Common benefits of the SOAP note activities cited by evaluators included (1) exposing students to cases closer resembling those in practice, (2) requiring students to think clinically about a patient, and (3) requiring students to document their thought processes and decisions. Several evaluators felt that these exercises would better prepare students for their advanced pharmacy practice experiences. Challenges of the system cited by the evaluators included a concern that the grading key did not (and could not) include every possible correct answer.

DISCUSSION

The IMC system was innovative in 3 ways. First, the IMC system was designed to meet several diverse objectives in one integrated system. It had to look like an electronic medical chart for students, but at the same time predictably control students' access to notes stratified by student author and simulated time. The system had to facilitate evaluation of student work and delivery of feedback. It had to archive simulated provider data and student data so that students could review their past notes and performance. The system succeeded in all of these areas, providing an educational tool that meets the logistical and administrative challenges inherent in providing individualized opportunities to apply therapeutic knowledge and document therapeutic decision-making thought processes.

The IMC system is also innovative as an electronic archive of student notes and evaluator feedback, making it possible to efficiently review evaluators' comments and identify areas that consistently challenged students and required improvement. We now know that almost 20% of the time our pharmacy students struggle to properly or completely identify 1 or more clinical problems. We know that about 15% of students struggle to define what follow-up laboratory tests and procedures should be ordered at the completion of a patient visit. We also know that students learn from the feedback they receive, which is extremely gratifying to see. Before the IMC system, we might have spoken qualitatively of areas we “felt” represented the biggest challenges to students. Because of this innovation, we can now speak quantitatively of the areas that challenge students.

The third way in which the IMC system is innovative is its portability across the curriculum. Although it was developed for the teaching laboratory course, it has proven to be a flexible tool that could be integrated into a pharmacotherapy course to promote active learning and knowledge application at the “shows how” level.² Over time, the IMC system will allow us to understand our students' clinical documentation development across courses that utilize this educational and assessment tool.

The ability to integrate clinical writing opportunities into an existing course while minimizing the constraints of large class sizes and multiple campuses has wide application. The IMC system requires both Macromedia ColdFusion and Microsoft Access to function. Fortunately, these components are available at many institutions. Even at institutions without access to a ColdFusion server, implementation of the IMC system would be quite practical – it could be administered remotely on any ColdFusion server. The administrative functionality has an Internet-based interface (no ColdFusion or Access programming knowledge was required for the incorporation of the base system into Pharmacotherapy III).

The Internet-based nature of the IMC allows a large degree of flexibility and access for both learners and evaluators. Extending the IMC from its teaching laboratory environment directly to another course within the curriculum provides significant synergy from the learner's perspective. Minimal time is expended teaching learners how to use and implement the documentation process. Internet accessibility also has the potential to facilitate use of evaluators located at sites remote from the educational facility. Conceptually, this functionality would allow implementation of the IMC system by colleges and schools that are heavily dependent on adjunct faculty members. Additionally, the IMC system could be used as a tool to engage clinical preceptors in didactic courses.

Cases and case-based learning have always been a part of classroom lectures and continue to be an integral part of teaching pharmacotherapy topics. However, it is difficult for the instructor to objectively assess each student's understanding of the patient case and patient care process in this setting. Also, in-class discussions do not provide an efficient mechanism for assessing students' ability to assimilate clinical information and incorporate it into the patient chart. The benefit of the IMC system is that each individual student is required to assimilate learned material, prioritize treatment recommendations, and organize written clinical communication.

The greatest challenge we encountered was devoting the time required for development of a system that was unproven and untested. The time investment has been well worth the effort, simply because of the time saved through the efficiencies of triaging students' work and evaluators' feedback. Furthermore, the database structure has proven robust and has required little manipulation since its development. It now serves as an efficient repository of cases, further providing a return on the time investment.

The expansion of the system's use to Pharmacotherapy III presented a unique challenge. In the teaching laboratory, all of the teaching laboratory documentation activities were done as part of the “in-lab” activities. In Pharmacotherapy III, students' interactions with the IMC system were completely off-campus. Connectivity was not an issue but on rare occasion students forgot to electronically sign notes, causing temporary confusion but no loss of students' notes.

In future course offerings that include use of the IMC system, the number one priority will be to make educational changes that help students avoid the errors and omissions that result in NI ratings. Table 5 describes several areas that need educational attention. Knowing that these errors/omissions exist is very important, but they need to be corrected to maximize the influence of the innovation. Some of these errors/omissions demonstrate clinical deficiencies, which will likely improve throughout the remaining 1½ years of instruction. Perhaps of greater concern are the deficiencies reflected by missing information. In either case, faculty members will continue to emphasize these challenging areas in learning activities within and outside of the IMC system.

In conversations with several students, we learned that they would appreciate more comprehensive debriefing opportunities on the cases, particularly in Pharmacotherapy III. Even when students received EX ratings for both their assessment and plan, they felt they could gain more from a discussion of the nuances of the case. One of the Pharmacotherapy III faculty members did try to debrief students in a follow-up lecture. It went well but the course schedule was not conducive to incorporating this across all cases in the last offering. This needs to be strongly considered when future course offerings are developed.

The evaluation process does require effort. Thus far, we have relied on faculty members and residents at the respective local campuses to serve as evaluators. The Internet-based nature of the system clearly would allow for practitioners across the State who already serve the College as preceptors and expert clinicians to also share their expertise and insight as evaluators of student documentation.

CONCLUSION

The IMC system is a unique and innovative tool which allowed us to efficiently incorporate clinical writing activities across the curriculum. The system proved easy to use and effective in identifying areas where students were challenged when writing health record documentation. Future work will focus on further incorporating this innovative system across the curriculum and assessing longitudinal development of students' clinical writing skills.

Appendix 1. Student view of a simulated patient chart in the Internet-based Medical Chart System