Using Virtual Site Visits in the Clinical Evaluation of Nurse Practitioner Students

Supplemental digital content is available in the text.

Background

The use of telehealth technology to conduct virtual site visits is an innovative strategy for evaluating the performance of nurse practitioner (NP) students in remote settings. Although there is an abundance of studies on telehealth for the remote monitoring and assessment of patients, there are limited data on its use for evaluating NP students during clinical learning experiences.

Purpose

The purpose of this project was to understand the perspectives of NP students and faculty on the feasibility of using virtual site visits to evaluate the students' performance during clinical experiences.

Methods

Online surveys were used to collect student and faculty perspectives on the use of virtual technology during clinical site observations.

Results

Overall, students and faculty reported positive experiences with the virtual site visits.

Conclusion

Virtual site visits are feasible in most clinical settings.

Virtual site visits are an innovative strategy for using telehealth technology to supervise students. A virtual site visit is a secure, synchronous, 2-way, audio-video telehealth software connection between faculty and nurse practitioner (NP) students. The purpose of this project was to determine student and faculty perspectives on the feasibility of using virtual site visits in the clinical observation and evaluation of student performance during clinical experiences.

Telehealth Technology and Supervision of Students in Clinical Settings

The American Association of Colleges of Nursing and National Organization of Nurse Practitioner Faculties (NONPF) mandate periodic faculty observations to evaluate students during clinical learning experiences.¹ Typically, faculty conduct these clinical observations on site and in person. However, it can be a challenge for NP faculty to monitor a large number of students in rural and distant clinical sites. Virtual site visits are an innovative use of technology to remove barriers and improve the efficiency of remote monitoring of student clinical learning experiences.² A literature review yielded 3 studies^3-5 and 2 poster abstracts^5,6 documenting the use of virtual technology in the supervision of graduate students. Of these studies,^3,4 2 were conducted with students in NP programs, the third in a pharmacy department.

Faculty in the University of California, Los Angeles, NP program were among the first to adapt telehealth technology to supervise and evaluate the clinical performance of graduate students.³ Faculty reviewed videotaped interprofessional collaboration between students in the schools of nursing and medicine during patient consultations. Faculty perspectives on the videotaped observations supported the use of technology to verify student assessment skills. Although student surveys (n = 29) showed that telehealth was perceived to broaden opportunities for the diversity of patient panels, using telehealth for patient rounding received the lowest mean scores on the questionnaire. Faculty and students indicated a preference for live clinical experiences compared with video technology.

The second NP-student study took place at a southern university and examined the effectiveness of using virtual site visits (n = 46) in an online NP training program.⁴ The objectives were to decrease cost and faculty travel time when evaluating student clinical experiences. Survey data were collected on students, faculty, and preceptors over 3 semesters. Results showed overall satisfaction and a cost savings of $288.46 per student visit when technology was used.

Virtual and on-site visits were compared in a third study.⁵ Survey data were collected (n = 44) on preceptors and pharmacy students. When compared with on-site visits (n = 28), results showed no significant differences on preceptor perceptions of virtual site (n = 16).

There were 2 abstracts on the NONPF website that described the process for conducting virtual visits and the remote supervision of nursing students. In one project, NP faculty selected equipment, developed guidelines, and trained students and faculty for use in a variety of community settings.⁶ Another project⁷ used high-definition video technology to record and evaluate the clinical performance of NP students during learning experiences. While both projects highlighted the convenience of using technology at distance education sites, neither project reported any outcome data.

To the best of our knowledge, the current literature reports only these 3 studies and 2 abstracts on this emerging field of interest. Our project was conducted to potentially enhance best practices for using telehealth as a modality to evaluate the clinical performance of NP students.

Theoretical Framework

Our project used the Community of Inquiry theoretical framework model as a basis for meaningful learning experiences on social, cognitive, and teaching presence.⁸ Social presence is defined as “the degree to which participants… feel affectively connected to one another.”⁸ To build student-faculty connectedness in a virtual learning experience, it made sense to us that our NP faculty should intentionally emphasize and maintain a high degree of social presence.

Virtual Site Visit Protocol

Students followed a faculty-developed protocol (Table 1, Supplemental Digital Content, http://links.lww.com/NE/A644) that included an audiovisual connection at each remote location that was tested in advance of the scheduled virtual visit. If for some reason the site showed poor connectivity, the student and faculty member enlisted the help of the hosting university's 24/7 information technology support services phone line. Before conducting the live virtual site visit, students ensured that facility, preceptor, and patient permission were obtained.

Then, the mobile device was positioned for faculty to observe the student's clinical performance. Students introduced faculty to the patient through the mobile tablet device. The preceptor and student carried out the remaining visit as they would normally do during a face-to-face setting. At the close of the visit, the students thanked the patients for their time and exited the virtual space.

Methods

Data Collection and Evaluation

After approval from the institutional review board, faculty and students first conducted preliminary testing on the quality of audio and video connections in our students' rural clinic locations. Students practiced signing into the software, checked for connectivity at the site, and then tested each step of the protocol with faculty before the patient encounter. We designed and implemented a voluntary online survey to collect data on student and faculty perspectives on the use of virtual site visits.

All survey participants were recruited from a convenience sample of the same student cohort (n = 19) and faculty volunteers (n = 5) across 2 semesters from an adult-gerontology primary care NP program. As part of course requirements, students received a minimum of 2 clinical observations each semester. The number of site visits varied depending on the number of preceptors selected by students.

Survey questions (Table 2, Supplemental Digital Content, http://links.lww.com/NE/A645) included perceptions on ease of use, communication quality between students and faculty, evaluation of the virtual visit when compared with in-person visits, and overall perspective on the virtual experience. Outcome variables on faculty perspectives were elicited from a similar questionnaire.

Security and Equipment

This project was conducted in conjunction with a well-established telehealth infrastructure at a state academic medical center encompassing more than 400 connections in hospitals, health centers, clinics, and schools and is particularly valuable for those living in rural, underserved areas. The network's expertise in telehealth and telemedicine greatly supported this project.⁹ For this study, we used a telepresence video communication server system provided by the corresponding host university. This encrypted, secure, HIPAA-compliant network included 24/7 technical staff support. This system utilized a meeting application to enable 2-way audio and video connections. An orientation was provided by a nurse expert in technology to help students and faculty learn how to use the software. During this training, each student was issued a mobile tablet device (pre-equipped with the secured meeting software) and learned how to connect to the virtual room. Students were counseled on the contraindications for using social media, texting, and other nonsecure portals to make virtual space connections.

Survey Data

Electronic survey software¹⁰ was used to collect survey data. Our faculty developed the Virtual Site Visit Questionnaire (Table 2, Supplemental Digital Content, http://links.lww.com/NE/A645) to measure individual perceptions of the virtual encounter. The survey was adapted from a questionnaire used by the Antenatal and Neonatal Guidelines, Education, and Learning Systems program. The Virtual Site Visit Questionnaire is a 7-item survey including a 3-point “agree, disagree, I don't know” response format. The voluntary questionnaire was administered to students in November 2017 and May 2018. A similar questionnaire was completed by faculty. Participants were also polled on the number of successful and unsuccessful site visits. Patients and families were not surveyed. Descriptive statistics were used to analyze the data.

Results

Nineteen student surveys were distributed: 15 students (79%) and 2 faculty members completed them. The diversity of those surveyed were 6 minority, including 2 male and 4 African American students. All students and faculty were novice learners without any experience in telehealth technology.

Seven of the 15 students surveyed did not participate in any virtual site visit, and 4 of these students reported this was due largely to facility policies that prohibited participation. Eight of the student respondents completed at least 1 or more than 1 virtual site visit (total virtual site visits completed by 8 students = 15). Of the 15 virtual site visits, 13 were successful and 2 resulted in failed attempts. One of the failed attempts was due to poor connectivity; the other was due to audio difficulties. The failed attempts resulted in traditional, face-to-face clinical evaluations.

Data showed that of the 8 students who participated in virtual site visits, 5 students reported that the virtual visit made it easier to connect with faculty and that the technology was a great way for them to stay in touch and periodically check in with their instructors. Six of the students reported that they could see faculty as well as in-person visits. When asked if they could talk with faculty as well as in person, 3 students agreed, 4 were undecided, and 1 disagreed. There were 3 students who reported that a virtual site visit was as good as an in-person site visit, 1 was undecided, and 4 disagreed. When asked if they would use virtual visits in the future as an NP, 5 agreed, 3 were undecided, and none of the students disagreed. The 2 faculty members who responded to the questionnaire agreed that the virtual site visits were an easy method for connecting with students and were a good alternative to in-person visits for evaluating students.

Students identified poor internet connectivity as a significant barrier to virtual site visits. One student stated, “I was happy to be involved in the virtual site visits. I believe it got easier the more I used it. The only issue I had was that sometimes the voice would be delayed.” Another student reported that “because the iPad had to be connected to the internet, many nursing homes and hospitals have secure Wi-Fi that won't allow you to get on without a password, so that is another barrier.” One other student explained, “The virtual site visit gave me access immediately to my coordinator and instructors to ask questions and to see patients with me during visits.” A less favorable opinion was “I prefer an in-person visit. It has a more personal feel for students and the patients. When long distances (rural sites) are present, it is a great option though.” Faculty reported, “If possible… the faculty member should visit because it helps build relationships.” Overall, students and faculty reported positive experiences with the virtual site visits.

Discussion

Technology is included in the competencies for graduate nursing programs.¹¹ This project supported the feasibility of implementing virtual site visits to evaluate NP students who will provide access to primary care. There is a faculty shortage¹² in NP programs. This is compounded by the shortage of faculty who are trained in gerontological nursing and rural health. The lack of trained faculty greatly strains the availability for direct supervision of students training in rural areas. The benefits of virtual site visits when compared with traditional face-to-face clinical evaluations include decreased faculty travel time and expense. The results of our project were consistent with the literature, showing that virtual site visits may offer support to faculty supervising and mentoring NP students in rural and underserved areas.

Lessons Learned

Our project identified several barriers for incorporating virtual site visits into educational programs including training and literacy, technological difficulties, and maintaining social presence and relationships when using technology. We learned that the benefits of innovative telehealth technology outweigh the barriers and enhance training NP students to care for patients, especially in rural and underserved settings.

In lieu of tablets, smartphones could provide easy access to video conferencing, which could also be used for clinical site visits. The availability of smartphones may be a barrier for some students and faculty; however, smartphones are a common method of communication for most nurses.¹³ The HIPAA-compliant, meeting app and software used in our project could be downloaded at no cost to phones (or tablets), and use could be replicated in other NP programs.

Overall, we found that technology literacy was a barrier to technology use in community settings. Administrators at some facilities declined the virtual site because they did not understand that the software was HIPAA compliant. Although we did not implement a formal educational program for the clinical facilities, future education on telehealth and telehealth security of patient information through HIPAA-compliant software may alleviate these misconceptions and restrictions. It might be helpful to rename virtual site visits to telehealth site visits to convey a more accurate intent of the interactions among patients, students, faculty, and preceptors during the clinical experience.

Training faculty, students, and preceptors at the clinical sites is vital to the proficient use of innovative technology in NP programs. There are several factors to consider for a successful virtual site visit. Testing the technology in advance of patient engagement in the virtual site visit should be a part of the training. The setting should be considered, and permission from the patient and family should be obtained. Prior to patient arrival, connectivity and audio should be tested. Lighting should be adjusted to avoid glare or the silhouette transmission of the persons in the room. The mobile device should be placed strategically so faculty can visualize the student-patient interaction for evaluation. A faculty-developed protocol (Table 1, Supplemental Digital Content, http://links.lww.com/NE/A644) was useful in our project.

We learned that connectivity was an important barrier to telehealth. Network security in a hospital or university setting presented challenges with connectivity to the iPad. Many nursing homes and rural clinical settings did not have internet Wi-Fi access that would accommodate a virtual site visit. If there was Wi-Fi, bandwidth to sustain a connection was poor. We used mobile phone hot spots to overcome connectivity issues.

To achieve a successful site visit, it is helpful for faculty to engage in at least 1 traditional face-to-face visit to build a relationship with the preceptor, test connectivity, and explain the procedures for the virtual site visit. Increasing face-to-face interaction with faculty may improve preceptor engagement. In our project, we found this traditional initial visit established relationships and improved the feasibility of using technology in the community setting. Subsequent visits successfully used telehealth to achieve the same goals as in-person traditional site visits for evaluating individual student clinical performance.

Limitations

This project is limited by the small sample size from convenience sampling of 1 cohort of students at 1 institution in 1 southern state. Large, randomized trials are needed to eliminate bias. In addition, patient perspectives on the virtual site visits should be collected and analyzed. Preceptors were invited, but there was limited engagement and no completed surveys from preceptors who participated in a virtual site visit.

Conclusion

There is untapped potential for using telehealth as a viable alternative to traditional in-person, on-site supervision and evaluation of student clinical performance in remote areas. Virtual site visits may not be effective if used exclusively or to replace every traditional clinical evaluation. A hybrid model of traditional on-site and virtual site visits uses the best of both strategies for clinical evaluations.

Best practices in technology are essential to transform nursing education. Virtual site visits may be an optimal compromise for NP programs that have not incorporated traditional face-to-face visits because of time constraints or other barriers, especially in rural settings. Randomized controlled trials across multiple institutions and varied patient populations and settings are needed to test best practices on the evaluation of NP students in clinical learning experiences using technology innovation.