Achieving Success Connecting Academic and Practicing Clinicians Through Telemedicine

Abstract

Objective

Practicing clinicians, especially in rural areas, are often isolated from learning opportunities and interaction with subspecialty providers. The Pediatric Physician Learning and Collaborative Education (Peds PLACE), an interactive educational telemedicine program, was developed to address this need. We evaluated the success of this program through surveys with practicing and academic physicians.

Methods

Peds PLACE was assessed using two evaluation forms collected from October 2007 to May 2008. One of them was completed by 197 attendees from the University of Arkansas for Medical Sciences (UAMS) and 172 from remote sites. Another form was completed by 131 participants from Arkansas Children's Hospital (ACH), an academic free standing children's hospital. Both evaluation forms asked participants to use a 5-point Likert scale to rank a number of criteria and included a section for participants to write comments and recommendations. Additional data was collected through an open-response email survey of participants.

Results

95% of the participants agreed that the presentations related to their professional needs, 98% agreed that it increased their subject matter knowledge, 81% evaluated the presentations as some of the best they have attended, and 93% agreed that the information would translate into professional practice, enhancing patient care. Health personnel from UAMS evaluated the presentations significantly higher than remote participants. Nursing staff evaluated the presentations significantly higher than medical staff. Comments were generally positive and correlated with the Likert scale data.

Conclusion

Participants reported being highly satisfied with Peds PLACE and considered it an effective way to address the continuing education needs of practitioners throughout Arkansas, especially in rural and underserved areas.

Rural areas are often medically underserved and represent about 20% of the U.S. population.¹ In Arkansas, 61 of 75 counties are rural and are designated as medically underserved areas.² Telemedicine holds considerable promise in alleviating many constraints imposed by time and distance on subspecialty health care encounters.³ As a consequence, services that only took place face-to-face in the past are moving out into the virtual world.⁴

Telemedicine can be used to provide subspecialty care for many health conditions in rural areas.⁵ For example, it can be used to provide real time clinical consultations between geographically dispersed physicians.^6,7 It can also be used to recruit patients and disseminate findings from translational research.^8-15

Finally, videoconferencing is used to deliver education, which is also known as e-learning. It is becoming widely used for continuing professional development of physicians, nurses, and allied health personnel.^16-20 E-learning is usually targeted at an adult learner group, which is different from traditional, school-age students.^21-23 Studies in both the medical and non-medical literature have consistently demonstrated that students are very satisfied with e-learning.²⁴ In general, online professional development is usually as good as, if not better than, traditional instructor-led methods such as lectures in contributing to demonstrated learning.^25,26

Peds PLACE was established in 2007 with 5 main goals: (1) support practitioners in rural areas of Arkansas; (2) build relationships between these practitioners and pediatric specialists at UAMS and ACH; (3) learn from the experience of these practitioners; (4) translate current research findings into useful advice and community health improvement; and (5) improve the health care of medically underserved children in Arkansas. This study is a response to the recommendations of leading experts to explore new and innovative delivery modes for educating medical practitioners and providing continuing medical education.²⁷ Second, user perceptions and attitudes toward the use of telemedicine have been identified in the literature as an area in need of further research.⁴ The purpose of this study was to assess the success of Peds PLACE and to elicit suggestions from participants for improvement. We sought to address whether or not participants were satisfied with the use of telemedicine as a delivery method of continuing medical education and to garner suggestions about how to make Peds PLACE more effective.

Methods

Subjects and Design

The telemedicine network operated by the Center for Distance Health (CDH) at UAMS facilitates connections to approximately 150 end points (video units). In addition the network directly links to the Arkansas Department of Health Bioterrorism Network located in all 87 hospitals in Arkansas and 43 county health units. The connections made through both networks are T1 communication lines capable of transmitting up to 1.5 mbps. This connectivity allows for real-time consultations, educational conferences and administrative meetings.

Each week faculty physicians from ACH and UAMS address multiple common pediatric problems using discussion, PowerPoint slides, and interactive methodologies (Table 1). More than 80 towns and cities are wired and capable of receiving Peds PLACE training but only 30 of these sites connect to Peds PLACE on a regular basis.

Table 1. Peds PLACE presentations in chronological order.

Peds PLACE presentations October 2007 to May 2008	UAMS/Remote participants	ACH participants
Anti-viral studies	22	2
Swallowing dysfunctions	25	6
All-terrain vehicle pediatric injuries in Arkansas	9	5
Intraventricular hemorrhage in VLBW infants	9	11
Newborn blood screening	18	3
Sexual abuse exams in young children	5	2
New recommendations in childhood obesity	18	4
Risk and protective factors for birth defects	21	4
Lead poisoning	15	2
New therapies for food allergies	18	2
Update on genetic testing 2008	16	13
Headaches	7	4
Newborn genetic screening	11	2
Gastroesophageal reflux disease in infants	15	5
Use of nasal cannula flow vs. nasal CPAP	31	7
Management and treatment of phenylketonuria	24	15
Otitis media and otitis media with effusion	13	3
Tubing and line access (UAC/UVC/PIC/Line)	12	8
Car seat testing neonatal guidelines	17	2
Primary care asthma	11	11
Oppositional defiant toddler	8	2
Maintenance of AAP certification	4	5
Nutrition for premature newborns after discharge	21	5
Enteral tube feeding	8	4
Cardiac murmurs	11	4
TOTAL	369	131

The sample consisted of all participants who completed evaluation forms for the 25 Peds PLACE scheduled presentations between October 2007 and May 2008. One of the evaluation forms, labeled UAMS/Remote, was filled out by 197 attendees from UAMS and 172 from the remote sites. The number of participants for each presentation varied from 4 to 31 (mean = 16). The other evaluation form was filled out by 131 participants at ACH. The number of ACH participants for each presentation varied from 2 to 15 (mean = 5).

Survey Instrument

The UAMS/Remote evaluation forms asked for demographic data, discipline, site location, and included a 5-point Likert scale with twelve standard evaluation criteria. In addition, a comments section (strengths, limitations, suggestions for improvement, etc.) was available. The Cronback's reliability coefficient of this evaluation form was measured at 0.948.

The ACH evaluation forms were completed online and asked for type of evaluator (faculty physician, resident physician, fellow, nursing, and other). It also included a 5-point Likert scale with five standard evaluation criteria. Items regarding their understanding of the subject discussed and how this presentation compared with others were included. Using an open response section participants were asked how could the lecture have been more helpful. The Cronback's reliability coefficient of this evaluation form was measured at 0.903.

In addition, a survey with 5 open-response questions was sent electronically to past attendees. This survey asked to list the main strengths of Peds PLACE, its limitations, and recommendations to improve it.

Statistical Methods

The data was analyzed with SPSS, using mostly descriptive statistics appropriate for the sample size and the type of data collected. ANOVA with Dunnett post-hoc test was used to discover differences in responses by groups of interest. The qualitative data (comments) were classified into several main types for analysis, such as positive or negative comments, comments aimed at the material presented or at the speaker, and comments on the technology used, among others.

Results

Participant Characteristics

A total of 500 participants completed the evaluation forms. Of those, 66% were females and 34% were male. The participants were overwhelmingly Caucasian (78.5%), with African-Americans (8.1%) and Asian/Pacific Islanders (4.9%) as the second and third largest group represented, respectively.

Physicians (50%) and nurses (26%) were the primary attendees of Peds PLACE. Other health disciplines represented included medical students, advanced practice nursing, allied health disciplines, administration, dietitians, dental, respiratory therapy, pharmacy, and social work.

Participants were able to access Peds PLACE from multiple locations throughout the state. Most of the participants used UAMS (39%) and remote (34%) teleconference facilities. Some of the remote sites with higher participation included Crittenden Memorial Hospital at West Memphis (34; 18%), Great River Medical Center at Blytheville (22; 11%), Conway Human Development Center (19; 10%), Mena Medical Center (17; 9%), and the Arkansas Area Health Education Centers at El Dorado (15; 8%).

UAMS/Remote Survey

Participants reported being highly satisfied with the professional development opportunity that Peds PLACE provided. Ninety-five percent agreed that the presentations related to their professional needs, 98% agreed that the presentations increased their subject matter knowledge, 94% agreed that the presentations met their professional expectations, and 97% agreed that a 1-hour presentation was an appropriate length.

In addition, 97% strongly agreed or agreed that the material was well presented, 98% strongly agreed or agreed that the professional resource leading each Peds PLACE presentation was knowledgeable, 99% strongly agreed or agreed that the presenter made time to address the participants' questions during and after the presentation, and 93% strongly agreed or agreed that the information presented through Peds PLACE will translate into professional practice and will enhance patient care. In terms of the telemedicine technology used for the presentations, 95% strongly agreed or agreed that it was satisfactory and 94% strongly agreed or agreed that it was as effective as traditional methods. More than 95% of the participants reported that they will be very likely or likely to take other professional development courses using the current technology and that it was the most convenient way to obtain professional development training (Table 2).

Table 2. Responses from UAMS/Remote Evaluation Form.

Evaluation Criteria	SA	A	U	D	SD
Presentation related to my needs.	64.3%	30.7%	4.1%	1.0%	0
Presentation increased my knowledge.	69.8%	27.0%	2.7%	<1%	0
Presentation met my expectations.	67.1%	27.0%	4.7%	1.0%	<1%
Length of the presentation was appropriate.	68.9%	28.1%	2.3%	<1%	0
Material was well presented.	72.3%	25.0%	2.5%	<1%	0
Presenter was knowledgeable.	79.5%	19.7%	<1%	<1%	0
Presenter made time for questions.	77.8%	20.9%	1.0%	<1%	0
Information will enhance patient care.	69.0%	24.2%	6.0%	<1%	0
This technology was satisfactory.	65.2%	29.3%	4.3%	1.0%	0
The technology was as effective as traditional methods.	65.2%	28.4%	4.0%	1.7%	<1%
I would take other courses with this technology.	73.5%	23.6%	2.3%	<1%	<1%
This technology is the most convenient way for me to obtain this training.	71.2%	23.8%	3.8%	1.0%	0

SA = strongly agree, A = agree, U = undecided, D = disagree, and SD = strongly disagree.

The survey responses were statistically similar when examined by gender and by race/ethnicity. However, it was found that in 9 of 12 evaluation criteria, participants from UAMS gave significantly higher scores to the presentations compared with participants from the remote sites (Table 3).

Table 3. Comparison of Average Ranks for UAMS and Remote participants.

Evaluation Criteria	UAMS	Remote	F	p
Presentation related to my needs.*	4.68	4.47	12.9	0.000
Presentation increased my knowledge.*	4.73	4.59	7.00	0.008
Presentation met my expectations.*	4.67	4.52	5.32	0.022
Length of the presentation was appropriate.	4.69	4.61	1.81	0.179
Material was well presented.*	4.75	4.63	5.70	0.017
Presenter was knowledgeable.*	4.84	4.72	7.45	0.007
Presenter made time for questions.*	4.82	4.70	6.77	0.010
Information will enhance patient care.*	4.70	4.51	8.73	0.003
This technology was satisfactory.*	4.65	4.51	4.12	0.043
The technology was as effective as traditional methods.*	4.65	4.46	6.07	0.014
I would take other courses with this technology.	4.69	4.71	0.10	0.752
This technology is the most convenient way for me to obtain this training.	4.60	4.71	3.08	0.080

Similarly, in 7 out of 12 evaluation criteria, nurses and nursing staff gave significantly higher scores to the presentations compared with physicians/medical staff as measured by ANOVA's Dunnett post-hoc test (Table 4).

Table 4. Comparison of Average Ranks for nurses and physicians.

Evaluation Criteria	Nurses	M.D.	Diff.	p
Presentation related to my needs.	4.61	4.61	0.003	1.000
Presentation increased my knowledge.*	4.79	4.54	0.244	0.000
Presentation met my expectations.*	4.75	4.50	0.243	0.001
Length of the presentation was appropriate.	4.74	4.59	0.156	0.056
Material was well presented.*	4.83	4.61	0.220	0.000
Presenter was knowledgeable.*	4.88	4.72	0.160	0.002
Presenter made time for questions.*	4.84	4.71	0.127	0.049
Information will enhance patient care.*	4.78	4.57	0.205	0.006
This technology was satisfactory.	4.60	4.55	0.046	0.926
The technology was as effective as traditional methods.	4.66	4.46	0.192	0.071
I would take other courses with this technology.	4.77	4.66	0.114	0.236
This technology is the most convenient way for me to obtain this training.*	4.78	4.58	0.206	0.010

The top five Peds PLACE topics were otitis media, childhood obesity, enteral tube feeding, phenylketonuria, and birth defects. The topics that were not evaluated as positively were maintenance of AAP certification, newborn genetic screening, intraventricular hemorrhage, pediatric tubing and line access, and swallowing disorders. The less popular topics were evaluated as 4.0 out of 5 in the Likert scale.

A plurality of the written comments (31%) praised the knowledge, preparation, professionalism, or speaking skills of the presenter. Comments such as “excellent knowledge base”, “well informed”, and “presenter was easy to understand” were commonplace. In particular, a few attendees pointed out how some of the topics were presented in a manner consistent with their applicability to active practice.

Another frequent category of comments (28%) praised the topics selected, especially its level of currency, relevance to practitioners, usefulness, practicality, and comprehensiveness. Comments such as “current information”, “exciting study”, “excellent review”, and “good information” were representative.

Not all comments were positive. About 14% of the comments were critical of some aspect of the technology during the presentations, including background noise, presenters could not be heard well, and connectivity issues. Another 14% of the comments provided recommendations for the speaker. Examples included preparing a reading list, providing handouts, increasing text and diagram size on slides to improve readability, reducing the lecture format eliciting interaction, and allotting more time to answer audience questions.

ACH Survey

In the ACH evaluation form, participants also reported being highly satisfied with the professional development opportunity that Peds PLACE provided. According to their evaluation responses, 98% agreed that the presentations contained clear language and terminology, 97% agreed that the audiovisual materials were legible and supported the presentation, 99% agreed that the presentations were carried out in a professional manner, 98% agreed that the presentations were well organized, and 98% agreed that the participants learned something new from attending the presentations. In addition, 98% of the participants reported understanding all or most of the concepts presented and 86% evaluated the presentations as some of the best they have attended (Table 5).

Table 5. Responses from ACH Evaluation Form.

Evaluation Criteria	SA	A	U	D	SD
The presentations contained clear language and terminology.	79.3%	18.6%	2.1%	0	0
The audiovisual materials were legible and supported the presentation.	77.1%	20.0%	2.9%	0	0
The presentations were carried out in a professional manner.	86.4%	12.9%	<1%	0	0
The presentations were well organized.	77.1%	20.7%	2.1%	0	0
[The attendees] learned something new from the presentations.	78.6%	19.3%	1.4%	0	0

SA = strongly agree, A = agree, U = undecided, D = disagree, and SD = strongly disagree.

Faculty physicians and resident physicians evaluated Peds PLACE in a similar fashion, except for one question in which, not surprisingly, faculty physicians ranked their understanding of the material higher compared with resident physicians (F = 7.85, p = 0.001).

Overall, the top five topics for this group of participants were food allergies, use of nasal cannula flow and nasal CPAP, cardiac murmurs, anti-viral studies, all-terrain vehicles, and sexual abuse exams. The less popular topics were swallowing disorders, newborn genetic screening, newborn blood screening, lead poisoning, and birth defects.

Most of the comments (42%) praised the knowledge, preparation, professionalism or speaking skills of the speaker, and 32% of the comments praised the topics selected. About 15% of the comments provided recommendations for the speaker, including tailoring the lecture to a teleconferencing format; focusing more on practice and treatments; new screening technologies; discussing more cases and their application; and avoiding tangential or unrelated discussions. The rest of the comments were critical of the small font size of the PowerPoint slides and the infrequency (and limited applicability) of some of the conditions discussed, such as phenylketonuria.

Electronic survey

Participants considered the convenient access to Peds PLACE training as one of its greatest strengths. Peds PLACE was also considered a good way to share information, meet practitioners and primary care providers from a variety of disciplines from all over the state, and learn in an informal atmosphere. Participants also praised its interactivity, currency, relevance, and conciseness.

One of the main limitations cited by participants was work, meeting, and other scheduling conflicts. Other limitations included audio problems, background noise, and visual images that did not synchronize with the audio. A few participants mentioned that the technology was intimidating and prevented them from asking questions or making comments. They suggested the creation of a post-session online forum or a listserv for presenters and participants to follow-up and share knowledge and experiences in a less intimidating environment. This forum also has the advantage of being asynchronous, an asset for those with scheduling conflicts.

Discussion

Both the Likert-scale and the comments data suggest that Peds PLACE is reaching a variety of health care personnel, especially physicians and nurses, in rural towns and cities. It is essential to provide medical information that can be used in rural and underserved areas of the state. For example, Singh has suggested that neonatal and infant mortality are affected by health disparities.²⁸ Telecommunication is one way of eliminating disparities in rural states. The high degree of satisfaction from the interactive format is consistent with previous findings and is more conducive to learning and actually retaining the material.^29,30

Most of the participants, regardless of gender and race, have a very positive impression of Peds PLACE. Positive remarks included the depth of the information, how it could be applied in practice, and the quality of the presenters, almost all of whom were experienced faculty at UAMS or ACH. Participants also expressed their support for the telemedicine aspect of Peds PLACE and the appropriateness of the current technology used.

It was interesting to find that nurses and physicians evaluated Peds PLACE presentations differently, with physicians evaluating them slightly less favorably. This suggests that these might be two different groups with different background, experiences and knowledge. Apparently, physicians might be more familiar with the topics compared with nurses. In fact, some comments suggested that Peds PLACE should not be “medical school all over again”, but that the emphasis should be on translational research. On the other hand, some nurses found the presentations “over their heads”, too fast-paced and abstract. A similar case could be made when participants from UAMS were compared with those from remote sites, who evaluated the presentations slightly lower than people from UAMS. It could be argued that some of the Peds PLACE presentations might not be fully addressing the needs of rural practitioners.

Most of the written comments were positive towards both the topics and the presenters at Peds PLACE. Many participants considered them excellent and relevant, which is consistent with the quantitative data. A number of participants mentioned technical difficulties and the use of PowerPoint slides overwhelmed with too much information and small text size. This last point is not exclusive of telemedicine training. Most faculty who have not been trained in education might make the mistake of equating more information per slide with more learning.³¹

Although participants were very satisfied with the quality of training offered through Peds PLACE, they offered important recommendations that can be easily implemented in this and similar continuing education programs that rely on telemedicine:

1. Strengthen efforts to explore the specific knowledge and training needs of physicians and nurses, two distinct groups with obvious differences that might not be fully addressed with the current approach. The same applies to participants outside of UAMS, who reacted differently to Peds PLACE compared with UAMS participants.

2. Increase the interactivity of the presentations by using more clinical scenarios and case studies, opening questions to the audience, and questioning strategies.

3. Avoid lecturing for the whole allotted time. Presentations already prepared for a traditional in-class lecture might not translate adequately to a teleconferencing environment.

4. Insert questions at key points in the presentation to keep the audience's attention and check for understanding.

5. Continue improving the teleconference technologies to avoid interruptions and to increase the quality of the images.

6. Share with future presenters some of the best practices in the creation of PowerPoint slides. A number of readily available, online resources discuss ways to enhance the slides and pitfalls to avoid.

Some limitations of this study must be mentioned. First, the evaluation survey was not completely anonymous. This might have prevented some participants from sharing their true insights in their evaluation. A replication of the study with anonymous follow-up interviews might provide a better perspective on Peds PLACE. Second, the effect of self-selection should not be ignored. Although all the evaluations were included in the analysis, this study still has a convenience sampling. A replication of the study measuring knowledge and translation ability of topics by comparing participants randomly assigned to an online Peds PLACE vs. a classroom-based Peds PLACE should provide new insights on online professional development in the health sciences.

A number of interesting topics for further research can be identified from this study. Replicating the current study after creating and using a single evaluation form for all participants will allow for a better comparison of UAMS, ACH, and remote site participants. The new evaluation could also be used to compare the feedback of people at the presenting site with participants that connect through videoconference. The literature suggests that these groups tend not to evaluate presentations similarly.³² Similar studies using pre- and post- surveys and other higher levels of training evaluation³³ and a mixed-methods research design with participant interview data are proposed. In addition, the integration of research-based learning models to Peds PLACE, such as Active Learning³⁴ and Hyperlearning³⁵ are being considered.

Conclusion

It is clear from the data that participants are very satisfied with Peds PLACE and consider it an effective way to address their continuing education needs. Several recommendations can be implemented to increase interactivity, tailor the topics to the needs of rural pediatricians, family practitioners and other groups, and emphasize the translational aspect of Peds PLACE.