One-on-One Proficiency Training: an Evaluation of Satisfaction and Effectiveness using Clinical Information Systems

Abstract

OBJECTIVE

To examine the effectiveness of a one-on-one training method for advanced proficiency in the use of clinical information systems (CIS) by clinicians (physicians, physician assistants, and nurse practitioners) in a large HMO.

DESIGN

A cross-sectional survey of 129 clinicians.

MEASUREMENTS

Satisfaction was measured using a multi-item satisfaction index. Perceived effectiveness of the training was measured by assessing self-reported improvements in efficiency in CIS.

RESULTS

Response rate of 80%. The one-on-one method was significantly preferred over any other teaching methods (p<.0001). Improvement in use of the electronic medical record was greatest following one-on-one training when compared to other CIS components. Major improvements (i.e., >3 on 5 point Likert scale) in use of the electronic medical record were reported by 61.4% of the clinicians. Overall satisfaction was significantly higher among women (p<.05).

CONCLUSION

The findings support the assumption that one-on-one training is of value to clinicians and that this training modality is valued above other methods.

INTRODUCTION

Clinical Information Systems (CIS) have been used in large managed care systems for many years. ¹ One such HMO (Kaiser Permanente Northwest) has used a CIS in its ambulatory settings since 1994.² CIS programs are complex computer systems that are modified and enhanced over time. Those that are most frequently used in day-to-day practice by Kaiser Permanente Northwest clinicians are an electronic medical record, a reports and lab retrieval system, an internal e-mail system, and an Intranet-based medical library.^1;3 In order to introduce and educate users to the features and operations of CIS programs, training sessions are conducted during the roll-out of such programs, and as part of new employee orientation.⁴ The goal of introductory training is to gain widespread use of a system as quickly as possible. Self-guided computerized training and group settings are typically used.

However, initial introductory training is unlikely to provide for total proficiency in all of the intricacies of a CIS. For this reason organizations often institute ongoing advanced proficiency training after clinicians have become familiar with the basics of the CIS.

Educating and training clinician users of CIS are conducted in a variety of formats or methods. These include classroom settings generally with more than 10 students at a time, small groups involving fewer than ten students, e-learning, (computer-assisted self-paced training), and individual tutoring using one-on-one personal training. All four types of teaching have demonstrated effectiveness in educating and training clinicians. ^5;6

The common goal of advanced training programs is two fold: First they are intended to improve proficiency by the clinician, and second they are intended to increase the clinician’s satisfaction with the system. The first objective will likely lead to improvements in work processes through increases in effectiveness and efficiency. The second objective, increased satisfaction, will likely lead to increased utilization of the CIS, which in turn will support the first objective. Continued satisfaction with CIS use and enhanced skill in the use of the components of the CIS by clinicians are essential to the persistent and optimal utilization and long term success of a CIS. ⁷

Each training method not only has its unique characteristics, but it may also be more or less appropriate to the learning style of clinicians who use the CIS. One step in determining which method is most appropriate is to first assess if the method is considered useful and effective from the perspective of the end-user, in this case the clinician. Preliminary assessment of user satisfaction is an important feature in the evaluation and planning of any learning program. ^8;9 This is particularly true when evaluating the success of training on an aspect that directly affects practice patterns, care delivery, and workflow such as a CIS. Satisfaction surveys are used to assess the strengths and weaknesses of learning programs as well as to assess the value of the training in improving proficiency of CIS use in daily practice.¹⁰ The results of satisfaction surveys can also be used to evaluate teaching methods, as well as discover topics of interest to incorporate in new curricula. ⁶

This paper reports on the results of a cross-sectional survey study that measured clinician satisfaction with one-on-one training in CIS in Kaiser Permanente Northwest Region.

The main research objective was to evaluate the effectiveness of the one-on-one training method for advanced proficiency training of clinicians in use of CIS. Effectiveness was defined as a perceived increase in efficient use of CIS and satisfaction with the one-on-one training sessions.

The aims of the study were to:

1. Assess level of improvement in CIS efficiency following one-on-one training

2. Assess the perceived value of one-on-one training compared to other teaching methods

3. Assess the overall satisfaction with the one-on-one training

METHODS

Setting

Following the initial implementation of the CIS in 1994, additional ongoing training sessions were held utilizing a number of modalities including formal classroom sessions, small groups, and interactive computer e-learning. These sessions were aimed at improving the overall proficiency of clinicians in the use of CIS systems and in teaching the use of new features and functions of CIS as these were added over time. This training covered material relevant to four different primary computerized systems used by clinicians: The electronic medical record, data retrieval results reporting, email, and the intranet-based medical library.

Beginning in 1999 the additional option of one-on-one proficiency training was made available to all clinicians on a self-selection basis. These one-on-one sessions were taught by one of two clinician expert users who were proficient in tutoring clinicians on how best to utilize the applications.

The purpose of the one-on-one sessions was to educate clinicians about features and function of CIS applications with which they were unfamiliar and to increase their skills and ability to use specific features of the CIS to perform relevant tasks which they were either performing inefficiently or were not aware of. The sessions, approved for Continuing Medical Education (CME) credit, included basic and core competency evaluations plus tailored instruction. The training occurred during a single dedicated session lasting between three and four hours. It took place in the clinician’s office at a time separate from patient care. The format was loosely structured allowing for focus on those aspects that were deemed most valuable from the perspective of the trainer and the clinician.

Design

A cross sectional study was conducted by using a self-administered paper-based survey that was sent to all clinicians of the managed care system who had received advanced proficiency one-on-one individualized training in the CIS during the time period between October 1999 and September 2002.

Study Population

A total of 162 clinicians were trained using the one-on-one method by the same two trainers since October 1999. Of these, 5 have left the managed care system. All 162 clinicians were located and were invited to participate in the study.

Survey Instrument

The survey instrument was used to assess clinician satisfaction and perceived value of the training session. The survey was developed and pre-tested in the summer of 2002 and was comprised of a self-administered paper-based questionnaire. A $3.00 gift card to a local coffeehouse was included, as an incentive to complete and return the survey.

Data Analysis

Descriptive statistics were used to report frequency, percentage and means for responses to individual and group questions for all respondents. Spearman’s rho correlation was used to test associations between rank ordered variables. Two-tailed t-tests were used to compare the means of individual, paired, and grouped questions when examining differences between gender, age, and other differentiating categories. ANOVA tests were used when comparing mean scores among multiple non-continuous categorical variables such as age groups.

RESULTS

Demographics

Of the 162 surveys distributed to the target population, 129 were returned for an overall response rate of 79.6%. Among the total sample, 55.8% were women (n=72). The age range was from 31 years to 67 years, with a mean age of 47.4 years.

The average length of time employed with the HMO was 10.3 years and ranged from 1 to 36 years. Fifty-three percent were primary care clinicians. Based on their demographic profiles, the sample population is considered to be generally representative of the total clinician mix in the entire managed care organization.

Effectiveness Of One-on-One Training

Effectiveness was measured by perceived improvements in efficiency in using CIS. The level of improvement was measured on a Likert type scale of 1 to 5, with five representing the highest level of improvement. The highest improvement score was noted for the electronic medical record application (mean = 3.55. see Table 1). Major improvements (score greater than 3) in use of the electronic medical record following one-one-one training were reported by 61.4% of the clinicians.

Table 1.

Level of Improvement in CIS Applications

CIS component	n	Mean (sd)
Medical Record	127	3.55 (.99)
Results Reporting	109	2.85 (1.20)
e-mail	120	3.17 (1.28)
Library	113	3.50 (1.20)

The one-on-one training session was more beneficial to women than men in terms of improved efficiency in the use of the CIS programs (see Table 2). There were no significant differences based on age.

Table 2.

One-on-One Training Analysis by Gender

	Male (n=47)	Female (n=70)
	mean (sd)	mean (sd)	p-value
Effectiveness in CIS from One-on-One Training
Medical Record	3.43(0.95)	3.71(0.90)	.100
Results Reporting	2.66(1.17)	3.03(1.17)	.119
e-mail	2.80(1.36)	3.52(1.06)	.003
Online Library	3.32(1.25)	3.72(1.05)	.079
Effectiveness of Teaching Methods
One-on-One	4.36(0.99)	4.68(0.63)	.036
Small Groups	3.77(0.83)	3.82(0.77)	.784
Class Rooms	2.64(0.93)	2.63(0.87)	.919
e-Learning	2.89(0.95)	3.03(1.17)	.497
Overall Satisfaction	3.67(0.63)	3.91(0.56)	.032

Effectiveness Of Teaching Methods

Respondents reported that the one-on-one sessions were more effective as a teaching method, when compared to small groups, classroom, and interactive computer e-learning. The effectiveness of teaching method was measured using as 5 point Likert type response scale ranging from 1 to 5, with 1 being ‘not effective’ and 5 representing ‘very effective’.

The one-on-one method was significantly more effective when compared to each of the other methods (p<.0001), as well as having a mean score 20% higher than the next highest method – small groups (means 4.51 vs. 3.76. see Tables 3 and 4).

Table 3.

Teaching Method Effectiveness

Method	n	Mean (sd)
One-on-one	125	4.51 (.84)
Small Groups	116	3.76 (.80)
Class Rooms	120	2.60 (.90)
e-Learning	113	2.90 (1.10)

Table 4.

Compare One-on-one to Other Methods

Pairs	Mean(sd)	p-value
One-on-one	4.51 (.85)	<.0001
Small Group	3.76 (.80)
One-on-one	4.51 (.84)	<.0001
Class Room	2.56 (.90)
One-on-one	4.47 (.87)	<.0001
e-Learning	2.90 (1.10)

When compared to men, women reported that the one-on-one training sessions were significantly more effective, but there was no reported difference in effectiveness between genders with the other training methods (see Table 2).

Satisfaction With One-on-One Training

The survey included five questions pertaining to the clinician’s satisfaction with the one-on-one sessions. The questions asked about satisfaction with length of session, satisfaction with instructor’s skills, degree to which expectations of the training were met, overall degree of effectiveness of the sessions, and degree to which the sessions made the clinician more efficient.

An overall satisfaction index was created using the mean score of responses to all five questions. Each question had a 5-point Likert type response which ranged from 1 to 5, with five indicating the highest value response. The overall satisfaction index averaged the responses creating a maximum score of 5.

The mean overall satisfaction score among all respondents was 3.8 with 91.3% reporting a moderate to high degree (≥ 3.0 on the index scale) of satisfaction. Women were significantly more satisfied with the one-on-one sessions than men (see Table 2). There was virtually no difference in satisfaction scores between younger and older respondents or when considering length of years with the managed care system.

Satisfaction and Perceived Improvement in CIS

Overall satisfaction was significantly correlated, measured by Spearman’s rho, with perceived improvement in efficiency at using each of the four CIS components: electronic medical record (r=.702 p<.001), results and lab reporting (r=.543 p<.001), email (r=.556 p<.001), and online library (r=.610 p<.001). Those who were more satisfied with the one-on-one training sessions reported higher improvements in efficiency in each of the CIS applications.

Perceived improvement in CIS was negatively correlated with age(r = −.28, p<.05) and years with the managed care system (r = −.37, p<.0001). Overall satisfaction was not significantly correlated with age or years with the managed care system.

DISCUSSION

This study set out to assess clinicians’ satisfaction with, and self-reported improvement in efficient CIS use resulting from, one-on-one advanced proficiency training. The results of this study indicate that clinicians were highly satisfied with the one-on-one CIS training sessions and that the training sessions led to reported increased efficiency in using CIS applications. When compared to other teaching methods, the one-on-one sessions were the most preferred method over classroom, small groups, and computer based training, irrespective of clinicians’ age or length of years using the CIS systems. Women clinicians found the one-on-one training method significantly more beneficial than men. Data obtained in the survey do not provide any explanation for the gender differences.

As would be expected, a very strong and highly significant correlation was found between overall satisfaction with the training and reported improvements in efficient CIS use. Those who apparently got the most out of the sessions in terms of improvements in CIS use were also most satisfied with the one-on-one teaching method. Based on the preference for one-on-one and its positive effect on CIS efficiency, this method should be seriously considered by managed care organizations in secondary CIS training programs following less costly introductory training methods

The most used component in a clinician’s typical day is the electronic medical record. Any improvements in efficient use of this component would likely translate into a greater perceived benefit for the clinician. The study findings show that the one-on-one training had the greatest positive effect on the use of the electronic medical record.

When asked to name the most useful item(s) learned that made a difference in the way they use the computer in their daily practice, clinicians noted that they valued the knowledge and skills gained. Many benefits mentioned focused on the use of short-cuts and other time saving features in the electronic medical record. For example clinicians noted that, “discovering that the ‘pink box’ represents the active screen, saved so much time”, “I had never used dot phrases [macros]of my own prior to this, [with the training] I created time-saving macros”, and “certain functions I did not know even existed made for more short cuts”.

The findings related to the electronic medical record indicate that prioritizing and focusing clinician training on optimizing the use of features and functions of the electronic medical record would provide high value and benefit to clinicians and the organizations’ training program curricula.

High ratings for the one-on-one sessions as compared to other teaching methods may be due to several reasons. This approach offers many advantages over other teaching methods. For example, teaching can focus on the specific needs of the individual clinician and assessment of weaknesses in using the CIS can more readily be identified. Trainee time is more efficiently used in one-on-one compared to other methods. The training occurs in the familiar surroundings of the clinician's own office and with the computer that the clinician uses on a daily basis. The competence and skills of the trainers may have also played a role in the high satisfaction ratings.

The results of significant clinician preference for one-on-one training should not be overlooked when organizations consider implementation and ongoing training of CIS. In addition, the low preference for the personalized computer based e-learning may appear counter intuitive. However, the findings may be indicative of how it was implemented or that this method may not be an optimal modality for clinician CIS training.

Despite the positive findings of this study a number of limitations exist. First, the study was a one-time post-implementation observational study. It did not assess any baseline measures for the study participants. Without baseline measures, we are unable to assess prior levels of proficiency in CIS or previous levels of satisfaction with other teaching methods. Future surveys of CIS would benefit from measuring competencies and satisfaction both pre- and post-intervention.

The study survey was administered in October 2002. The one-on-one training session program spanned a longer time period beginning in October 1999 and continuing to the present. Experience with other training methods may have occurred further in the past. It is possible that currency of experience may affect self-reported data and be subject to recall bias. Recall bias may lead to either understating or overstating self-reported values. It is unclear what effect, if any, recall bias may have on the results of the survey. The potential bias could be mitigated by administering the survey instrument immediately following the training session and by comparing self-reported measures with objective measures to assess the correlation between the two. Despite this limitation and potential bias, the value of the clinician’s perceptions, whether immediately following the training session or some time period thereafter, are still very relevant. If clinicians continue to perceive the benefit of the training sessions, even after a lengthy time period, this may attest to the overall value of the training sessions.

Second, improvement in CIS use was not measured objectively. In this study neither workflow improvements, true proficiency measurements, nor work attitudes were explicitly evaluated as an outcome measure. The use of objective measures and additional research in this area are recommended and essential for thorough evaluation.

Third, the target study population consisted only of those clinicians who had requested the one-on-one training. This group may be qualitatively different than those who did not ask for this type of training. Therefore, due to selection bias, generalization to the entire clinician population in the HMO cannot be made. A randomized controlled study or a matched cohort study design is required to test the generalizability of the findings to all clinicians.

Finally, and perhaps most importantly, the study evaluated the one-on-one sessions only from the perspective of clinicians. Cost analysis of the various training approaches and analysis of objective metrics of proficiency in CIS were not performed. Without these two analyses, the cost-effectiveness of one-on-one training in comparison to other teaching methods cannot be determined. Admittedly, creating measurable metrics to conduct a robust cost-effectiveness analysis is not a trivial task. What this study does demonstrate, however, is that in the eyes of the end-user, i.e. the clinician, the one-on-one training sessions are effective and valuable. The next step is to determine if they are cost-effective to the organization, if and when they should be implemented, and objectively determine how effective they are when compared to other methods.

CONCLUSION

This study was designed to assess clinicians’ satisfaction with and perceived benefit of one-on-one training in computer information systems in a managed care system. Both overall satisfaction and perceived effectiveness were ranked high by clinicians and were highly correlated. The findings support the assumption that one-on-one training is of value to clinicians and that this training method is valued above other methods. Additional research is suggested to further evaluate the benefits and methodologies of advance proficiency training in CIS.