Participant Perceptions of the Influences of the NLM-Sponsored Woods Hole Medical Informatics Course

Abstract

This report provides an evaluation of the National Library of Medicine–sponsored Woods Hole Medical Informatics (WHMI) course and the extent to which the objectives of the program are achieved. Two studies were conducted to examine the participants' perceptions of both the short-term (spring 2002) and the long-term influences (1993 through 2002) on knowledge, skills, and behavior. Data were collected through the use of questionnaires, semistructured telephone interviews, and participant observation methods to provide both quantitative and qualitative assessment. The participants of the spring 2002 course considered the course to be an excellent opportunity to increase their knowledge and understanding of the field of medical informatics as well as to meet and interact with other professionals in the field to establish future collaborations. Past participants remained highly satisfied with their experience at Woods Hole and its influence on their professional careers and their involvement in a broad range of activities related to medical informatics. This group considered their knowledge and understanding of medical informatics to be of greater quality, had increased their networking with other professionals, and were more confident and motivated to work in the field. Many of the participants feel and show evidence of becoming effective agents of change in their institutions in the area of medical informatics, which is one of the objectives of the program.

As a new and developing field of study, medical informatics is an exciting area. Evaluations of new curricula and training methods should be initiated to help expand the field. However, limited evidence exists to determine the most effective methods and elements of training in medical informatics and the effects of such training on patient care.¹ Given that training may occur through a variety of venues, such as tutorials, short courses, and degree programs, it is important to investigate which components of training are most valuable for new students interested in the field. For example, an evaluation of one recently instituted program at Indiana University found that diversity in the topics covered and the ability to interact with experienced guest speakers were important in making the program meaningful to students.² Other evaluations of medical informatics programs have found that involving a multiprofessional group of participants is important to generating ideas and engaging in rewarding discussions,³ and opportunities for continuing education are valuable elements to the training process.⁴ The current report focuses on one of the most well-established short courses offered in medical informatics, the National Library of Medicine (NLM) fellowship program at the Marine Biological Laboratory (MBL) in Woods Hole, Massachusetts.

Designed as an intensive, week-long survey course, the fellowship program at MBL in Woods Hole seeks to familiarize individuals with the application of computer technologies and information science in medicine, i.e., medical informatics, and provides a deeper understanding of the interrelated disciplines involved in the field. It is directed at physicians, educators, librarians, administrators, and faculty who are not currently knowledgeable about medical informatics but are, or could become, agents of change in their institutions, where fellows bring a diverse range of experience and goals to the course based on their professional backgrounds. Through a combination of lectures and hands-on computer exercises, participants are introduced to the conceptual and technical components of medical informatics. The conceptual components include principles of database design, human–computer interfaces, medical terminologies and coding systems, medical decision analysis methods, clinical information systems architectures, and methods for measuring costs and benefits in health care systems. The technical components include use of the Internet for biomedical applications, current and emerging wide area network technologies, use of literature and molecular sequence databases, and systems for telemedicine.

Taught by a nationally known faculty, the course prepares fellows to become actively involved in making informed decisions about computer-based tools in their organizational environment and improves the fellows' own computer skills. The faculty makes themselves available for informal consultations throughout the week, often taking their meals with fellows in the MBL cafeteria and joining fellows at the local watering hole after evening sessions. Offered twice a year, in the spring and the fall, the course is limited to 30 fellows per session. The small class size creates an intimate learning environment, allowing for ample interaction between fellows and faculty and the fellows with each other. The classroom is outfitted with wireless for 30 student laptops, and all note taking is done on the computers. Students can send files back to their home institutions via the Internet or take the course material home on a CD, thereby creating a pure paperless course experience. Fellows, who represent a wide range of occupational settings, educational backgrounds, and computer expertise, cite the opportunity to network with professionals from other health care fields, roughly divided into three groups—librarians, clinicians, and educators—as another valuable benefit of the program.⁵ As NLM Director Donald A. Lindberg, MD, explains at the opening session, the program seeks to ensure that the fellows “don't go away with any burning questions about medical informatics.” Costs of attending the course, including travel, housing, and meals at MBL, are fully supported by the National Library of Medicine, NIH. Based on the objectives of the course, the current report aims to provide an evaluation of the influences participation in the Woods Hole course has on the knowledge, skills, and behavior of course participants in the field of medical informatics and its ability to encourage participants to become effective agents of change in their own institutions.

Evaluation Design

To provide an insightful analysis and more detailed description of these experiences in an effort to improve future courses and better understand the course participants and their needs, two separate evaluations were carried out using a combination of quantitative and qualitative techniques. The evaluations provided a richer understanding of what participants expect from the course, their assessment of the course, and the participants' perceptions (with some evidence) of the influence the course had on their postcourse informatics-related activities.

The first study was an evaluation of the Spring 2002 Woods Hole Medical Informatics course, which was chosen as a convenience sample and was based on pre-/postcourse questionnaires and interviews with the 29 participants. Specifically, this study aimed to determine the course participants' knowledge of medical informatics and their involvement in the field before taking the course, their expectations and reasons for enrolling in the course, and their assessment of the course after its completion. A baseline questionnaire was developed and distributed to all of the participants before taking the course. Questions focused on gathering information about their baseline knowledge and involvement in the field of medical informatics, as well as their expectations for the course and its anticipated impact on their professional careers. Interviews were also conducted with the participants to elaborate on their reasons for taking the course and what they expected to achieve through their participation. A postcourse questionnaire was developed and distributed to the same 29 participants after completing the Spring 2002 WHMI course, where questions focused on gathering information about how well the course fulfilled their expectations, the aspects they liked best about the course, and any recommended changes to future course offerings. Follow-up interviews were conducted to determine participants' new informatics-related activities since completing the course, how confident they felt in performing these activities, whether they were receiving support from their department or institution, and any other comments about the course. Descriptive statistics were computed for all quantitative data to establish trends in knowledge and involvement in the field of medical informatics and the participants' satisfaction after completing the course. All qualitative data were coded according to the reasons for taking the course, specific expectations of the course and anticipated effects of the course, why expectations were or were not satisfied, the specific aspects of the course that were liked best, and the specific aspects of the course that participants would recommend changing.

The second study involved an assessment of the long-term influence of the Woods Hole Medical Informatics course on participants' informatics-related activities, based on a follow-up questionnaire and interviews with a sample of 121 past course participants from 1990 through 2001. Specifically, 360 questionnaires were mailed out to a random sample taken from a total of 720 past course participants. On average, the majority of total course participants traditionally have been medical librarians (52%) and physicians (41%) with a small number of students coming from other areas, such as nursing (7%). Questions focused on gathering information about their current knowledge and involvement in the field of medical informatics, how well the course fulfilled their long-term course expectations, and the perceived influence of the course on their professional careers. Descriptive statistics were computed for all quantitative data to establish trends in knowledge, involvement in the field of medical informatics, perceptions of influence, and long-term satisfaction. All qualitative data were coded according to specific informatics-related activities, how their experiences at Woods Hole influenced the perceptions of others, why expectations were or were not satisfied, the specific aspects of the course that affected the participants personally, and the long-term departmental and/or institutional influences as a result of course participation. It should be noted that two of our authors, Dr James J. Cimino and Cathy Norton, are affiliated with the course, the former as director of the spring 2002 course and the latter as principal investigator of the course grant. However, neither of these authors participated in the execution or analysis of the evaluation. Surveys and interviews for both studies were conducted and analyzed by research assistants who had no direct involvement in the Woods Hole course or the National Library of Medicine, where all participants were identified only by a file number to remain anonymous.

Results

Study 1: Short-Term Influence of the Woods Hole Medical Informatics Course

Many respondents who took part in the 2002 Woods Hole course were medical librarians (41%). Physicians (21%) and physicians with faculty positions (17%) also made up a large proportion of the respondents (38%). The remaining respondents were nurses with faculty positions (7%) or from “other” professions (14%), such as engineering and pharmacology. Thus, the distribution of participants in the 2002 Woods Hole course was similar to previous courses, which are largely made up of medical librarians and physicians as noted previously. The participants' backgrounds and their involvement in the field of medical informatics before taking the course included a substantial amount of experience with informatics-related activities. However, despite a broad range of experience in the field of medical informatics, most of the participants considered themselves as having an intermediate-level knowledge based on their self-ratings. When asked to rate the importance of the course, the participants considered it to be important to their careers. Specifically, when the participants were asked to rate how important the course was to themselves and their colleagues on a five-point scale (1 indicated “not important” and 5 indicated “very important”), the mean rating was 4.72 (SD = .53). Furthermore, 83% of the participants felt that the course would help them make an impact in the field of medical informatics at an individual level, whereas 79% felt it would help them make an impact at the institutional level and 62% at the departmental level. Thus, the participants considered the course to be important to their careers and anticipated that it would have an impact on their future informatics-related activities.

As indicated in ▶, the main reason participants cited for taking the course was to increase their knowledge and understanding of the field of medical informatics, as well as to fill in any gaps in existing knowledge, and was cited by 55% of the participants. Almost as important was the chance to meet and interact with other professionals interested in medical informatics (45% of the participants) to pursue potential future collaborations. Other reasons for taking the course cited by many of the participants included assisting in the development of new courses and curricula (34%), acquiring the tools to enhance existing information systems and to develop new ones (31%), and encouraging the integration of multiple disciplines at their own institution (28%).

Table 1.

Reasons for Taking Spring 2002 Woods Hole Medical Informatics Course as Indicated by Number (n) and Percentage (%) of Participants

	Participants
Reasons for taking course	n	%
Understand informatics better	16	55
Networking	13	45
Develop courses/curriculum	10	34
Enhance information systems	9	31
Integration of disciplines	8	28
Reenergize thinking	5	17
Formulate research ideas	4	14
Write grants	3	11
Learn evaluation techniques	2	8

After completion of the course, participants were very satisfied with their experiences. When asked to rate the extent to which their expectations were satisfied on a five-point scale (1 indicated “not at all” and 5 indicated “completely”) the mean rating was 4.61 (SD = 0.55). Positive feelings about the course as expressed by participants are reflected in the following representative statement given by one of the participants:

The quality of instructors was incredible, not a single hour went by that didn't apply to me. Very current, very applicable information. It was the most important professional development of my career.

All of the spring 2002 participants evaluated the course highly on its completion and felt it was an important contribution to their professional careers. Furthermore, as seen in ▶, when the participants were asked to rate their perceived postcourse knowledge of informatics on a scale, where 1 indicated “basic” and 5 indicated “advanced,” the highest percentage of their responses was a rating of 4 and the distribution pattern is skewed to the right. “Advanced” is used here as a relative term compared with basic level knowledge in this context and does not suggest a sophisticated knowledge of the domain after such a short period.

Figure 1.

Percent frequency of trainees and their ratings of informatics-related knowledge after taking the Spring 2002 Woods Hole Medical Informatics course compared with their precourse knowledge ratings (1 = basic, 5 = advanced). Dark bars represent postcourse and gray bars represent precourse.

Consistent with the main reasons that participants cited for taking the course, the best aspect of the course (as identified by 41% of the participants) was the broad scope of the course material. For example, one of the participants evaluated the course in the following way:

I was expecting an overview of medical informatics and that's exactly what I got. It was an excellent overview of the field that has enabled me to identify areas of interest.

Other aspects of the course that the participants liked included the high quality of the speakers and the opportunities to network with colleagues, as mentioned by 34% and 31% of participants, respectively. The importance of interacting with other professionals in the field is reflected in the following response given by one of the participants:

The best aspect of this course was the ability to meet and personally discuss informatics with national experts in the field. This was a wonderful opportunity for networking and learning that only comes once in a lifetime.

This kind of response was seen commonly in the participants' evaluations. Furthermore, 24% of the participants felt the learning environment was one of the best aspects of the course, as seen in the following participant's response:

It was an absolute luxury to be able to systematically study the subject of medical informatics in an uninterrupted and relaxed environment. The organization of the course content and logistics were excellent!

Finally, 21% of the participants referred to the acquisition of specific knowledge that they were seeking as a positive part of their experience.

Although the participants' overall evaluations of the course were predominantly positive, they did suggest several changes that might be considered for future courses. These changes were mainly related to the course content and instruction format. Thirty-one percent of the participants mentioned specific topics in the course that should be covered in greater detail, whereas 11% thought more topics should be added to the curriculum. For example, one of the participants indicated a lack of emphasis on research topics:

I really think there should be some more emphasis on grants. We need to know more about writing them and working on them. A brief overview of areas in which publications could be done by us beginners would be very good.

In terms of format, the participants felt that there should be more time for structured collaboration or team-oriented tasks (17%), the course should be longer (14%), there should be more time for group discussion (11%), and a precourse introduction to the general topics to be discussed would be an improvement (11%). For example, one of the participants had the following suggestion for improving the quality of their interactions:

It would be nice to have more structured sharing among the participants. This would show everyone's creativity in how they apply what we are learning. We learn almost as much from each other as from the faculty.

Given that there appeared to be people in favor of a broad coverage of topics in medical informatics, whereas others who already had a broad range of experience would like some detailed discussion of specific topics, lengthening the course is one way to satisfy both of these expectations.

Interviews conducted four months after taking the Woods Hole course suggested that participation had a positive influence on their careers and involvement in the field of medical informatics. Although none of the participants had a new professional title since completing the course, and four months is too short to expect such changes, they were all currently involved in new activities related to medical informatics. This fact is evidence of positive changes, although whether this is a direct influence of the course alone cannot be determined in this study. As indicated in ▶, the respondents were involved in the development of new programs related to medical informatics and had initiated new collaborations (80%), updating curricula at their institutions to include more informatics-related courses and topics into existing courses (60%), in the process of instituting new information systems (60%), teaching new informatics courses (40%), and recruiting more medical informatics professionals (40%). These changes in the lives of the participants since the course provide further evidence of the influence the course has had in their careers. Furthermore, all of the participants indicated that they had much more motivation and confidence to pursue future activities related to the field of medical informatics since taking the course and had received a great deal of support from their colleagues.

Figure 2.

New (in addition to ongoing) informatics-related activities since participating in the Spring 2002 Woods Hole Medical Informatics course as indicated by percentage of participants.

Study 2: Long-Term Influence of the Woods Hole Medical Informatics Course

Of the 360 questionnaires sent to past course participants, 33 were undeliverable (9.1%), 124 (37.9%) were returned, and three were deemed unusable. A total of 121 usable questionnaires (response rate = 36.8%) were used in the analysis. The respondents were a mixture of medical librarians (42%), physicians (50%), and nurses (8%). Thus, the respondents' backgrounds were similar to total course participants since 1992. The results of the long-term influences were described in the following categories: (1) current information technology activities, (2) current academic activities, (3) current knowledge improvement activities, and (4) perceived effects and reflections of the Woods Hole course.

Information Technology Activities

Since taking the course, the participants had been responsible for making strategic health care information technology–related decisions (64%) and training other people to use information technologies, systems, or tools (69%). Furthermore, 70% of the participants had recommended, specified, or approved new health care equipment or information systems/services for their institutions. The most frequent systems/services that were initiated were Internet/intranet services (18%), databases (15%), library automation systems (14%), and clinical information systems (13%). The participants perceived these changes to be directly related to the course, which had a positive impact by helping the participants engage in a wide variety of information technology–related activities.

Academic Activities

The results indicated that 58% of the participants had been involved in developing or presenting informatics-related courses at their institutions. This often included the development of new informatics courses being offered by the participants' home institutions, where 70% of their institutions offered zero to two new informatics courses, 22% offered three to five new informatics courses, and 8% offered five or more informatics courses. Most of these courses were intended for undergraduate or graduate students (45%) and professionals (36%). Thus, many of the institutions that past course participants came from had increased their involvement in educating others about medical informatics.

Knowledge Improvement Activities

The results indicated that 70% of the participants currently subscribed to one or more medical informatics journals, where the mean number of journal subscriptions for each participant was 2.79 (SD = 3.16). They most frequently subscribed to the Journal of the American Medical Informatics Association (68%), Bulletin of the Medical Library Association (39%), and the Proceedings of the Annual American Medical Informatics Association Fall Symposium (33%). These are the journals traditionally associated with medical informatics, and one can infer that most of the participants are generalists, because only a few subscribed to specialized journals. However, the participants likely read more of these journals than indicated since a number of them noted that their institutions provided them with access to informatics journals, and this is why they didn't need subscriptions.

Another key activity for learning about current topics in medical informatics was professional memberships, as indicated by 64% of the participants, where the mean number of professional memberships for each participant was 1.05 (SD = 1.11). These included the American Medical Informatics Association (68%) and the Medical Library Association (33%). Almost half of the participants (48%) had also attended or organized seminars since their experience at Woods Hole to continue learning about medical informatics.

Perceived Influences and Reflections of the Woods Hole Experience

The results showed that participation in the course influenced the participants' perceptions of how they were seen by their colleagues and their credibility in the field according to others. For example, when the participants were asked to rate the extent to which they felt they were perceived differently since taking the course on a 5-point scale (1 indicated “little change in perceptions” and 5 indicated “much change in perceptions”), the highest percentage of their responses was a rating of 4 with a mean rating of 3.25 (SD = 1.08). Specifically, the main effect of course participation on the past participants' (29%) perceptions was related to their credibility or recognition as an expert in the field of medical informatics. One of the participants described the perceived influence of the course in the following way:

Taking this course gave me personal credibility and made me a “local authority” for informatics issues. I have more credibility with colleagues and other professionals. Also, my superiors recognized and publicized my attendance.

The results also indicated that the participants perceived their experience at Woods Hole to influence their current knowledge. As seen in ▶, when the participants were asked to self-assess their current knowledge of informatics on a five-point scale (1 indicated “basic-level knowledge” and 5 indicated “advanced-level knowledge”), the highest percentage of their responses was a rating of 4, with a mean rating of 3.97 (SD = .783).

Figure 3.

Percent frequency of participants and their long-term informatics-related knowledge ratings since taking the Woods Hole Course compared with the precourse knowledge ratings of the Spring 2002 participants (1 = basic, 5 = advanced).

The participants' perceived personal influences of the course included interaction with colleagues (30%), increased understanding of the field (26%), confidence (21%), future involvement (17%), and current knowledge (14%) related to medical informatics. One of the participants described the personal influences of the course in the following way:

The Woods Hole course gave me a good basic knowledge of medical informatics, providing me with a foundation when I hear about new developments. It made me much more aware of a whole range of medical informatics processes going on all around me as I perform my job and how they are related to my position and interrelated with other fields.

The participants also felt their experience at Woods Hole had influenced certain aspects of their departments and/or institutions. The main influence for participants (29%) was on the development of new programs or curriculum related to medical informatics at their institutions, as seen in the following participant's response:

It gave me confidence to push for the inclusion of informatics for medical students. The integration of information training into the medical school curriculum at my institution may not have been a direct result of the Woods Hole experience, but it was definitely one of the major contributing factors.

Other important influences were on the development of new information systems and technology (17%), an increased awareness of medical informatics by others (14%), and increased discussion and consultation related to medical informatics among colleagues (13%).

Past participants of the Woods Hole Medical Informatics course remained highly satisfied when asked to rate the extent of their satisfaction on a five-point scale (1 indicated “not at all” and 5 indicated “completely”), the highest percentage of their responses was a rating of 4, with a mean rating of 4.07 (SD = .886). The reasons given for their satisfaction included increased knowledge of medical informatics (24%) and new collaborations that were formed as a result of taking the course (16%). The following is a typical response given by one of the participants when asked to reflect on their experiences at Woods Hole:

My experience exceeded my expectations in terms of the quality of the faculty, the range of topics covered and inclusion of broad overviews with practical, hands-on sessions, as well as underlying principles. I enjoyed interacting with other participants and gaining perspectives from a variety of viewpoints. I especially appreciated the opportunity to learn from top experts in the field and to talk to them in informal settings.

As seen in this response, the participants still had very positive feelings about the Woods Hole experience several years after attending the course. Many of the participants had new professional titles since their participation in the course, where these titles included top-level promotions such as two participants who were promoted to a full professor of medical informatics and several others who had become the vice president or director of their departments. However, these promotions and career changes did not occur immediately and were more likely to be seen after many years had passed after taking the course. This indicated that there could also be additional factors that could have influenced the change.

The majority of participants were extremely positive in their reflections of their experience taking the Woods Hole Medical Informatics course. Only one of the participants indicated “not at all” satisfied and explained this rating based on the following unfavorable experience:

I gave feedback to the presenters and planners at the course starting on the first day. They not only did not alter their instruction based on the feedback, they didn't even acknowledge that I gave any feedback. This was despite the fact that we were encouraged to give them daily feedback and told that it would be used immediately. I was extremely disappointed!

Others that were somewhat unsatisfied mentioned that they wanted more postcourse development at their home institution (6%), felt that the course topics were either too general or too specific (5%), would have liked a follow-up to update their knowledge on the current topics in the field (4%), wanted more practical applications of concepts during the course (3%), and felt that the course assumed too much prior knowledge of medical informatics (2%). Nonetheless, the majority of participants were extremely positive in their reflections of their experience taking the Woods Hole Medical Informatics course.

Discussion

Given an expressed desire to learn more about the field of medical informatics, the participants considered their experience at Woods Hole to be very important to their careers and their future informatics-related activities at personal, departmental, and institutional levels. There was a high degree of convergence among the participants' reasons for taking the Woods Hole course and what they hoped to gain from their participation. Specifically, attending the course was considered by the participants as an excellent chance to increase their knowledge and understanding of the field of medical informatics, as well as to meet and interact with other professionals in the field to establish future collaborations. As indicated by the participants' high level of satisfaction after completing the course, the “Woods Hole experience”⁵ appears to be successfully meeting the expectations of participants. This is further reflected in the aspects of the course that participants liked best, such as the broad scope of the course material, the high quality of the speakers, and the opportunities to network, which are consistent with the participants' reasons for attending. Furthermore, these key elements are consistent with evaluations of other medical informatics programs.^2,3

After completing the course, (1) participants continued to be very involved in a broad range of activities related to information technology, academia, and knowledge and skill improvement activities for working in the field. Many of these activities were aimed at continuing to update their knowledge of the current issues and topics of interest in the field of medical informatics today, demonstrating their commitment to the field; (2) participation changed the way they saw themselves and how others saw them. Specifically, the participants considered their knowledge and understanding of medical informatics to be improved relative to their precourse knowledge, had increased their networking with other professionals, were more confident and comfortable with their ability to work in the field, and were more motivated to pursue informatics-related activities. Furthermore, they felt their credibility within the field of medical informatics had significantly increased, which was perceived to have led to departmental- and institutional-level influences, such as the development of new programs, implementation of new information systems, and an increased awareness of informatics among others.

The long-term influences of the program included participants being aware of the impact of the program on their professional careers, which was attributed to the fulfillment of their two main goals for taking the course: the increase in their understanding of medical informatics and the increase in their collaborations with other professionals related to medical informatics. Although improvements in perceived knowledge of medical informatics since taking the course cannot be compared directly with the 2002 participants' self-assessment, the superimposition of these results does suggest that former participants have continued to increase their knowledge of the field. This is further demonstrated by the wide variety of informatics-related activities that past participants have become involved with in their careers since taking the course. Furthermore, the reasons for long-term satisfaction with the Woods Hole Medical Informatics course were congruent with the expectations of the 2002 participants and their reasons for attending, suggesting that there are commonalities between past and current participants.

There are several limitations that should be considered when evaluating the impact of the Woods Hole Medical Informatics course on participants' perceptions. First, the participants at the 2002 Woods Hole session may not be representative of past course participants, which, thus, limits the generalizability of these findings to other courses. Specifically, the participants' reasons for taking the course and their evaluations of its ability to meet their expectations do not provide any information regarding participants of other sessions of the course. It should be noted that the distribution of librarians, clinicians, and others in this study was roughly consistent with that of other years. Second, the response rate for the long-term follow-up of past course participants was relatively low. Although the study may be biased toward the participants who thought more positively about the course, it is possible that people who did not respond are much further removed from any direct influence of the course, and this does not mean that they were not influenced favorably. However, it should be noted that effects could be much more subtle and not easily attributed to the course directly. Some of these indirect effects, such as increased informatics-related activities that cannot be directly attributed to course participation, were captured in the data. Finally, it cannot be determined with any certainty whether the positive effect of the course on participants' perceptions is actually a causal result of their participation. For example, the increased involvement of past participants in informatics-related activities may be a result of their prior intentions to increase the emphasis of medical informatics in their careers before enrolling in the course rather than being the result of participating in the course itself. However, this fact itself is of interest, whether self-selected people with strong intentions to improve their knowledge, skills, and attitude toward medical informatics could be the very agents of change.

Conclusions

Participation at the Woods Hole Medical Informatics course resulted in positive perceptions of the influences on the skills, knowledge, and behavior for many of those who attended since its initiation. A number of those who have participated in the course continue to be eager to offer positive comments on their experience at Woods Hole, whether they attended in the past year or 10 years ago. The majority of the participants who responded to the questionnaire have since become effective agents of change in their institutions in the area of medical informatics, perhaps as a direct result of these positive experiences or as a result of a combination of other factors, in addition to the course.