Implementation and Evaluation of a Medical Informatics Distance Education Program

Abstract

Objective: Given the need for continuing education in medical informatics for mid-career professionals, the authors aimed to implement and evaluate distance learning courses in this area.

Design: The authors performed a needs assessment, content and technology planning, implementation, and student evaluation.

Measurements: The needs assessment and student evaluations were assessed using a combination of Likert scale and free-form questions.

Results: The needs assessment indicated much interest in a medical informatics distance learning program, with electronic medical records and outcome research the subject areas of most interest. The courses were implemented by means of streaming audio plus slides for lectures and threaded discussion boards for student interaction. Students were assessed by multiple-choice tests, a term paper, and a take-home final examination. In their course evaluations, student expressed strong satisfaction with the teaching modalities, course content, and system performance. Although not assessed experimentally, the performance of distance learning students was superior to that of on-campus students.

Conclusion: Medical informatics education can be successfully implemented by means of distance learning technologies, with favorable student satisfaction and demonstrated learning. A graduate certificate program is now being implemented.

Health care professionals, including physicians, other practitioners, administrators, and librarians, are lifelong learners. They must keep up with new information and knowledge to perform their jobs effectively and obtain advancement or promotion. Many areas of new learning are increasingly important, not the least of which is medical informatics—the study of acquisition, storage, and use of information in health care. Medical informatics is an important area for most health care professionals in which to gain and maintain knowledge, since the impact of information technology is becoming so widespread in health care.

The interest in such education in medical informatics became apparent to us when we received increasing numbers of enquiries about whether the courses or the entire program in the Oregon Health and Science University (OHSU) Master of Science in Medical Informatics degree could be taken via distance learning. This interest led us to develop and evaluate the distance learning program described in this paper.

The Division of Medical Informatics and Outcomes Research (DMIOR) at OHSU houses one of the largest academic medical informatics programs in the United States.¹ The MS program was launched in 1996 and matriculates about one dozen students annually. The program operates on a quarter system of four 10- to 11-week terms a year and requires 60 credit-hours for graduation; it usually takes two years on a full-time basis to complete. As of June 2001, the MS program has had 20 graduates, most of whom have taken positions in academia and industry.

The DMIOR also features a postdoctoral fellowship program funded by the National Library of Medicine (NLM) and the Department of Veterans Affairs (VA). It also has a large externally funded research program and has been designated an Evidence-based Practice Center by the Agency for Healthcare Research and Quality (AHRQ).

This paper describes the process we used to develop, implement, and evaluate our distance learning program. We began with a needs assessment to determine what topics were of interest to potential students. This was followed by evaluation and selection of available technologies for course and content delivery. Next, we implemented the courses and adapted our on-campus evaluation instrument to assess their effectiveness. Finally, we planned for development of a complete program in the future.

Background

Distance learning is usually defined as education that takes place when distance or technology separate the teacher and learner.^2,3 Since the Medical Subject Headings (MeSH) term Distance Education was adopted in 1999,^* a total of 167 medline records have been indexed using it. A catalog of distance learning programs shows that nursing is the area of health care with the greatest number of programs, as 65 institutions offer degrees at the bachelors, masters, and doctorate levels.⁴ There are a smaller number of programs in public health, dental technology, and mental health. There are also an increasing number of online continuing clinical education offerings for all types of health care professionals.

The number of distance learning programs in informatics is small, as noted by the list of academic and training programs on the Web site of the American Medical Informatics Association (AMIA).^† As with health care programs generally, the largest number of programs is in nursing informatics, with a total of five programs. The need for nursing informatics distance learning programs has been assessed, with the conclusion that such programs could provide career opportunities for those with interest.⁵ Online programs in medical informatics as of this writing include the graduate certificate program at OHSU, the Stanford University Short Course in Medical Informatics, and the planned certificate program from the Association of Medical Directors of Information Systems.⁶ Several medical informatics programs, such as those at the University of Illinois at Chicago, the University of Texas Houston Health Sciences Center, and the University of Alabama at Birmingham, have also begun to employ distance learning technologies in their on-campus courses.

Distance learning programs have been around for more than a century. As noted by Phipps et al.,⁷ they began as mail-based correspondence courses in the 19th century, followed by the adoption of radio, television, and videotape transmission in the 20th century. In the last decade, with the growth of the Internet and interactive videoconferencing, these newer media have achieved substantial use.

Many evaluations of the effectiveness of distance learning programs have been done over the years. A systematic review from 1990 concluded that learning can occur with equal effectiveness via distance learning.⁸ More recent studies have reached the same conclusion.⁷ These reports do, however, warn that the methodologic quality of such studies is often poor, indicating that results should be interpreted with care.

On the basis of these studies and personal experience, a variety of “principles” have been promulgated for distance learning, as exemplified by Sherry.⁹ These are similar to principles one might offer for any informatics application, such as meeting the needs of learners (users), using technology appropriate for the task, providing adequate support for the learner (user), and focusing on the student and not the technology. The National Education Association has developed a set of benchmarks for measuring quality in Internet-based distance learning, which are categorized under institutional support, course development, teaching/learning, course structure, student support, faculty support, and evaluation/assessment.¹⁰

Many of these principles and benchmarks, however, have not been subject to formal evaluation. An example of this is something that was key to the development of the courses described in this paper, which is the appropriate length of time for segments in online lectures. A common rule of thumb expressed to us by many persons with experience in distance learning was that such segments should be no longer than 20 minutes, although we were unable to verify the “truth” of this from any distance learning literature.

Needs Assessment

Based on the interest from persons who had enquired about the availability of our courses via distance learning, we decided to undertake a needs assessment to measure what content and type of program were of interest to them. We decided to develop a survey and mail it to those who might take our courses.

Methods

The survey instrument was developed by consensus with on-campus medical informatics faculty and interested students. It was determined that the survey should ask questions in the following areas:

• Whether they or someone they knew would be interested in medical informatics distance learning

• Demographic information, e.g., age group, gender, whether or not they were a clinician

• Type of certification, e.g., master's degree, certificate, completion of individual courses

• Modes of access, e.g., modem, broadband network

• Preferred delivery mechanism, e.g., Web-based, e-mail, satellite

• Interest in visiting OHSU campus and frequency, e.g., once a year, twice a year, or once a quarter

• Access to local courses (e.g., computer science or statistics) and libraries likely to contain medical informatics books and journals

• Relative importance of course attributes, from faculty availability to cost

• Interest in specific topics, e.g., electronic medical records, information retrieval, outcome research

The final survey had 39 multiple-choice questions (yes/no or Likert ratings). It also provided spaces for respondents to indicate their e-mail address if they wanted a copy of the survey and to make general comments.

Since the most interest in our offerings had been expressed to us at the annual AMIA meetings, we decided to send the survey to members of medical informatics professional groups. We selected 500 members each at random from the AMIA and Health Information Management Systems Society (HIMSS) mailing lists. The initial survey was sent by postal mail in the spring of 1999. A follow-up survey was sent to those who did not initially respond one month later. Since this was an exploratory survey, we had no a priori hypotheses and for this reason did not perform any statistical analyses.

Results

A total of 288 surveys were returned (response rate 28.8 percent). There was definite interest in medical informatics distance education, as 57 percent said they would be interested themselves, and another 19 percent said they knew of someone else who would be interested. We assumed those who did not return the survey were not interested. About 58 percent of the respondents were clinicians. The median age was 41 to 50 years. About 63 percent were male. Half had doctoral degrees (MD or PhD), about 30 percent had master's degrees, and about 20 percent had bachelor's degrees.

For certification, respondents were allowed to select more than one option. About half said they would be interested in a non-thesis MS and about half said they would be interested in a certificate program. About a third were interested in an MS with a thesis and a third were interested in individual courses with professional continuing education credit.

Respondents overwhelmingly favored Web-based delivery, followed by e-mail. Virtually all had modem access to the Internet, and more than two-thirds had higher-speed network access. (We did not ask whether the network access was from home or work; given the time when we did the survey, it was likely that most had this access from work.) The preferred frequency of student on-site visits was once per year (52 percent) followed by none at all (28 percent). There was minimal interest in visiting once or twice a quarter (16 percent) or more frequently (5 percent). Virtually all respondents had nearby library access and either had access to or had already taken courses in computer science or statistics.

Factors influencing decision to enroll were measured via a Likert scale. The results are shown in Table 1▶, with content, self-paced schedule, and delivery mode ranking highest. A similar process was performed for curricular interests. The categories listed were based on the major courses in our current on-campus program. Table 2▶ shows a ranking of the most important curricular interests, with electronic medical records and outcome research ranking highest.

Table 1 .

Mean Ratings for Factors Influencing the Decision of a Potential Student to Enroll in the Program

Factor	Score
Content	4.6
Self-paced schedule	4.3
Delivery mode	4.2
Faculty credentials	4.2
Cost	4.1
Access to faculty	3.7
Computer support	3.5

note: On the Likert scales, 1 indicated low influence; 5, high influence.

Table 2 .

Mean Ratings for Subjects That Potential Students Would Like to See Covered in the Program

Subject Preferences	Score
Electronic medical record	4.3
Outcomes research	4.1
Database systems	3.8
Artificial intelligence and decision support	3.8
Information retrieval and digital libraries	3.8
Telecommunications and computer networks	3.7
Organizational behavior and management	3.6
Telemedicine	3.7
Research study design	3.2
Computer programming	2.9
Bioinformatics	2.2

note: On the Likert scales, 1 indicated low influence; 5, high influence.

The needs assessment indicated to us that there was definite interest in medical informatics distance education. Of key importance to potential enrollees were that the courses have appropriate content, flexibility in scheduling, and an acceptable mode of delivery. In particular, the preferences of the respondents were that the courses be Web-based but accessible via modem, lead to some sort of certification, and parallel the curriculum of our on-campus MS program.

Content and Technology Planning

The implementation of the content and technology was guided by the needs assessment. Further planning consisted of evaluating technology for course implementation and delivery.

Course Delivery

The first step in planning was to determine whether to use a distance learning hosting company, such as eCollege (www.ecollege.com) or manage our own server and course materials. Because we had the expertise to develop content and manage a server, we chose the latter option. A Sun Enterprise 250 server (Sun Microsystems, www.sun.com) with half a gigabyte of RAM, dual processors, and 30 gigabytes of hard-disk space was acquired for initial testing and later implementation. The system was implemented on the campus network with 100 megabit per second access to our Internet gateway.

We looked at a variety of course delivery shells. One option was a locally developed system that had already been used effectively.¹¹ While this system was created when commercial course delivery shells were in their infancy, by the time we undertook this process (summer 1999), the commercial systems had long surpassed our system in functionality. Our major evaluative decision was between WebCT (www.webct.com) and Blackboard CourseInfo (www.blackboard.com). Sample materials were produced for all the teaching modalities (described below), mounted on both systems, and presented to the seven faculty and four students participating in the evaluation process. In a consensus process, the near-unanimous sentiment was that while WebCT provided more “bells and whistles,” Blackboard CourseInfo presented a much simpler and consistent user interface, especially for the teaching modalities we planned to use.

Teaching Modalities

In selecting teaching modalities, we adhered to two guiding principles:

• We wanted to provide parallel experiences to all aspects of on-campus learning, from lectures describing the content verbally to high-quality readings and interaction among students and faculty.

• We needed the modalities to be deliverable over a modem connection, which ruled out high-bandwidth content such as video.

The first course to be implemented would be MINF 510, Introduction to Medical Informatics. Since OHSU is on an academic quarter system, this course would cover 11 weeks. This introductory survey course is usually offered on campus in the fall quarter. It is taken by entering medical informatics MS students as well as students in public health, graduate nursing, and other programs. Like most courses in the MS curriculum, it is a three-credit course that meets three hours per week. In addition to weekly lectures, in which interaction is encouraged by the instructor, the course also includes weekly readings, weekly homework assignments that attempt to demonstrate application of the course content, a term paper that allows the student to explore a specific area of interest in greater detail, and a final examination.

The first online version of the course was implemented in parallel with the on-campus course in the fall of 1999. This was done to minimize the amount of additional curricular development that would be necessary. The syllabus for both the on-campus and online versions of MINF 510 is shown in Table 3▶.

Table 3 .

Syllabus for MINF 510, Introduction to Medical Informatics.

Week	Topic	Textbook Chapter
1	Acquisition, storage, and use of medical data	1, 2
2	Medical computing	4, 5
3	Medical decision making and evidence-based medicine	3 (§§ 1–5)
4	The electronic medical record	9, 10
5	Standards, security, and confidentiality	6, handout
6	Information retrieval and digital libraries	15
7	Imaging and telemedicine	14, handouts
8	Artificial intelligence and decision support	16
9	Computer networks and the Internet; ethics of medical informatics;projects due; final examination distributed	7
10	Nursing, public health, and consumer health informatics; final examination due	11, 12

Implementation

As noted above, the first course was offered in the fall quarter of 1999. Participation in the needs assessment as well as exposure to advertising on the DMIOR Web site led more than 100 persons to express an interest in taking the course. Because it was our first offering, we limited enrollment to a size we felt could be managed, which was 15 people. A computer experience survey was distributed to all who were interested, and we chose a group with adequate experience in using the Web and browser plug-ins. Those not selected were assured that they would be accommodated in subsequent terms. A second offering of MINF 510 was made in the winter quarter of 2000. Continued interest led us to offer the course again in the spring and summer terms of 2000. A second course was added in the spring quarter of 2000. The enrollment in subsequent terms was not capped, although it never exceeded 19 students.

Each course was taught by the same instructor (W.R.H.), aided by one teaching assistant (K.G., M.M., or P.T.). The instructor led the course, prepared all the curricular materials, and led the discussion boards. The teaching assistants maintained the server, posted materials, and provided technical support. A departmental administrator handled registration and other administrative details. Students paid the standard OHSU graduate tuition and fees ($849 for in-state students and $1,203 for out-of-state students, $170 of which was refundable with documentation of existing medical insurance).

Course 1: Introduction to Medical Informatics

Each week the instructor would prepare the following materials for both courses:

• Learning objectives aiming to describe the most important topics to be learned

• Reading assignments taken from pre-prints of the forthcoming second edition of Medical Informatics: Computer Applications in Health Care,¹² which we used as a textbook.

• Lectures developed in Powerpoint (Microsoft Corp., www.microsoft.com) and played using RealPlayer (RealNetworks Corp., www.real.com).

• Discussion questions aiming to encourage discussion on the most important issues surrounding the topic

• Homework assignments aiming to require application of the concepts taught in the week's materials

The online lectures were produced using the Record Narration feature of Powerpoint that captures WAV sound files and slide timings. All the lectures were recorded in a quiet, carpeted office using an Optimus Nova 20 microphone and a standard Windows laptop computer. A Powerpoint add-in, RealPresenter (RealNetworks Corp., www.real.com), was used to convert the file into a RealPlayer presentation that could be delivered by streaming over the Web. RealPresenter compresses the images and audio to minimize bandwidth usage. The resulting streaming file, which has a very low frame-rate for the video (since it is composed of relatively static Powerpoint slides), gives acceptable performance even over a modem connection.

In the first offering, the weekly homework assignments were the same short-answer and matching problems given to on-campus students. However, it was found that these were difficult to grade, since they were submitted as Microsoft Word files and the attempt by the teaching assistants to provide grading and explanations in red font proved to be very time-consuming. Assignments in future courses were converted to multiple-choice format so that they could be graded automatically by the Blackboard software.

All the weekly materials mentioned above were posted every Wednesday, with students given one week to complete the homework assignments. Once posted, all material was kept on the server for the duration of the term, enabling students to review prior material. Discussions began the week the material was posted but typically continued into the following week or two. The instructor composed three or four discussion questions per week, although students were free to post their own questions (but rarely did). The instructor read the discussion boards two or three times a week and tried to balance letting the students interact with each other against adding his own knowledge and perspective.

Technical support was provided by both phone and e-mail, with the promise that all messages would have a response by the following working day. Students were also given access to all the online resources of the OHSU Library. These included the library card catalog (from which they could check out books), our licensed bibliographic and full-text databases (medline and other databases accessible via Ovid, plus full-text journals). Additional reading assignments, beyond those in the textbooks, were made only if the reading materials were freely available on the Web.

Students registered for the course as non-degree students. Because it was a graduate-level course, they had to demonstrate proof of at least an undergraduate degree. They paid the same tuition and fees as on-campus students, with the exception of the waiving of the student health fee.

Figures 1 to 5▶▶ ▶ ▶ ▶ show screen captures of various aspects of the course. Figure 1▶ shows the course information that students see when they log on. Figure 2▶ shows a Powerpoint slide from a RealPlayer lecture presentation. Figure 3▶ shows an overview of the discussion board, while Figure 4▶ shows a message posted by a student. Figure 5▶ shows some sample questions from the homework. A sample lecture is available at the course demo site at http://distance.ohsu.edu.

Figure 1 .

Course information.

Figure 2 .

RealPlayer-based lecture.

Figure 3 .

Discussion board.

Figure 4 .

Message from discussion board.

Figure 5 .

Questions from homework.

Course 2: Information Retrieval and Digital Libraries

In the spring of 2000, we implemented a second course, MINF 514, Information Retrieval and Digital Libraries. We used the same general approach that had been used in MINF 510, with online lectures, readings (from the author's textbook, Information Retrieval: A Health Care Perspective¹³), discussion questions, term paper, and final examination. The syllabus for MINF 514 is shown in Table 4▶. The syllabus includes supplemental readings of innovations in information retrieval since the publication of the textbook. Each student was asked to take the lead on one article, providing a synopsis and one or two questions on the discussion board.

Table 4 .

Syllabus for MINF 514, Information Retrieval, and Digital Libraries

Week	Title	Textbook Chapters	Supplemental Readings
1	Terms, models, and resources; health information	1, 2	None
2	Evaluation; databases	3, 4	Analysis of clinical questions
3	Indexing; retrieval	5, 6	Medical core metadata
4	Evaluation	7	Resource description framework
5	Word-statistical systems	8	PubMed Central
6	Linguistic systems	9	Accessibility of information on the Web; search engine features for Webmasters
7	Assisting the user	10	Evaluation of medline searching; the Text Retrieval Conference (TREC)
8	Processing the clinical narrative; hypermedia	11, 12	MedWeaver; natural language information retrieval
9	Internet and the Web; projects due; final examination distributed	13	UMLS Metathesaurus; building hypertext using information retrieval
10	Final examination due	None	None

Faculty Observations

The formal evaluation is presented in the next section, but a number of subjective observations were made by the course faculty. In general, we found the students to be more enthusiastic about learning than their on-campus counterparts. We believed this was because they tended to be in mid-career (unlike many of our on-campus students, who are still in their initial training) and were more inclined to learn to advance their careers. The fact that they were devoting considerable time, not to mention tuition money, no doubt gave them impetus to want to maximize their learning.

Although actual hours worked were not tracked by faculty, it was estimated that the course leader spent 3 to 5 hours per week preparing materials and another 2 to 4 hours per week participating in the discussion boards and answering direct e-mail enquiries. Neither course required new curriculum development, since both were adaptations of existing on-campus courses. Each teaching assistant spent 10 to 20 hours per week managing the server, mounting materials, answering technical questions, and otherwise interacting with students.

A number of observations were made about the discussion boards. The amount of discussion on these boards greatly exceeded the amount of discussion that occurs in the on-campus classes, which is quantified in the next section. The course faculty, including the senior instructor, also learned new aspects of medical informatics themselves from this accomplished and diverse group of students. A number of them, for example, were involved in implementation of informatics applications in their practices or hospitals and had perspective to augment the academic theory being presented in the class.

Few problems were encountered through the entire academic year. As noted below, general satisfaction with the modalities and the software were quite high. The course content was almost always available when students tried to access it, with the exception of a problematic period near the end of the spring 2000 term (which affected both MINF 510 and MINF 514), when a major router of the OHSU Internet gateway failed at the time its Internet connectivity was being upgraded. For several days, students had difficulty accessing course materials, with RealPlayer slide shows timing out. The timing was unfortunate, as students were preparing for final examinations. Ultimately, an extension was given for turning in the final examination, and great effort was made to win back the positive frame of mind that had accompanied our success to that point.

Probably the least popular aspect of the courses was the weekly homework assignments. As noted above, the goal of these exercises was to apply the weekly content. The multiple-choice format precluded rote recitation of facts (which the instructor avoids in his courses anyway). As with many multiple-choice questions, one or two questions of the ten each week could be interpreted differently than the instructor intended. These usually generated e-mail discussions, sometimes heated, from students, who were assured that such interpretations would be taken into account when determining their final grade. A number of questions were changed each term to reflect the problems identified in prior terms.

All class activity was asynchronous. In the first three terms, several attempts were made to organize real-time sessions using the “virtual classroom” of Blackboard, which provides a chat room capability. However, no consensus could be reached among the students for times to do it. Finally, in the summer 2000 course, one session was scheduled and a free-form question-and-answer session was held. Although no formal assessment of this event was done, the consensus among students and faculty was that the session offered no advantages over bulletin board discussion, in no small part because of the poor response time of the Blackboard real-time software.

Evaluation

Because our evaluation was not a randomized controlled trial, the main focus was on student satisfaction. We did, however, measure some aspects of the course to determine whether learning was taking place and to quantify the interactivity of the discussion boards.

Student Learning

To compare online with on-campus learning, we compared two aspects of the courses that were relatively similar in both, the final examination and the term paper. Direct comparison of final examination results is not definitive for which teaching setting is “better,” since the student groups were different and the amount of time to take the examination was different (on-campus students took the examination in an open-book, 3-hour classroom setting whereas distance learning students took it “take home” style and had a week to complete it). However, the comparison can be used to show that learning took place among the distance learning students.

Online students performed better on the final examination in both courses that were offered online and on-campus simultaneously. In MINF 510, the average grade on the final examination was 83.6 percent (range, 60–97 percent) for the online students compared with 72.7 percent (range, 48–94 percent) for the on-campus students. In MINF 514, the average grade on the final examination was 93.0 percent (range, 86–98 percent) for the online students compared with 74.4 percent (range, 66–89 percent) for the on-campus students. The online students also wrote better term papers. In MINF 510, their average grade was 95.4 percent (range, 80–100 percent) compared with 91.0 percent (range, 80–100 percent). Likewise, in MINF 514 their average grade was 92.5 percent (range, 75–100 percent) compared with 86.7 percent (range, 60–100 percent).

Interactivity

As noted above, the course faculty perceived the online courses to be highly interactive. An analysis of the number of discussion board postings confirms this. As shown in Table 5▶, an average of 41.6 postings were made to the discussion board each week, approximately 3.0 per student. The average number of words in each posting was 111. Although we do not measure the number of student comments or their word counts for on-campus courses, we are certain that in-class discussion does not occur at the level of the online class, i.e., three comments per student each week for all students. The table also shows that the number of postings varied considerably, with a definite trend for them to diminish over the term as they became more occupied with completing the term project and preparing for the final examination.

Table 5 .

Number of Discussion Board Messages Posted for Each Week's Questions in Each Course

Week	MINF 510 Fall 1999	MINF 510 Winter 2000	MINF 510 Spring 2000	MINF 510 Summer 2000	MINF 514 Spring 2000	Average per Week
1	105	43	54	69	38	61.8
2	87	62	48	65	59	64.2
3	74	34	45	55	44	50.4
4	55	37	37	42	18	37.8
5	36	34	26	58	29	36.6
6	35	36	29	52	42	38.8
7	24	24	25	41	41	31.0
8	30	23	27	38	30	29.6
9	24	23	20	20	32	23.8
Average per week for each course	52.2	35.1	34.6	48.9	37.0	41.6
Average per week for each course per student enrolled	3.5	2.3	3.5	2.6	3.4	3.0

Student Satisfaction

The bulk of the evaluation focused on student satisfaction, which was performed by expanding the instrument used to evaluate on-campus courses. Questions were added to elicit feedback about specific aspects of the online learning environment. In the first term, we also documented technical support contacts. In the spring 2000 term, we added, for both courses, additional questions about how and when the students accessed the materials as well as the acceptability of server and software performance.

Table 6▶ shows the number of students who started, completed, and filled out the evaluation form for each course. A total of 52 students took MINF 510 during the year. Of the nine students who took MINF 514, eight had already taken MINF 510. Among student occupations, the most common were physician (36), nurse (6), and medical librarian (3). Among the others were a library science student, a computer science student, and a medical researcher.

Table 6 .

Number of Students Who Started, Completed, and Evaluated Each Course

	MINF 510 Fall 1999	MINF 510 Winter 2000	MINF 510 Spring 2000	MINF 510 Summer 2000	MINF 514 Spring 2000	Total
Students who started course	15	15	10	19	11	70
Students who completed course	14	14	9	15	9	61
Students who completed evaluation form	13	10	8	11	8	50

Table 7▶ shows Likert-scale evaluations for a variety of questions for each course. Students felt that the slide plus audio (as opposed to just audio) lectures were valuable. They generally enjoyed the discussion boards, although a minority felt they spent too much time reading messages. There was general satisfaction with the textbooks, and most but not all students had access to adequate library resources. All students felt the support staff was prompt and helpful or said they had no basis to judge, i.e., they never needed technical support. Virtually all students they had adequate preparation for the technical aspects as well as the content of the course.

Table 7 .

Student Evaluations of Course Modalities, with Number of Responses for Each Category

	Strongly Agree	Agree	Disagree	Strongly Disagree	No Basis for Judgment/Not Applicable
The lectures were a valuable addition to the text materials (book and handout):
MINF 510 Fall 1999	11	1			1
MINF 510 Winter 2000	8	1	1
MINF 510 Spring 2000	6	2
MINF 510 Summer 2000	9	1	1
MINF 514 Spring 2000	6	2
TOTAL	40	7	2	0	1
I would have preferred access to just the audio portion of the lectures
MINF 510 Fall 1999			1	11
MINF 510 Winter 2000	1		3	6
MINF 510 Spring 2000		1	5	4
MINF 510 Summer 2000		2	5	4
MINF 514 Spring 2000	1		3	5
TOTAL	2	3	15	30	0
I liked using the discussion board:
MINF 510 Fall 1999	3	9	1
MINF 510 Winter 2000	5	3	2
MINF 510 Spring 2000	4	3	1
MINF 510 Summer 2000	4	6	1
MINF 514 Spring 2000	1	6	1
TOTAL	17	27	6	0	0
I spent too much time reading through the discussion board messages to find useful comments:
MINF 510 Fall 1999	1	3	8	1
MINF 510 Winter 2000		4	2	4
MINF 510 Spring 2000			7	1
MINF 510 Summer 2000		2	9
MINF 514 Spring 2000		1	6	1
TOTAL	1	10	32	7	0
The content of the textbook was appropriate for this course
MINF 510 Fall 1999	6	7
MINF 510 Winter 2000	7	3
MINF 510 Spring 2000	5	3
MINF 510 Summer 2000	5	6
MINF 514 Spring 2000	5	3
TOTAL	28	22	0	0	0
I had adequate access to a library (local or OHSU) and other research resources for completing the course paper:
MINF 510 Fall 1999	3	5	3	1	1
MINF 510 Winter 2000	5	3	1
MINF 510 Spring 2000	4	3	1
MINF 510 Summer 2000	2	6	2
MINF 514 Spring 2000	2	3	1	1
TOTAL	16	17	8	2	2
					continued
The support staff provided prompt and valuable assistance with technical issues (the use of Blackboard and the required plug-ins):
MINF 510 Fall 1999	8	2			3
MINF 510 Winter 2000	1	4			5
MINF 510 Spring 2000	4	2			2
MINF 510 Summer 2000	6	2			3
MINF 514 Spring 2000	5	3
TOTAL	24	13	0	0	13
The support staff provided prompt and valuable assistance with course-related issues:
MINF 510 Fall 1999	10	3
MINF 510 Winter 2000	4	4	1
MINF 510 Spring 2000	6	2
MINF 510 Summer 2000	7	4
MINF 514 Spring 2000	3	5
TOTAL	30	18	0	0	13
I feel that I had adequate experience with and preparation for the technical (computer usage) aspects of this course:
MINF 510 Fall 1999	10	3
MINF 510 Winter 2000	5	5
MINF 510 Spring 2000	7		1
MINF 510 Summer 2000	8	2	1
MINF 514 Spring 2000	5	2	1
TOTAL	35	12	3	0	0
I feel that I had adequate experience with and preparation for understanding the content of this course:
MINF 510 Fall 1999	5	6	2
MINF 510 Winter 2000	4	6
MINF 510 Spring 2000	7		1
MINF 510 Summer 2000	4	7
MINF 514 Spring 2000	3	4	1
TOTAL	23	23	4	0	0

More students accessed the courses with a telephone modem than dedicated network connection (21 vs. 7) and a majority (16 vs. 11) used a national Internet service provider such as America Online or Earthlink. Students were widely distributed across all North American time zones, with the exception of one student from New Zealand, who connected by telephone modem and rarely had access problems (Table 8▶). Most students rated the response time for all course modalities as usually acceptable (Table 9▶).

Table 8 .

Time Zone Distribution of Students

	MINF 510 Spring 2000	MINF 510 Summer 2000	MINF 514 Spring 2000
Pacific	5	3	3
Mountain		2
Central		2	3
Eastern	1	4	1
New Zealand			1
Not stated	2

Table 9 .

Acceptability of Responsiveness for Course Modalities

Response Time Relative to Other Web Sites	Always Acceptable	Usually Acceptable	Usually Unacceptable	Always Unacceptable
For Blackboard:
MINF 510 Spring 2000	1	6	1
MINF 510 Summer 2000	3	8
MINF 514 Spring 2000	1	7
TOTAL	5	21	1	0
For discussion boards:
MINF 510 Spring 2000	1	7
MINF 510 Summer 2000	5	6
MINF 514 Spring 2000		7
TOTAL	6	20	0	0
For RealPlayer lectures:
MINF 510 Spring 2000	3	5
MINF 510 Summer 2000	1	9	1
MINF 514 Spring 2000	3	4
TOTAL	7	18	1	0

Figure 6▶ lists the technical and instructional support interactions for the teaching assistant in the first offering of the course. All interactions except one were by e-mail. Most contacts were related to non-technical issues, such as disputing or clarifying homework answers. The interactions were spread across the entire term.

Figure 6 .

Technical and instructional support interactions for the teaching assistant in MINF 510 Fall 1999.

In general, technical problems were rare. A number of students had problems configuring their browsers or RealPlayer plug-in, but once these were corrected, subsequent problems were minimal. There were occasional reports of temporary inability to access the server, which were most likely due to Internet congestion somewhere between OHSU and the student's Internet service provider. The most serious technical problem occurred during finals week of the spring term, when a router failure compromised the OHSU Internet gateway. This caused all Internet traffic in and out of OHSU, including traffic from our courses, to slow significantly. This caused considerable distress to students and resulted in a four-day extension of the deadline for turning in the final examination. This experience actually highlighted how good server access had been during the rest of the year.

The evaluation forms also asked a number of open-ended questions. Several concerned the value of the online lectures. Almost all students stated that lectures were a valuable learning experience. Most mentioned that they provided greater insights into key concepts than would be possible from reading the book . One student, however, said that two hours of lectures was inefficient compared with the 30 minutes required to just read the Powerpoint slides. Virtually all students said that breaking the lectures into 20-minute segments was about right, and that those over 20 minutes were too long to view. Some mentioned that anything over 15 minutes was too long.

When asked how the online lectures compared with other lectures in classes they had in their previous education, students indicated that they recognized trade-offs between having convenient time to access lectures and having direct interactivity with an instructor. Many liked the fact that they could pause lectures or go back to a previous portion if they missed something. Some did mention, however, that they missed being able to stop the instructor and ask questions.

A variety of responses were given in response to a question about whether there was anything they would do differently in the use or presentation of the lectures. A number noted that the compression of graphics by RealPlayer made some of the images difficult to view on the screen or in print. Many students noted that RealPlayer did not demarcate specific slides, making navigating through them more difficult than it otherwise might have been. Several students mentioned that they would have preferred to download lectures rather than view them by streaming.

Some students complained that sessions about medical decision making in MINF 510 and about recall and precision metrics in MINF 514 emphasized the mathematical aspects too heavily. Another student complained that MINF 514 should have placed more emphasis on emerging Internet technologies.

Future Directions

Based on the success of these offerings, along with results from the initial needs assessment and subsequent discussions, a graduate certificate program has been implemented. The program consists of a subset of courses in the current MS program that are considered to be most useful for a target audience interested in gaining skills in medical informatics to complement their existing career as physician, nurse, other health care professional, administrator, or librarian.

Table 10▶lists the curriculum. Five courses (MINF 510, 512, 514, 518, and 564) are identical to those on campus, except that they are delivered in the online format. Each of these courses will be offered at least once a year, beginning the first year of the program.

Table 10 .

Courses in the Curriculum of the Graduate Certificate Program

MINF510 Introduction to Medical Informatics

MINF512 Clinical Systems

MINF514 Information Retrieval & Digital Libraries

MINF517 Organizational Behavior and Management

MINF520 Consumer Health Informatics

MINF571 Bioinformatics: Computers in Bioscience

MINF546 Computer Science Topics

MINF564 Introduction to Outcomes Research

MINF528 Advanced Topics in Medical Informatics

Graduation from the graduate certificate program requires eight three-credit courses, a total of 24 credit hours. Courses taken by distance learning can be applied for credit toward the on-campus MS degree for those accepted into the MS program. Students are also allowed to have one course consist of a research (MINF 501) or practicum (MINF 509) experience that may take place in a setting local to the student. They can also substitute up to one course pertinent to medical informatics, from a local institution offering graduate studies.

As with any informatics application, future activities will require attention to overcoming our initial problems as well as to scaling to handle growth. Our initial problems have been minimal, but we have tried to keep up with technology changes (such as the revamping of the RealMedia product line with the newest version) as well as limitations of this teaching medium (such as using multiple-choice tests instead of trying to grade short-answer responses in Microsoft Word documents). We have also anticipated problems of scale by monitoring the load on the server and its network connection as well as hiring appropriate administrative and technical staff to handle more students and courses.

Another important direction for the program will be to keep up with changes in the needs and desires of students. We will continue evaluating each course with the instruments developed for this paper (in a similar way to what is done for on-campus courses). We have also established an e-mail listserv for students to voice their concerns and desires for changes. Certainly, as graduates emerge from the program and enter the marketplace, we will seek their feedback on how to improve the program to enhance job skills and employability. We will also aim to meet commonly accepted distance learning benchmarks, such as those developed by the National Education Association.¹⁰

Discussion

We have demonstrated that a distance learning program in medical informatics can be implemented successfully from technical, student satisfaction, and student learning standpoints. Building on a needs assessment from potential students, we were able to implement a program based on our existing on-campus curriculum that was highly accepted. Development of additional courses under the rubric of a graduate certificate program has already begun.

The lessons learned from this work indicate that there is strong interest in distance learning–based medical informatics education among many mid-career professionals in health care. The major concerns of students are related to the content, time flexibility in learning, and the way a course is delivered. We found that our existing curriculum was suited to meeting their need and that we were able to deliver it in a distance learning format with high student satisfaction. Although we did not measure results experimentally, it was apparent that online students obtained better test and term paper scores than did our on-campus students.

Distance learning does challenge students and teachers. Students need to make a commitment to keep up with class progress despite being busy with their regular jobs and other obligations. Especially for those who are busy professionals, an asynchronous approach using standard Web-based technologies (e.g., RealMedia, PDF documents, etc.), along with traditional textbooks and other paper-based documents, is preferable to real-time interaction. For teachers there are new challenges as well. For example, teaching asynchronously requires diligence in keeping up with e-mail and discussion boards. The inability to simply photocopy handouts and other documents requires more careful planning in the use of curricular materials.

Distance learning has the potential to revolutionize education in much the same way that other medical informatics applications are changing health care. Given the growing need for professionals in health care to learn about medical informatics, this approach to education using technology should have a significant impact. Future research must focus on identifying the topics that are most appropriate for different professionals to learn and the modalities by which they are most effectively delivered.