Evaluating IAIMS at Yale

Abstract

Abstract Objective: To evaluate use of information resources during the first year of IAIMS implementation at the Yale—New Haven Medical Center. The evaluation asked: (1) Which information resources are being used? (2) Who uses information resources? (3) Where are information resources used? (4) Are multiple sources of information being integrated?

Design: Measures included monthly usage data for resources delivered network-wide, in the Medical Library, and in the Hospital; online surveys of library workstation users; an annual survey of a random, stratified sample of Medical Center faculty, postdoctoral trainees, students, nurses, residents, and managerial and professional staff; and user comments.

Results: Eighty-three percent of the Medical Center community use networked information resources, and use of resources is increasing. Both status (faculty, student, nurse, etc.) and mission (teaching, research, patient care) affect use of individual resources. Eighty-eight percent of people use computers in more than one location, and increases in usage of traditional library resources such as MEDLINE are due to increased access from outside the Library. Both survey and usage data suggest that people are using multiple resources during the same information seeking session.

Conclusions: Almost all of the Medical Center community is using networked information resources in more settings. It is necessary to support increased demand for information access from remote locations and to specific populations, such as nurses. People are integrating information from multiple sources, but true integration within information systems is just beginning. Other institutions are advised to incorporate pragmatic evaluation into their IAIMS activities and to share evaluation results with decision-makers.

The Yale—New Haven Medical Center is in the process of developing a comprehensive information environment based on the National Library of Medicine's vision of an Integrated Advanced Information Management System (IAIMS).¹ A fundamental goal of the IAIMS vision and of our efforts is to provide relevant information when and where it is needed in support of our missions of education, biomedical research, and patient care. In this paper we present usage data by type of information resource, type of user, nature of task, and location. We discuss the use of our data to guide investment decisions and to determine factors that affect utilization.

Background

Information Access at Yale

Located in New Haven, Connecticut, the Yale—New Haven Medical Center includes the Yale University Schools of Medicine and Nursing and Yale—New Haven Hospital. The Medical Center's primary missions of patient care, education, and research are pursued by a community of approximately 1,300 full-time faculty, 1,105 students, 900 post-doctoral trainees, 400 residents, 1,250 nurses, 3,350 full-time staff at the Hospital and 2,400 full-time staff at the schools (including 1,000 full-time managerial and professional staff^*).

The Medical Center has a campus-wide, high speed data network based on Ethernet and token ring segments. This network is connected to the Internet via the University network and thus provides access to both local and remote information resources.

Public Workstations

Public workstations are available for use by people as they move about the Medical Center. The user interface must have a similar framework in different locations so that retraining is unnecessary, and identification must be required for access to restricted information. At Yale, we developed menu software, NetMenu,² to provide this functionality. It logs user activity, it has an online survey feature, it supports scripting of connecting and disconnecting to online resources, it launches applications such as computer-assisted instruction (CAI), and it can be configured to take over the screen and hide the underlying operating system from the user while protecting the workstation from user changes. The same menu software is used on public library workstations (23 Macintosh and Windows-based devices) and on public clinical workstations in the Hospital (71 Windows-based devices). These two sets of public workstations share a common set of resources, such as MEDLINE, electronic mail, and the World Wide Web (WWW). We chose to tailor the content and structure of the menus to meet the different information needs, security requirements, and license restrictions in the two environments. The public library workstations include an array of bibliographic, full text, and Macintosh-based medical education resources, while the public clinical workstations provide access to hospital information systems for order entry and results reporting and a variety of online medical textbooks and guides, and procedure recorders. (In addition, there are 1,300 dedicated hospital workstations in high traffic nursing stations to enable staff to quickly query its patient care and clinical laboratory systems and MicroMedex.)

The Windows-based public library workstations also contain InfoFinder,³ an integrated information resource search tool created at the Yale Center for Medical Informatics and based on the National Library of Medicine's Information Sources Map (ISM), a component of the Unified Medical Language System (UMLS).⁴ InfoFinder enables users to search for and select appropriate online resources to query.

Private Workstations

All personal computers connected to the University network are configured with icons to access Ovid⁵ (the software delivering MEDLINE, CINAHL, 15 full-text journals, and two other bibliographic databases), Orbis (a NOTIS-based system containing the Yale Library catalog and three additional bibliographic databases), the World Wide Web via Netscape, electronic mail, the biomedical mainframe computer (to run molecular biology programs, statistical software, and other applications), the University personnel directory, the Nexis/Lexis databases, and FTP software.

These frequently used resources are also accessed from MedMenu, a Web-based menu developed by the Medical Library to mirror the menu on the public library workstations (with the exception of medical education software). This Web-based menu (http://www.med.yale.edu/medmenu) has many of the advantages of our original NetMenu. It runs on both Macintosh and Windows-based devices, it provides a common core menu of resources, and the core menu can be maintained centrally. In addition, the Web menu can be coordinated with the users' own customized menus of resources, and the Web software is in the public domain, whereas the NetMenu software requires commercial software components.

Evaluating Information Use and Its Impact Within an IAIMS Environment

Previous reports from other IAIMS sites^6,7 have emphasized a variety of institution-specific goals and related strategies for evaluation including cataloguing accomplishments^8,9 and lessons learned,¹⁰ analyzing costs and benefits,^8,9 assessing costs versus utilization,¹¹ reporting usage,⁹ and recording oral history.¹² In the present report we emphasize a detailed analysis of information use across classes of users, means of access, and purpose (e.g., clinical care).

To evaluate progress toward our IAIMS goal of providing information users need in the course of their work, we asked four basic questions:

1. Which information resources are being used?

2. Who uses information resources?

3. Where are information resources used?

4. Are multiple sources of information being integrated?

The questions build on one another and parallel stages in the development of an IAIMS environment. At the first stage, simple usage is monitored to determine how heavily information resources are being used. At a second stage we ask how different types of users (e.g., students or teachers) use these information resources. At a third stage we ask whether we are “providing information when and where it is needed.” Finally, we ask whether people are integrating information from multiple sources for a single task such as diagnosing a patient. Integration marks the most ambitious of IAIMS goals, and this report focuses on the simplest form of such integration—that which the users perform themselves.

Our evaluation strategy has two underlying principles:

1. Collect information necessary to act. Our evaluation efforts need to be focused and pragmatic; we need to have a concrete use for any data we collect. Our resources for evaluation are limited, so we need to be very careful to ask only those questions whose answers will influence later resource allocation. Furthermore, we do not want to burden members of the community or service providers with requests to participate in evaluation unless there is a clear benefit to them. We issue a monthly report on usage and trends (http://www.med.yale.edu/computing/acadcom/iaims/reports.html). The report is distributed to senior Medical Center administrators, library and computing professionals, and information providers. We have used the data to help us make decisions about the number and location of public workstations; about adding, deleting or modifying delivery of specific information resources; and about increasing offerings in categories such as full-text databases and medical education software.

2. Use multiple methods and take multiple measurements. By comparing results from a variety of evaluation methods, we assess the reliability of our data. We also conduct our evaluation in several phases to recognize trends.

This evaluation extends previous work at IAIMS sites in three ways. First, we collect and analyze usage in greater detail, examining patterns and trends in different environments and among different groups of users. Second, evaluation at Yale is an ongoing activity that is integrated into the management of information resources. Finally, we include users and non-users of online information systems in our evaluation activities to measure the impact of networked information upon the entire Medical Center community and to identify any barriers to our goal of benefits for all.

Methods

Measures

We collect evaluation information from usage logs, online surveys of users of public library workstations, a Medical Center-wide survey, and user comments.

Usage Data

Each month we collect usage data for major resources (e.g., MEDLINE, electronic mail) and for all information services available on the public library and clinical workstations (see Table 1 for a summary).

Table 1.

Usage Data Sources

Major Resources	Available on Public Library and Clinical Workstations	Available on Clinical Workstations	Available on Library Workstations
Current Contents	Bibliographic databases and search tools: 16 applications (1 Windows only) in Library, 4 in Hospital	Hospital information systems and recording tools (9 applications)	Grant and research information (4 applications)
Electronic mail	Clinical assistance (3 applications in Library, 1 in Hospital)		Medical Education (49 Macintosh applications)
Ovid databases: CINAHL, Health Planning and Administration, MEDLINE, and 15 full-text biomedical journals (e.g., JAMA, NEJM, Science)	E-mail/news/networks (7 applications in Library, 3 in Hospital)
WWW server	Full-text and factual databases: 30 applications (1 Windows only) in Library, 9 in Hospital
Yale biomedical gopher	Word processing and utilities (8 applications in Library, 4 in Hospital)

The most complete usage information is available for the Ovid databases (MEDLINE, CINAHL, Health Planning and Administration, and 15 full-text journals), Current Contents, the WWW server, and the Yale biomedical gopher. Those applications have internal logs and may be tracked network-wide. For each of the Ovid databases, we collect data identifying the users, their departments, the number of searches they conducted, the number of times they accessed Ovid, and whether they accessed Ovid from the Library or from some other location.

The Current Contents server logs the number of sessions, the user, the number of sessions that were conducted from within libraries, and the number of automatic searches that were run. (Current Contents enables users to specify that a specific search be run automatically each week when the database is updated. Users receive citations resulting from the automatic searches by e-mail.)

Usage data for the Medical Center World Wide Web server are generated by MUSAGE (http:///www.blpes.lse.ac.uk/misc/musage.htm), a perl script created at the London School of Economics and Political Science. MUSAGE logs the total number of pages accessed, the number of times each page is accessed, and the IP address accessing each page (enabling us to track usage from Yale, other educational institutions, major network providers, U.S. commercial sites, U.S. non-profit sites, U.S. government sites, and foreign countries).

Usage logs for the Yale biomedical gopher record the number of times Yale users and the number of times external users access each gopher document.

Although there are several desktop- and mainframe-based electronic mail systems in use at the Medical Center, most e-mail traffic is processed through the School of Medicine's VAX 7610 minicomputer by PMDF-MTA¹³, a mail transfer agent from Innosoft International. PMDF logs the number of messages sent to and from the various email systems. Mail sent to the Hospital and some internal desktop electronic mail messages are not processed (or logged) by PMDF.

Usage data for the information resources accessed through the public library and clinical workstations are generated by the NetMenu software. Each time a user clicks on a “connect” button to access an information resource, the menu software logs the date, time, machine, and resource name.

As objective as they might seem, usage data are subject to limitations and measurement error. Our electronic mail data are almost but not quite complete. The Web data include accesses from people developing and testing the pages and so at times usage for individual pages or directories is artificially inflated. For some resources, we can only track usage from public workstations because no usage data are collected at the resource level. Our means of logging application usage from public workstations is dependent on counting the number of clicks on the `connect' button. Examining the Ovid usage data has taught us that this does not always mean a user actually uses the application. Finally, on rare occasions, logging is disrupted and usage goes uncounted. With these limitations in mind, we interpret most usage data in relative rather than absolute terms, examining trends over time, comparing usage levels by different users of the same application, or comparing usage levels of different applications collected by the same mechanism (e.g., the public menu logs).

Online Surveys

We wrote a questionnaire in Visual Basic to survey users of public library workstations. When a survey is active (typically for short periods of time), the NetMenu software displays the survey every 20th time someone tries to connect to a resource. The user must complete the survey before being connected to the resource. To lessen the interruption of work, the survey occupies a single screen, and consists of only four or five questions. Questions to provide demographic information are repeated during each survey period. Topical survey questions vary from one survey to another. Using this online survey we collected over 900 responses during three 2-week surveys between December 1994 and May 1995, and 321 responses over a 5-week period in the spring of 1996.

Medical Center-wide Survey

We administer an annual questionnaire to a random, stratified sample of 240 members of the Medical Center faculty, postdoctoral trainees, students, nurses, residents, and managerial and professional (M&P) staff. The questionnaire enables us to learn what proportion of the Medical Center community are users, to learn how various subpopulations use IAIMS resources and to identify barriers to usage.

The survey consisted of questions about library use (whether respondents used library resources, libraries used, frequency of use, purposes for using) and computer use (whether respondents used computers, locations where they were used, frequency of use, purposes for using, specific applications used, and computer ownership).

The names of faculty, postdoctoral trainees, and M&P staff were drawn from the University's Human Resources database. Students' names were drawn from medical, nursing, physician assistant, and graduate student enrollment lists supplied by their registrars. Residents' names were supplied by the Hospital's Medical Staff Office. The Hospital drew a sample of nurses registered in their patient care system database. Samples were drawn by selecting every xth person from the target population, where x = (population size)/(desired sample size). Every original list or database was sorted alphabetically by last name except the populations in the Human Resources database, which were sorted by university ID number (social security number except for foreign nationals). We mailed people up to three copies of the survey, over a period of 6 weeks, until they returned a complete survey. People who did not respond to any of the three mailings were telephoned as often as three times and asked to either return the survey or complete it over the phone.

With the possible exceptions of medical students and nurses, the people who responded seemed to be representative of the Medical Center. The overall response rate was 80%. Only 55% of medical students and 35% of nurses responded. (Our sample of nurses was poorly drawn, as five of the nurses had left the Hospital, and another four could not be located for the telephone calling. We later learned that the database from which they were drawn included ex-employees for record-keeping purposes.) Of the 49 people who did not complete the survey, 9 had left the Medical Center (including 5 nurses), 10 were out of town for the summer, 9 had unlisted or disconnected telephone numbers, and 21 (including 6 students and 5 residents) did not return the phone calls. A preliminary analysis to determine whether people who returned their surveys earlier had different patterns of library or computer use than later respondents revealed no significant differences, suggesting that non-respondents neglected to participate for reasons unrelated to the survey's content.

User Comments

We value written and oral feedback from our users as a means of validating our quantitative results, suggesting issues to explore in the future, and, most important, confirming benefits implied by usage data. We collect users' comments (1) in the surveys mentioned above, (2) with an electronic suggestion box on the public library workstations, and (3) from reference librarians and computing user support professionals.

Validity

We found a high degree of validity in our data, based on comparing current results with similar questions from different evaluation sources. We compared eight measurements from the present surveys and usage data with results from surveys and inventories that we had conducted in the past 5 years. We chose measurements that were unlikely to have changed over the time intervals of the comparisons to avoid confounding validity with trends. Comparisons included data on computer usage and ownership, word processor and medical education software usage, interest in electronic journals, and points of access needed. None of the comparisons was significantly different using binomial tests; the median difference was 3.5 percentage points, ranging from a minimum difference of 1 percentage point (for computer ownership) to a maximum of 11 (for desire for off-site access to network resources).

Results

Which Information Resources are Being Used?

By all measures, Medical Center information resources are being used widely and with increasing frequency. For most resources, usage follows a yearly cycle and increases from year to year. Figure 1 displays usage per month of the public library workstations over the past 2 years. The fluctuations in usage from month to month follow a predictable pattern for an academic institution; however usage over the past year has increased an average of 14% from the previous year.

Figure 1.

Use of Library NetMenu workstations: total connections per month. (Source: usage data from library's 21 workstations).

Public workstation usage in newly established sites is increasing more dramatically. The School of Nursing's NetMenu workstation was used 85% more in 1996 than in 1995. The Hospital has undertaken a major expansion of its public clinical workstations, adding more workstations (from 16 to 71 with at least 30 more planned) and applications (from 12 to 30 so far) and training users. With so many additional workstations, an overall increase in usage is to be expected. What is noteworthy is the increase in public clinical workstation monthly usage per workstation (Fig. 2). Individual public clinical workstations were used an average of 75% more in the summer and fall of 1995 compared with 1994.

Figure 2.

Use of Hospital NetMenu workstations: connections per workstation per month. An asterisk indicates an estimated figure due to interrupted logging. (Source: usage data from clinical workstations).

Use of some of the major resources accessible over the entire Medical Center network is also growing (Fig. 3). MEDLINE use is increasing by more than 30% per year, from 8,536 sessions in May 1995 to 11,259 sessions in May 1996. Although we have only recently begun tracking use of the Medical Center's World Wide Web server and electronic mail, those services are also expanding. Use of the Yale biomedical gopher has begun to level off as the Web grows in popularity among users and information providers. About 15% of usage of the Web server is from within Yale; the remaining 85% of usage is external. MedMenu is one of the sites most heavily used by both Yale (300-500 times a month) and external users (700-1,000 times a month). Other frequently used sites include those of the Medical Library and academic departments for Yale users, and the Medical Center and School of Medicine main pages, the Center for Advanced Instructional Media's Web Style Manual, information about applying to the School of Medicine, academic departments, and the Library for external users.

Figure 3.

Use of selected resources.

Who Uses Information Resources?

Use of information resources is widespread and high, according to our 1995 survey. Most of the Medical Center community is using library resources (82% of all faculty, students, nurses, residents, and managerial & professional staff), computers (96%), and networked information resources (83%).

Electronic information and tools are as varied as the people who use them. Resources range from tools and applications that are potentially useful to almost all of the Medical Center community, such as electronic mail, to specialty-specific applications such as molecular biology software. In our analysis we focused on network based information resources which are more indicative of an IAIMS environment.

Who uses global applications and tools? We asked annual survey recipients whether they used such net-worked applications as MEDLINE, the University library catalog, Internet access tools (e.g., Netscape), molecular biology computing tools, and the Business Management System, a networked collection of administrative databases. Use of networked resources varied among subpopulations from only 57% of nurses to 100% of students, residents, and research faculty (χ² = 31.98, ρ >.001) (Table 2).

Table 2.

Use of Networked Applications

Status	% Using	N
Research faculty	100	19
Students	100	165
Residents	100	13
Teaching faculty	94	16
Post-doctoral trainees	88	16
Clinical faculty	87	23
Research M&P staff	80	20
Administrative M&P staff	61	19
Nurses	57	7

Special resources for specific populations. Next we examined usage of resources whose primary purpose is to support either research, patient care, or education to measure the impact of IAIMS on the primary missions of the Medical Center. For example, how many physicians are using clinical information systems? As use of the relevant class of resources by researchers, clinicians, or teachers and students becomes wide-spread, IAIMS becomes integral to achieving the Medical Center's primary missions. Low use of relevant resources becomes a warning signal that we may need to look for possible barriers to information delivery of those resources to the people who need them. To explore this issue, we first looked at mission-related uses of the public library workstations. Then we moved beyond users to the Medical Center community by examining questionnaire results.

Based on our May 1995 online survey of public library workstations, most usage is clearly related to support of research, clinical care or education, with research being the most common goal. Almost half (46%) of all 321 respondents reported using the workstations to find information for a research project. One fifth (20%) of respondents were looking for course-related information, and 16% were looking for information to treat a patient. Analyzing the data separately for faculty, postdoctoral fellows and residents, and students, we learned that although research was still the primary reason for using the public library workstations, the three groups had quite different information needs (Table 3).

Table 3.

Reason for Using Public Library Workstations

	Faculty	Postdoc/resident	Student
N	66	44	165
Research	49%	66%	35%
Education	23%	16%	19%
Patient care	32%	25%	11%
Electronic mail	0%	5%	34%
Total	∼100%*	∼100%*	∼100%*

^*Totals slightly exceed 100% because 8% of respondents reported using the workstation for more than one reason.

Faculty were the most likely to be using library workstations for patient care and students the least likely (χ² = 15.9, ρ <.001). Significantly more postdoctoral fellows and residents, and fewer students, used library workstations for research (χ² = 14.5, ρ <.001). There were no significant differences in the groups' use of library workstations for education. Students had a fourth major reason for using the workstations: electronic mail. The library is many students' primary point of access for electronic mail. Forty-three percent of medical students use email solely from the library, and only one-third of medical students connect to their email accounts via modem from home.¹⁴

We analyzed data from the annual Medical Center survey to learn the extent to which researchers, teachers and students, and clinicians were using information resources relevant to their domain.

For people conducting research (research faculty, post-doctoral trainees, research M&P staff, and graduate students), we selected five resources: using the library to find research information, using computers to find research information, analyzing research data, using computers to prepare grant proposals, and molecular biology computing. Overall, significantly more researchers than non-researchers used four of the five research resources (χ² ranged from 9.476-17.388, ρ <.01), but we found no differences for use of computers to prepare grant proposals (χ² = 1.71, ρ =.191). More graduate students (50%) reported using molecular biology computing than other respondents, followed by research faculty and research M&P staff (∼27%). Fewer than 10% of post-doctoral trainees used molecular biology computing.

For patient care, we compared residents, clinical faculty, and nurses with other respondents on their use of libraries to find clinical information, of computers to find clinical information, and of the Hospital's CCSS patient care system. In all cases, significantly more patient care providers used those resources than non-clinicians (χ² ranged from 14.07-52.57, ρ <.001). However, nurses were far less likely than clinical faculty or residents to use libraries or computers to find clinical information. Residents were heavy users; each resource was used by over 80% of residents.

We compared teaching faculty and students with other respondents in their use of three educational resources: computers to prepare presentations, libraries to prepare for courses, and medical education software. As a whole, teachers and students were significantly more likely to use educational resources than all other respondents (χ² ranged from 4.02-18.82, ρ <.05). Physician assistant (PA) students were not significantly different from non-educational respondents in their use of computers to prepare presentations. The primary users of educational software were PA students and medical students; fewer than 20% of all other respondents reported using it. Over 70% of medical students used each of the three resources.

We drew three conclusions from our analyses of why resources are used:

1. IAIMS resources are being used to support research, clinical care, and education.

2. There is room for growth. For example, only about two-thirds of research faculty are using computers to prepare grant proposals, only 71% of nurses are using the Hospital's online patient care system, and, only 12% of nursing students are using medical education software. We need to determine whether we can improve tools, training, or points of access to facilitate these groups' use of these resources.

3. Most resources have secondary, as well as primary uses and users. For example, although clinicians are the primary users of clinical systems, the same information is also used by students as part of their training, and by faculty for clinical research.

Where are Information Resources Used?

Where are people using computers? From the 1995 annual survey, the most common location was University offices, where 75% of non-students reported using computers. More than half of respondents said they used computers on the Hospital floors (clinicians only), at home or in the library (Fig. 4). Eighty-eight percent of respondents use computers in more than one location, with some using them in as many as seven.

Figure 4.

Where people use computers.

Use of the MEDLINE database dramatically illustrates what people do when barriers to information access are removed. Used by an estimated 72% of the Medical Center community, MEDLINE is the Medical Center's most popular database. Several recent efforts have made MEDLINE more accessible: (1) The introduction of OVID MEDLINE in April 1993 was the first time that a full MEDLINE file was provided on a networked system without direct charges for Medical Center users. (2) The Hospital, which added MEDLINE to its public clinical workstations in 1994, is in the process of increasing its public clinical workstations from 6 in June 1993 to a present total of 71. (3) The School of Medicine created a Desktop Computer Support Unit to provide and support standardized tools for connecting to networked resources, including MEDLINE. (4) The University and the Hospital are improving tools for connecting from off-site locations. While Figure 3 demonstrates that overall MEDLINE usage has been increasing, the trend lines in Figure 5 reveal the growth is due to the fact that people are increasingly assessing MEDLINE from Hospital, office, laboratory, and home computers (R² =.76). We estimate that 8% of non-library use is from public clinical workstations, and 92% is from office, laboratory, and home computers.

Figure 5.

MEDLINE usage. Source: MEDLINE usage data.

Are Multiple Sources of Information Being Integrated?

We have three indications that people are integrating information from multiple sources. First, in our December 1994 online survey of users of the public library workstations, over half of the 325 respondents reported using more than one application during their session (Fig. 6). We are reasonably certain that these people are using multiple applications to meet a single information need, because in our May 95 online survey, 92% of respondents were using the workstation for a single purpose.

Figure 6.

Number of different applications used in a single session.

Second, most respondents (79%) in the 1995 annual survey reported using both libraries and computers when looking for either clinical or research information. A small percentage (17%) use only libraries and only 4% use only computers to find such information. Third, 79% of the respondents to the online survey on InfoFinder considered InfoFinder to be a useful tool. People most commonly use InfoFinder to supplement MEDLINE and online catalog searches for research information (χ² = 13.3, ρ <.001). People using InfoFinder for this reason rated it as more useful than people using it for other reasons (χ² = 6.02, ρ <.05), such as searching for patient care information or needing to find information but not knowing exactly where to look.

Discussion and Conclusions

Factors Affecting Usage

Many factors help determine whether an information resource is used. Our experience and the present evaluation lead us to believe that cost, location of workstations, use of menus and search tools, and developments in information technology all affect usage.

Providing information when and where it is needed apparently led to increased use, as indicated by the dramatic increase in use of information resources when public clinical workstations were introduced in the Hospital. We hypothesize that what has changed is not the number of times people would like to consult MEDLINE, but instead the number of situations in which they are able to do so easily.

Improved access also affects how work is done. For example, a professor of cell biology has told librarians that he no longer uses a reprint file now that he can conduct MEDLINE searches whenever and wherever he needs to. He has also begun advising his graduate students to use MEDLINE searching as a replacement for their own reprint files. An emeritus professor of genetics conducts a regular monthly search of MEDLINE to update a comprehensive bibliographic database. He performs the search at the most convenient location, which might be his office, his home in New Haven, or his home in Florida.

Menus help users select and access frequently used resources. In addition menus highlight important online information resources and thus encourage their use. In one experiment we moved the InfoFinder to the top of the menu and it moved from the eighteenth-most-used resource to the eighth-most-used resource. This suggests that there may be some merit in customizing menus for different environments, as we have done for public library and clinical workstations, and in defining an explicit goal or strategy for organizing the content of menus.

Search tools such as InfoFinder and various Web search engines are of great help to the users we surveyed in finding the ever-increasing number of online resources beyond MEDLINE.

Areas for Improvement in Information Delivery

Our evaluation has identified several areas for improvement in access to online information. We need to improve the delivery of specific resources to particular subpopulations. For example, we need to learn whether problems with access, training, and/or resources themselves are limiting nurses' use of networked information resources. Addressing that issue, Yale—New Haven Hospital is adding electronic mail access to its public clinical workstations and is providing nurses and residents with e-mail accounts.

Although people are using computers in a variety of locations, there is clearly room for improvement in outreach, especially in providing access to information resources in outpatient care settings and Hospital offices, where fewer than half of clinicians currently use computers. Anecdotal data suggest that locating electronic information resources in the midst of a patient care setting can provide indirect as well as direct benefits. For example, one Hospital resident reports he can now make productive use of slow moments on the floors by using nearby public clinical workstations to conduct MEDLINE searches and access Internet resources.

Integration Issues

Although most integration currently occurs in the minds of people, integration among information systems is beginning to occur in some situations. The library has an online database system from OVID that provides access to MEDLINE, CINAHL and a core collection of 15 biomedical journals (text plus graphics). People can use the OVID interface to run the same search strategies on several databases and, with a keystroke, move from a MEDLINE citation to the full text of the article if it is one of the 15 core journals. Even when MEDLINE sources are not among the online journals, users can eliminate a visit to the library by requesting a document via a Web form. We may be able to achieve another advance in integration with the release of Web front-ends to all our bibliographic and many full-text databases, including MEDLINE and Yale's online library catalog. We hope to experiment with adding links to InfoFinder on search result pages to suggest additional information sources to users if they need them.

Integration also occurs within clinical systems. Most departmental clinical systems can now be accessed through a common order entry and results reporting system for inpatient care. In addition data repositories are being developed to consolidate clinical data within the Hospital and to consolidate financial and clinical data within the Medical School.

Plans for Future Evaluation Activities

In our second year of IAIMS implementation, we are continuing our basic evaluation strategies of collecting evaluation data based on its capacity to inform the decision-making process and of using multiple measures at multiple times to improve reliability and assess trends. However, as our IAIMS environment matures and as the nature of information technology and applications evolves, those strategies are beginning to lead us in new directions. We are changing the format of our IAIMS monthly report to focus more on overall use (network-wide and remote) and less on use at any one location such as the library. Upcoming evaluation projects will explore the effectiveness of the World Wide Web and the impact of medical logic modules on clinical care.

We believe we have begun to answer the questions: which resources are being used, who uses resources, and when and where are resources being used? We are less satisfied with our knowledge of integration and its impact, primarily because we are still in the process of developing integrated applications. Thus evaluation of the impact of systems integration will be a goal of the next stage of our evaluations.

We are also designing an evaluation of the impact of information delivery on research, patient care, and education. We will begin by assessing users' perceptions of the impact of information resources on their work, and then we will try to relate specific outcomes measures to specific changes in the information environment.

Concluding Remarks

We believe our evaluation is not simply documenting the impact of IAIMS upon the Yale Medical Center community's access to information; it is also shaping that impact. We ensure that key decision-makers receive and discuss evaluation information by regularly putting it on meeting agendas. Evaluation data have helped justify initiating projects (providing connectivity to the medical students' dormitory), expanding projects (the Hospital's deployment of public clinical workstations), and refining projects (a series of computer-based medical education quizzes).

The benefits we have derived from our evaluation lead us to encourage other IAIMS sites to incorporate evaluation into their own activities. Specifics will differ, but we believe the key components to a successful evaluation are collecting data that will be used to make decisions, sharing and discussing evaluation data with administrators and other decision-makers, instituting processes for the ongoing compilation of information about usage and users, mounting formative evaluation studies on an as-needed basis, and periodically reviewing the evaluation process itself.