George Hripcsak, Columbia University—
Let me play devil's advocate. We have 15 years of studies, many of them randomized trials, that demonstrate beyond the shadow of a doubt that we can improve the process, and presumably the outcome, of health care through reminder systems. Yet today, less than one percent of hospitals use reminder systems. Will more studies make a difference ?
Bill Tierney—
No, we do not need more studies of bias reminders when we already have the answer. We know that when you remind people to do something they intend to do, they do it. Accordingly, we at Indiana University have programmed a comprehensive OB reminder system and never studied it. People have not implemented reminder systems because they do not have the critical mass of infrastructure to make them work. Without a computer-based patient record, we can implement only reminders for simpler things like mammograms or fecal blood tests, which require nothing more than knowledge of gender, age, and perhaps when the last test was performed.
Jim Harrison, Tulane University—
Two questions come up when studying the effect of clinical information systems on patient outcome. The first is “Did we do what we meant to do ?” The second is “Did what we meant to do really work for the patient ?” Information systems help us do a better job of achieving what we meant to do, and that allows us to measure whether what we meant to do really helped the patient in a way that was better than before. I would argue that in evaluating information systems, we want to look at things like guideline compliance (presuming that guidelines are codifying the standard of practice) and evaluate them. We then can go on to evaluate the various standards to determine whether they have useful outcomes in patient care.
Bill Tierney—
Operationalizing guidelines is very time-consuming. If we can demonstrate to ourselves and our peers that the use of guidelines has an effect on real patient outcomes, it encourages us to make that effort. To date, there is little evidence that process improvements resulting from reminder and guideline systems affect patient outcomes. We need to document that linkage.
Bill Stead, Vanderbilt University—
As I listened to Bill Tierney, it sounded as though he thinks we can evaluate IAIMSs if we would just do it. Yes, we don't seem to be doing it. We don't have a way of answering the question “Is it worth the cost ?” Is there a way the IAIMS Consortium could bootstrap IAIMS sites to teach them how to do this ? Put differently, what would be the outcome measure if we wanted to evaluate the effect of making MEDLINE available on workstations throughout the hospital ?
Bill Tierney—
In answer to your first question, evaluating an IAIMS is like evaluating a city. There are so many different aspects that I'm not sure you can look at the whole at once. I believe you have to look at the individual pieces of the IAIMS. Your second question is more cogent. We once designed a study to evaluate the effort of making MEDLINE available. We wanted to assess whether or not the appropriate articles were being accessed. We had a group of experts, both generalists and specialists, identify the articles that should be referred to for treatment of a patient with a particular condition. We could then look for the rates at which those articles were accessed. We could also look at lengths of stay in the hospital. One would expect less time spent casting around for appropriate information when there were easy methods of finding out the appropriate treatment. The last measure in our study was how well students did on their medicine tests. We first measured those who were randomly assigned to our hospital, as opposed to ones assigned to other hospitals, because they all had been exposed to the same educational program.
Bob Beck—
How would you evaluate the impact of the IAIMS on the third leg of the academic mission—the scholarly mission of research ?
Sherrilynne Fuller—
The University of Washington IAIMS program has served as an incubator for new research ideas and to encourage research projects. There are a couple of clinical genetics projects that directly grew out of the IAIMS effort. They were able to start under the auspices of IAIMS funding before they could compete independently for peer-reviewed funding.
Bob Beck—
That is the IAIMS proper. How about support of the research enterprise ? How do we solve problems such as how much supercomputing we support at the medical center, and how we're able to trade off the needs of the high-demand users versus the needs of people in scientific areas who just need word processing and spreadsheets ?
Bill Tierney—
I can provide two examples. The first is getting data for researchers. We can answer questions such as, “I want to do a study on x number of patients, how many do we have ?” The second example is less obvious. There is isolation in academia, and the IAIMS is one way of bringing about collaboration that otherwise wouldn't happen. It brings together electronically people who normally would not be interacting with each other, and it calls attention to problems that people often don't see.
Perry Miller—
IAIMS sites have focused on research in different ways. One way that we're focusing on the needs of the bioscience researchers is by trying to promote interdisciplinary research projects through development of things that are useful to the projects, like shared databases. We have tried to focus on collaborative research projects that have one or more laboratories at Yale and laboratories elsewhere. How would one evaluate such a resource ? You could monitor the level of use and find out to what degree you are meeting the perceived needs of the researchers.
Ed Hammond, Duke University—
Evaluation would be easy if we could agree on what we need to measure and what change makes a real difference in outcome. Take cost as an example. We don't know what the current process costs, we don't know what it should cost, and we don't know what we can afford to pay. Unitl we can answer those questions, it is difficult to prove or disprove the value of changing the process through the IAIMS.
Bill Tierney—
An IAIMS is required for efficient implementation of guidelines. To be useable, guidelines must be presented to the provider in a way that is tailored to the patient. Assessment involves tracking the effects of guidelines on the behaviors of individual providers and on clinical outcomes. If guidelines affect quality of care and cost, and if the IAIMS makes the use of guidelines possible, then the IAIMS is justified. The IAIMS is not going to be proven useful using criteria similar to criminal law, where we have to prove something beyond the shadow of a doubt. It is going to be more like civil law, where you have to show a preponderance of evidence.
Sherrilynne Fuller—
How do we use the IAIMS to better support the teaching mission, which is the core mission of many of our institutions ?
John Paton, Yale University—
One answer from our perspective is that we've always tried to give our students the same tools they would use as professionals—that is, as clinicians or researchers. In my mind, there is a great overlap between the needs of those two communities.
Don Detmer, University of Virginia—
The Institute of Medicine is hosting a preliminary meeting to look at the future of professionalism in medicine. The discussion that has taken place at the Institute of Medicine Board on Healthcare Services is that before we start talking about education or how to change the curriculum, we need to cast back and ask what are we educating to and for ? The profession needs to engage that issue with the academic environment. But where is that dialog ? The silence is deafening if you listen to it.
Bob Beck—
Morrison's “The Two Ocean War” says, “The general staff colleges keep training people to fight the last war.” We are definitely doing that in the health professions. The LCME self-study survey on information technology asks about networks and a learning resource center—things we were all doing five to ten years ago.
Kevin Johnson, Johns Hopkins University—
Sherrilynne, as a result of your planning process, what kinds of educational goals did you set and were they achieved or are they being implemented ?
Sherrilynne Fuller—
Our goals are evolving as our institution takes a careful look at what are we training to and what the practitioners today are saying they needed to get in medical school or nursing school but did not get. A random set of practitioners, now outside the academic medical center, all said front and center, “We need to teach our students to be better users of information technology in computing and to be lifelong learners.” We also have an enormous challenge because we're so distributed. How do you maintain quality of care across 100 clinical clerkship sites and multiple residencies, some that we manage and some that we do not. We credential and say that these people have achieved a University of Washington education. We're signing on the bottom line. How do you manage that in a decentralized environment ?
Kevin Johnson, Johns Hopkins University—
Are the residents who are graduating from IAIMS institutions more computer-literate by whatever metric we use to measure that ?
Bob Beck—
There would probably be a positive correlation, but it might not be great. We should take that up for next year and then get an answer to you.