Training Synergies Between Medical Informatics and Health Services Research

Abstract

Stanford's two decades of success in linking medical informatics and health services research in both training and investigational activities reflects advantageous geography and history as well as natural synergies in the two areas. Health services research and medical informatics at Stanford have long shared a quantitative, analytic orientation, along with linked administration, curriculum, and clinical activities. Both the medical informatics and the health services research curricula draw on diverse course offerings throughout the university, and both the training and research overlap in such areas as outcomes research, large database analysis, and decision analysis/decision support. The Stanford experience suggests that successful integration of programs in medical informatics and health services research requires areas of overlapping or synergistic interest and activity among the involved faculty and, hence, in time, among the students. This is enhanced by a mixture of casual and structured contact among students from both disciplines, including social interactions. The challenges to integration are how to overcome any geographic separation that may exist in a given institution; the proper management of relationships with those sub-areas of medical informatics that have less overlap with health services research; and the need to determine how best to exploit opportunities for collaboration that naturally occur.

Training in medical informatics and health services research has been closely linked at Stanford University for almost two decades. Although the close linkage was deliberate, it was facilitated by historical circumstances, in particular the common academic structures in which both programs arose. In this paper, we describe some of that rationale and history, identifying the areas of overlap that we have pursued in coordinating the training opportunities for graduate students and fellows in both areas of study. As we shall note, the synergies have been great, and in some cases trainees have collaborated closely on research while also taking some of the same courses. We believe that these interactions can be a model for the design of training programs that encourage scholarly interactions between medical informatics and health services research. Although our initial charge was to describe both the successes and failures in integrating the programs, we found that we could not identify any outright failures and that it would be better to delineate the complexities and challenges that we have faced in bringing together these two disciplines.

Setting and Historical Perspective

The Stanford training programs in medical informatics and health services research have benefited from the geography of the educational and research environment. The medical school is located on the university campus, immediately adjacent to the engineering school (where computer science is housed) and also near relevant departments such as economics, psychology, and statistics. Courses in the school of medicine and other schools also follow the same schedule (standard class times during the day are the same, and the beginnings and ends of academic sessions and examination periods are also identical), which has facilitated classroom coordination for students who want to take some courses in both the medical school and elsewhere on the main campus. Novel programs have also benefited from a university philosophy that encourages cross-disciplinary degrees and cooperation among departments and schools.

It happens that both the health services research and medical informatics training programs arose from the same division in the department of medicine (the division of general internal medicine), and the similarities in clinical interests among faculty and many of the trainees also facilitated some of the interactions we describe here. Also important was a shared recognition that quantitative skills are important in these two fields; both the curricula for the programs and the skills of recruited trainees reflect this philosophy. Some of the required courses were also shared between the two programs as they evolved, so there are substantial cultural similarities and shared skill sets among our trainees and among the faculty.

Medical Informatics Training

The historical roots for our informatics training program date back to the early 1970s and the initiation of a federally funded research resource called SUMEX-AIM (Stanford University Medical Experimental Program for Artificial Intelligence in Medicine).¹ SUMEX was created to serve as a national resource, accessed over the early version of the Internet, but much of the research was carried out at Stanford. Over the years, many people with an interest in biomedical computing came to Stanford, drawn by that resource, and began to look for ways that they could acquire formal training at the interface between biomedicine and computer science. The first students created their own interdisciplinary degree programs (a special opportunity that Stanford provided for graduate students whose scholarly interests did not readily fall within any single existing department). Growing interest led in time to the creation, in 1982, of a formally constituted interdisciplinary degree program that offered MS degrees and PhDs in medical informatics.^2,3 A few years later, we received a training grant from the National Library of Medicine that continues to support both pre- and postdoctoral training in medical informatics at Stanford.

Because of its research roots in artificial intelligence, the training program began with an emphasis on clinical decision making. In time, however, as students developed other interests, took courses in other parts of the university, and chose to do research in areas that differed from some of the program's historical roots, we enhanced the scope and the core curriculum of the program to include broader coverage of medical informatics, including additional application areas such as imaging, population health, and bioinformatics.

With experience in training students, our philosophy evolved to view medical informatics as a core scientific discipline with many areas of application.⁴ We chose to use “medical informatics” as the name for a set of techniques, theories, and methods that form a basic research discipline. Like many basic fields, it is driven by applications that motivate scientists to work on the pertinent fundamental research problems. We use such terms as “clinical informatics,” “imaging informatics,” “bioinformatics,” and “public health informatics” to refer to those applied research domains, all of which draw on the scientific underpinnings embodied in the field of medical informatics.⁵

Among the applied research areas in medical informatics, two have particularly strong overlap with health services research—public health informatics (and related population-based work) and clinical informatics. The boundaries are imprecise, however, since the population databases in public health informatics are often composed of data sets drawn from clinical systems used in patient care settings.

Figure 1▶ provides some insight into the kinds of methods and theories that drive the core discipline of medical informatics.^* The key notion is that informatics involves the use and management of biomedical knowledge, including its codification, and the use and management of data acquired in some applied domain of biomedicine. The goal is generally to draw suitable inferences from those data using the knowledge of biomedicine that is pertinent. As the diagram shows, this conceptualization of the field helps identify a wide range of research topics that have broad applicability across biomedicine. Furthermore, techniques developed in response to a perceived need in one area of biomedicine—say, in the field of bioinformatics—may well be found to be relevant in one of the other areas of application, such as clinical medicine or population health. This is one of the strong reasons for keeping all informatics training focused within single academic units, even though individual students may have diverse areas of application interest. The core scientific issues are generally the same, despite the differences in biomedical motivation for methodology development.

Figure 1 .

Medical informatics research areas. Many of the methods addressed by medical informatics researchers are reflected in this diagram. Diagram based on a concept originally developed by Gregory F. Cooper. See also Shortliffe.⁷

Health Services Research Training

There are many ways one could date the beginning of health services research at Stanford. A master's program in health services research had been established in the 1970s, evolving from a curriculum initially emphasizing health care administration. The master's program trained medical students and others with a wide range of interests and backgrounds, and the faculty participating in the program represented an equally diverse range of disciplines. A watershed event for the health services research environment came with the establishment of the Robert Wood Johnson Clinical Scholars' Program, which started in the 1970s as a joint program with the University of California at San Francisco (UCSF). The Clinical Scholars Program did not require enrollment in a degree-granting program at Stanford, and it was always administratively separate from the formal health services research training program. Nevertheless, it formed an important focus of health services research activity at the university.

A second key event occurred in the 1980s, when the Department of Health Research and Policy was established as a freestanding department in the school of medicine. The department functioned rather like a small but vigorous school of public health that has very strong connections with the rest of the school of medicine. It accordingly became the home for much of the health services research activity at Stanford, even though many of the researchers and instructors in health services research were based in other departments.

A third event further stimulated health services training at Stanford and more closely linked the health services research master's program with postdoctoral training. In the mid 1980s, under the directorship of Dr. Harold C. Sox (then chief of our division of general internal medicine), Stanford was awarded a training grant in health services research from the National Center for Health Services Research (soon to be reincarnated as the Agency for Health Care Policy and Research, which in turn has become the Agency for Healthcare Research and Quality, or AHRQ). Professor Garber has directed this program for more than 10 years. Like the informatics training program, the Stanford Fellowship in Health Care Research and Health Policy was established as a combined predoctoral and postdoctoral program, although the majority of our trainees have always been postdoctoral—typically, post-residency—physicians. They obtain at least a master's degree. Although fellows can choose to get a PhD or an MS degree in another discipline related to health services research if they are accepted into the relevant departmental degree program, nearly all have enrolled in the health services research master's program, sometimes in addition to a PhD program.

From the outset, there was a very tight link between the Fellowship in Health Care Research and Health Policy and the medical informatics program, including overlapping course requirements, professional and personal relationships among the trainees, and joint annual retreats. Subsequently, members of our faculty who were based at the Stanford-affiliated Palo Alto Veterans Administration Medical Center (now the Veterans Affairs Palo Alto Health Care System) were successful in attracting support for VA-based fellowships in ambulatory care, medical informatics, and health services research and development. Many of these fellows became degree candidates in health services research or medical informatics, so there has been copious interaction and integration among the various programs and their trainees.

Ongoing research projects in health services research at Stanford have played a central role in enriching the training environment. Examples include the cardiac arrhythmia Patient Outcomes Research Team (PORT II), part of a large program supported by the AHRQ. The team was directed by Dr. Mark Hlatky and involved trainees from both the health services research program and the medical informatics program. Their experience is a good example of the high level of interactions that occur among these groups. It also shows us how, drawing on both health services research and informatics, a clinical issue (e.g., how high-grade ventricular arrhythmias should be treated) can be used to develop a comprehensive set of studies and instruments that will lead to practical use.

The University of California at San Francisco joined Stanford to form one of the AHRQ evidence-based practice centers. The program has engaged several health services research trainees. Similar training opportunities are available through support by the National Institute of Aging for our Center for Demography and Economics of Health and Aging. Equally important are our links to the Department of Veterans Affairs (VA) health services research and development field program and the Center for Health Care Evaluation. More recently, the VA Palo Alto Health Care System has become home to the VA-wide Health Economics Resource Center, whose members are superb health economists with considerable expertise in the analysis of large observational databases and the development of methodology for imputing costs from clinical and administrative databases. Close ties to the VA have been a very important tool to gain us both people resources and access to the clinical data systems of the VA, which are in some respects quite extraordinary and fertile ground for research projects by trainees in both informatics and health services research programs.

Also important to our current training opportunities in health services research are two new centers housed together at Stanford, both of which are directed by Dr. Garber—the Center for Health Policy and the Center for Primary Care and Outcomes Research. The two centers are housed and operated together because of their common interests, methods, and data sources. They are separate centers because of the unique expertise and audience of each within and outside the university. Programs of the Center for Primary Care and Outcomes Research apply health services research to improve clinical practice and public health, whereas the Center for Health Policy conducts interdisciplinary research to inform pressing public and private policy decisions. Our fellowships in health services research are now administratively based in the Center for Primary Care and Outcomes Research. By coordinating the activities of these two centers and drawing on the wide range of expertise available in the university, we are able to assemble research teams and train in a full spectrum of areas. These range from clinically oriented patient-based research, decision analysis, and guideline development to broad health policy research questions, such as what is happening to Medicare expenditures and how various policies to control Medicare expenditure growth might work.

The Stanford Curricula

To illustrate how medical informatics and health services research are enmeshed operationally at Stanford, we briefly summarize the curricula for the two training programs. Some of the course offerings naturally reflect the interests of our faculty and of the trainees who choose to come to Stanford, but most are broadly based and suggest curricular approaches that may be appropriate for any institution seeking to formulate interacting and complementary programs in medical informatics and health services research.^†

Medical Informatics Curriculum

The design of our informatics curriculum starts with the assumption that there are core disciplines that students need to learn to be broadly educated in the field. The diversity of the disciplines makes it a heavy bill to learn something about all of them. However, many of our incoming students have backgrounds in computer science, medicine, or one of the other health professions, allowing them to place out of some of the requirements we describe here. Others, with more limited backgrounds, have to work their way through the course requirements in essentially all the major categories (computer science software and hardware, bioengineering, basic biology and clinical medicine, cognitive and decision sciences, epidemiology, statistics, and management issues).

With the exception of computer science and bioengineering, these topics are also key foci for study in health services research (especially as the topics relate to health policy and clinical medicine).

We have organized the curriculum in informatics into five categories—medical informatics itself, computer science, decision science, biomedicine, and public policy or social issues.³ For the medical informatics component, all students get some exposure to bioinformatics and imaging informatics as well as clinical applications. We have maintained an emphasis on decision making and quantitative skills, and everyone gets some programming experience in addition to the computer science requirements.

We also emphasize experience in giving formal talks, attending scientific colloquia, and learning a bit about the realities of federal funding and similar “civics” issues. We have also written a textbook for teaching introductory medical informatics.⁶ Decision science courses include data analysis, the use of statistical packages, probability theory, decision analysis, cost-benefit and cost-effectiveness analyses, and the psychology of human problem solving. We also require some exposure to courses on health and society, public policy, technology assessment, health economics, medical ethics, scientific integrity, and ethical issues as they relate to computing in general and to medical informatics in particular.

The Stanford informatics program has a steady-state size of about 30 students. We get applications from more than 100 students a year, of whom we take four to six new students, depending on how many are graduating. There is a core faculty of about nine people, but 30 to 40 additional faculty throughout the university work with our students or have announced their willingness to work with our trainees. Many of these faculty members are also involved as participating faculty for the health services research training program. We have about 80 graduates to date, roughly half of whom have earned PhDs and the rest MS degrees. About half are in academic positions and the other half in industry, hospitals, government positions, or clinical practice.

Health Services Research Curriculum

Stanford University requires 36 to 45 units of credit for students to earn a master's degree. The health services research master's degree requires 45 units of credit, distributed in any of several tracks. Although there are very few degree-wide course requirements, the interests of the core faculty and other researchers strongly influence the courses that typical trainees take. Thus, compared with the curricula at many other institutions performing health services research training, our program places a heavy emphasis on economics, decision analysis, and quantitative skills. Although we believe that qualitative research has an important role, it is not a focus of our core faculty. The quantitative emphasis may further account for our tight linkages with informatics. We expect our trainees to graduate with rigorous quantitative skills, and our applicants are informed of this expectation.

Trainees in the Fellowship in Health Care Research and Health Policy are required to take four core courses; they must take these courses regardless of whether they are formally enrolled in the health services research master's program, and these courses are not required of all health services research master's students unless they are also enrolled in the fellowship. These courses are then further supplemented by elective courses or by degree requirements, depending on the degree program in which a student is enrolled.

The first course, "Analysis of Costs, Risks, and Benefits of Health Care," has been offered for many years in the Graduate School of Business, where it was started by Professor Alain Enthoven. For several years, Professors Enthoven and Garber taught the course jointly, but with the retirement of Professor Enthoven, the course is now taught by Professor Garber and two other faculty from the Center for Primary Care and Outcomes Research, Douglas Owens and Gillian Sanders. We have also required a course on computer-assisted medical decision making, offered by the faculty from the medical informatics group. For the third course requirement, in decision analysis and statistics, we have a large number of courses at Stanford from which trainees can chose the most suitable. Finally, we offer a choice between two health economics courses, one that emphasizes large-database outcomes research and the other focusing on the political economy of health care in the United States.

If we had a school of public health at Stanford, we might have had a large number of custom-made courses for our trainees. We have never attempted to do that. Lacking the size and ability to grow of some other institutions, Stanford has both the geography and administrative structures to promote collaboration across departments and schools within the university. Our philosophy has been that our trainees should learn statistics from a world-class statistician in any part of the university (they can be found in the Department of Health Research and Policy, the School of Education, and elsewhere, in addition to the statistics department), economics from top economists, and so on, even though these faculty may not have a specific interest in health services research or even health-related subjects. Thus, our trainees take courses throughout the university, with all the advantages and potential disadvantages that this implies. For example, when a trainee is learning a methodology, the examples may not be taken from medicine or health services research at all, since the course may be offered without a biomedical emphasis or motivation (e.g., in the school of education, in the statistics or economics departments, or in the engineering school). Some trainees take undergraduate or mixed undergraduate and graduate-level courses. The advantages of tight integration into the university far outweigh the disadvantages, although we recognize that some candidates for health services research training would prefer the customized courses offered in other settings, such as a school of public health.

Although our principal health services research fellowship is sponsored by AHRQ, as mentioned earlier, we have related postdoctoral fellowship programs sponsored by the VA—one in health services research, one in medical informatics, and one in ambulatory care (which is available only to physicians). Virtually all the trainees in these programs do health services research or a combination of health services research and medical informatics. Most obtain formal degrees, usually at the master's level in health services research or medical informatics, and occasionally PhDs.

In addition to the master's degree in health services research, some of our trainees choose to pursue PhDs or master's degrees in other fields, such as statistics, sociology, economics and, of course, medical informatics. Although some applicants are interested in obtaining MBAs, we do not offer that option. However, the curriculum that they take in health services research can be broadly similar, and if they choose to get a health services research MS degree, they have the choice of diverging along any of several tracks. Because only a few courses are required of all degree candidates, the program is intended to allow considerable flexibility through electives, and the fields covered by our participating faculty include biostatistics, communication, economics, epidemiology, ethics, international health, internal medicine, psychology, public policy analysis, law, and sociology as well as medical informatics. Considerable emphasis is placed on effective mentoring, and both faculty mentors and the leaders of the master's program are expected to help students choose a set of courses that will give them both broad competence in the methods of health services research and expertise in a discipline or subject area.

The AHRQ-funded fellowship program currently has 16 postdoctoral and six predoctoral graduates, with degrees ranging from health services research and medical informatics to economics, sociology, statistics, and psychology. A wide variety of clinical areas have been represented, both in the background training of the students and in the projects that they have pursued while in training. All our alumni, except one, have assumed academic positions on program completion.

Bridging the Gap

The Stanford experience has shown that health services research and medical informatics training programs can both coexist and form the basis for highly complementary educational and research opportunities. Many of our graduates are cross-trained at the intersection of the two disciplines and now carry out research and educational programs that clearly embody elements of both fields and demonstrate the synergies and interdependencies that exist between them. Representative examples include:

• Dr. Gillian Sanders, an informatics PhD graduate who has joined the Stanford faculty explicitly to help us in bridging the two disciplines. Her dissertation was entitled “Automated Creation of Clinical Practice Guidelines from Decision Models.”

• Dr. Harold Lehmann, who also holds a PhD in informatics, now at Johns Hopkins University, and active in the Society for Medical Decision Making. His dissertation work dealt with building Bayesian statistical expert systems to aid in clinical decision making.

• Dr. Doug Owens, a graduate of the MS in health services research program and the AHRQ fellowship, who is playing a key role on the faculty of the Stanford centers directed by Dr. Garber. His work deals broadly with technology assessment, and he served as the lead author for the chapter on clinical decision making in Dr. Shortliffe's textbook of medical informatics.⁶

Dr. Owens and Dr. Sanders played central roles in the Cardiac Arrhythmia PORT (Patient Outcomes Research Team), drawing in large part on tools from informatics. For example, they were responsible for leading the decision analytic model and the cost-effectiveness analysis used to integrate all the other parts of the PORT. Both of them had been trained in this area as part of both the informatics curriculum and the health services research curriculum.

Additional examples of graduates doing research at the intersection of medical informatics and health services research include Dr. Ida Sim (now at UCSF), Dr. Glenn Rennels (doing informatics with the Kaiser Permanente Medical Group in Northern California), and Dr. Lucila Ohno-Machado (now with the Decision Systems Group at Harvard Medical School).

Although occasional trainees enter one of the two programs with a plan to work at the intersection of medical informatics and health services research, it is more common for them to develop such interests after they have taken courses at Stanford and have interacted with colleagues in both training areas. Thus, for example, Dr. Mary Goldstein was trained in health services research with an initial interest in health services topics far removed from informatics. But she took courses in informatics and other courses with trainees from the informatics program, and subsequently deepened her interest in guideline implementation. Her work in that area naturally turned to informatics-based solutions, and as a faculty member she began collaborating with Mark Musen and others in the informatics program. Even a brief exposure to informatics for health services research trainees, or to health services research for medical informatics trainees, can give them an appreciation for the area outside their primary fields, which can subsequently lead to effective collaborations and multidisciplinary research.

Our experience has demonstrated that successful programs will require areas of overlapping or synergistic interest and activity among the involved faculty and, hence, in time, among the students. Furthermore, the design of the programs requires a mixture of casual and structured contact among students from both disciplines, including social interactions. Issues accordingly include the offering of common classes, with joint projects that bring trainees together to work as colleagues; physical proximity when possible among the training sites; shared colloquia and research seminars; social events; and scientific retreats that build esprit, understanding, and a sense of shared commitment.

Perhaps the greatest challenge to our programs has been the shortage of faculty in each area, with limited ability to recruit new faculty despite the great need for incremental faculty in overlap areas, such as decision analysis and large database analysis. Other challenges include overcoming geographic separation that may exist in a given institution, a problem that is often inevitable with growth; the proper management of relationships with those sub-areas of medical informatics that have less overlap with health services research (e.g., imaging informatics and bioinformatics); and the need to determine how best to exploit opportunities for collaboration that naturally occur (e.g., problems in database analysis and data mining, organizational theory and its application, evaluation of clinical systems, delivery systems for evidence-based guidelines, and “just in time” delivery of information in clinical settings).

Although the primary motivation for our programs has been the creation of the next generation of academic researchers and teachers in health services research and medical informatics, many of our top students have, of course, pursued other career goals. Our graduates have encountered remarkable opportunities in industry, both in startup companies and in the research arms of established large commercial organizations. This trend has demonstrated the importance of health services research and medical informatics in the country's evolving economic base, and we view it as encouraging evidence of the role that these fields will continue to play in biomedical research and the health of the nation.