Looking Forward to Impact: Moving Beyond Serendipity

When research findings go from the printed pages of professional journals to everyday application by health care institutions, clinicians, and patients, one of our most important missions as health services researchers is realized.

John M. Eisenberg

A year ago in this space, our friend and colleague, the late John M. Eisenberg discussed the importance of measuring the impact of research sponsored and conducted by the Agency for Healthcare Research and Quality (AHRQ). In discussing the difficulty of tracking the impact of such research, and often the sheer serendipity that news of its use reaches AHRQ, John recalled two of his favorite stories.

The first was about his initial appearance before a House Appropriations subcommittee, and the question then Chairman John E. Porter put before him: whether AHRQ research was being used, followed, and given to patients.1 The second was a story about his first few weeks at AHRQ, when he learned about use of AHRQ Patient Outcomes Research Teams (PORT) findings in a flyer he had received in his previous capacity as chief of the medical service at Georgetown, and as a participant in several managed care plans.

Connecting the stories, John then wrote: “To address a question as important as the one John Porter asked me in my first Appropriations subcommittee hearing, we need more than serendipity. We need systems in place to track and report impact, and we need help from the researchers we fund.”

In the year since that AHRQ Update, the health services research community has responded to the call, and the Agency's portfolio of how AHRQ research is being used—called Impact Case Studies—is growing. Many of you have sent the Agency information on how your research is being used. Now, systems, not serendipity, prevail. But as John himself might write (and often said), waiting to think about impact until after a study is completed is not enough; it is just one step along the way.

This AHRQ Update outlines the next step in tracking how what we know istranslated into what we do; that is, identifying how findings will be used (and by whom) even before our research begins. Prior to discussing this newest step, the Update discusses the Agency's requirements, by statute, to clearly state expectations for outcomes of research investments. The Update then showcases several Impact Case Studies and describes ways the health services research community can look forward to impact.

Accountability as Public Trust and Professional Obligation

In many respects, the question John Porter posed of John in his 1998 debut as Agency administrator was in keeping with a Federal reporting responsibility that was to become effective two years later. While the Agency had always been subject to oversight by Congress, and hence accountable for Federal appropriations, the new millennium brought with it a new level of accountability called the Government Performance and Results Act of 1993 (GPRA, PL 103-62), which became effective in the spring of 2000. Specifically, GPRA requires Federal agencies to report to Congress on the establishment of process, output, and outcome measures directly linked to program activities.

The statute applies to all government agencies and programs but creates certain unique challenges for research agencies like AHRQ and the National Institutes of Health (NIH). Because of the inherent nature of scientific inquiry, these research agencies have embraced a reporting strategy that includes a strong qualitative dimension with descriptive case studies. Consequently, in drafting each strategic plan since GPRA took effect, the Agency has worked carefully to make certain that its essential mission to ensure health care quality can be measured through process, output, and outcome measures.

In last year's AHRQ Update, John reminded us “As health services researchers, particularly those of us who are stewards of the public purse, we must keep in mind that the ultimate purpose of our work is to improve health care and, as a result, improve the health of the public.” As his statement implies, accountability is both a public trust and a professional responsibility, and one that is shared by private funders as well. In fact, in a 2001 report prepared for AHRQ, The Lewin Group examined the common issues faced by AHRQ, the Department of Veterans Affairs, and nonprofit organizations such as the Robert Wood Johnson Foundation in tracking the impact of research they sponsor.2 Aparticular challenge for all funders is identifying use of information generated by the research community beyond publications. However, there is a clear consensus among funders regarding the desirability of assessing broad impact as well as strong interest in conceptual approaches and effective methods for clarifying the contributions of research to practice and policy.

Systems, Not Serendipity

To capture and record the impact of AHRQ-sponsored research, the Agency has established a diverse portfolio of activities including targeted independent evaluations of specific programs (e.g., the Evidence-based Practice Center program), regular customer surveys (e.g., the National Guideline Clearinghouse, the Medical Expenditure Panel Survey, or MEPS), bibliometric techniques to compare the productivity of intramural and extramural researchers, and an active Impact Case Studies Program. This last program, located within the Office of Health Care Information (OHCI), systematically tracks, updates, and documents examples of how AHRQ-supported research impacts the health care system. (In addition, a model that clarifies how research is translated into impact, the impact pyramid, has been previously described in this journal.) Below are a few of those examples.

Providing the Foundation for Evidence-based Policy

Today, government agencies at both the Federal and state levels use AHRQ-funded resources as the foundation for evidence-based health care policy. Among the examples cataloged by OHCI are uses by the Departments of Commerce, Defense, and Treasury; the Congressional Budget Office; the Council of Economic Advisors; sister agencies within the Department of Health and Human Services, such as the Centers for Medicare and Medicaid Services (CMS), the Food and Drug Administration, and NIH; and state health departments.

The Medicare program relies upon technology assessments supported by AHRQ to inform coverage decisions made by the Medicare Coverage Advisory Committee.3 For example, CMS used the AHRQ-sponsored Technology Assessment for Actinic Keratoses Treatment to revise its national coverage policy on actinic keratoses (AK), a common skin condition that is often the precursor of skin cancer. The assessment suggested that, more than other factors, the presence of AK was associated with the development of squamous cell carcinoma (which has the potential to metastasize and accounts for a large percentage of all nonmelanoma skin cancer deaths in the Medicare population). Combined with other source information, CMS used this assessment to revise its Medicare Coverage Issues Manual to include coverage for the treatment of AK.

Another set of Impact Case Studies involves use of data from MEPS. For example, the Commerce Department's Bureau of Economic Analysis has been using the MEPS Insurance Component (MEPS-IC) since 2000 to calculate its estimate of the national Gross Domestic Product (GDP).4

Providing the Tools That Improve the Quality of Care

AHRQ funds and/or sponsors the development of a wide range of tools that help improve the quality of care. Among these resources are measures, decision support tools, databases and related products, education materials on prevention, and recommendations from the U.S. Preventive Services Task Force (USPSTF).

For example, since 1997, the Healthcare Association of New York State, HANYS, has been using the HCUP Quality Indicators (QIs) to produce annual comparative reports for its member hospitals. HANYS represents more than 500 nonprofit and public hospitals, long-term care facilities, and home health agencies. The purpose of each annual report is to provide individual hospitals with comparative data on a broad range of indicators to help them target areas for improving quality of care and efficiency.

Recommendations developed by the AHRQ-sponsored USPSTF have been used by individual providers, professional societies, health plans, and policymakers to guide practice and policy regarding clinical prevention in primary care.5 To make information on clinical preventive services available as soon as possible, the Task Force is now releasing each new recommendation as it is finalized. (Previously the Task Force waited for all the recommendations to be available before publishing them in its Guide to Clinical Preventive Services.) Among several case studies about use of USPSTF recommendations is one regarding the American Academy of Family Physicians (AAFP). This professional organization, which represents more than 85,000 family practitioners, used the Task Force's assessments as the foundation for its “Recommendations for the Periodic Health Examination” that outlines standards and guidelines for preventive care.

Materials from Put Prevention into Practice (PPIP), another set of resources, are disseminated and used by a number of AHRQ partners, including health care plans, clinics, health centers, and even state governments. Since 1994, for example, the Texas Department of Health (TDH) has promoted implementation of AHRQ's PPIP program in various clinical settings throughout the state. To encourage implementation, TDH began by providing grants to local health departments, community health centers, family practice residencies, and other nonprofit primary care clinics. Funds were also awarded to the University of Texas—Austin to evaluate the impact of PPIP implementation at the grant-funded sites. Among the PPIP resources TDH customized for adaptation in the state were the individual health risk profile, readiness tool (which assesses a clinician's readiness to implement PPIP), patient education materials, and a policies and procedures manual.

Ensuring the Quality and Value for the Money Spent on Health Care Services

AHRQ research helps improve clinical outcomes while also ensuring quality and value for the money spent on health care services. Much of this research has involved the development and testing of new technologies.

For example, AHRQ-funded research led to development of softwarethat helps doctors more quickly diagnose cardiac ischemia. The software itself runs a new electrocardiogram (EKG) machine. Using a program called ACI-TIPI, or Acute Cardiac Ischemia-Time Insensitive Predictive Instrument, the software analyzes the EKG for signs of cardiac ischemia or heart attack, enabling physicians to better determine whether thrombolytic drugs should be administered. While the impact of this research has been far reaching, its full effect has yet to be measured completely. However, we can currently employ two gauges of its success. First, the two companies whose combined revenues constitute 80 percent of the EKG market—Hewlett-Packard and Marquette Electronics—employ this software as a standard feature. Second, one AHRQ-funded study estimated that widespread use of the instrument could result in 204,000 fewer hospital admissions a year and 112,000 fewer coronary care unit admissions, for an overall annual savings of $728 million.

In another example involving software, AHRQ-funded research helped three major Boston facilities in their efforts to reduce medication errors. AHRQ research on the “Physician Computer Order Entry” demonstrated how adding that software to an institution's computerized information system helped eliminate medication errors. Instead of relying upon handwritten orders, the software enabled doctors to enter medication orders (including dosage, route, and frequency) directly on a computer terminal. The system also alerted doctors when an order contained a possible error, such as a potential drug interaction or allergic reaction by the patient. A test of the software at one hospital found that it decreased the rate of serious ”nonintercepted” errors (mistakes that could have or did cause adverse drug events, and were not caught before reaching the patient) by more than half. In addition to protecting patients, the new software is estimated to have saved the hospital between $5 and $10 million annually, even after accounting for development, start-up, and maintenance costs.

Improving Patients’ Clinical Outcomes

Research conducted by one of AHRQ's seven Centers for Education and Research on Therapeutics (CERTs) at the University of North Carolina resulted in offering infants in North Carolina vitamin D to protect against rickets. The policy was implemented after researchers demonstrated that many exclusively breast-fed, (nonwhite) infants would benefit from vitamin D supplementation. Consequently, the North Carolina Pediatric Society requested that the state of North Carolina distribute a vitamin D supplement free of charge to any exclusively breast-fed infant or child, six weeks of age or older. Funding for the supplement was provided through a Maternal and Child Health Block Grant and distributed through the Supplemental Nutrition Program for Women, Infants, and Children. As a result, more than 1,500 children received this supplementation over a 16-month period at a cost of about $1.50 per month, per child.

On the other end of the health care spectrum are the nation's older citizens. Here as well, Agency research is improving clinical outcomes. Medicare's Peer Review Organizations (now called Quality Improvement Organizations or QIOs) have been using findings from AHRQ stroke research and have implemented 73 projects in 42 states. Specifically, the findings called for greater utilization of anticoagulation therapy for Medicare beneficiaries who had suffered a stroke. As a result, the percentage of Medicare patients discharged on anticoagulation therapy has increased from 58.4 to 71.1 percent. Further, CMS estimates that this improvement has prevented as many as 1,300 strokes in 1995.6

Helping Patients Become More Involved in Their Health Care Decisions

As many readers of this publication know, AHRQ's Consumer Assessment of Health Plans (CAHPS®) is the survey that gives consumers information on the experiences of their neighbors and colleagues regarding the quality of care offered by health plans in a given organization, state, or region. Consumers then use this information to make informed choices among health plans. CAHPS® is now used by the Federal Employees Health Benefits Program (FEHBP), Medicare, more than 20 states and a wide range of private-sector companies and employer coalitions. In fact, CAHPS® data are available to more than 90 million Americans (roughly one-third of the U.S. population) who face choices about their health care.

To help Federal employees and retirees choose from among the 400-plus health plans in the FEHBP, the Office of Personnel Management (OPM) uses adecision making tool funded by AHRQ. Developed by Decision Innovation of Research Triangle, North Carolina, PlanSmartChoice is an interactive, multimedia health plan selection tool provided on CD-ROM. In 1998, OPM conducted a five-state pilot study using the decision support tool. The pilot's success led a year later to the award of a contract to Decision Innovation with OPM for worldwide implementation of its decision support tool. As a result, 6 million Federal employees and retirees now have the option of using the tool to help select the best health coverage for themselves and their families.

Impact at the Beginning of the Research Cycle

Each of the above examples addresses the fundamental ”Porter question” John reminded us of almost every day. And each is illustrative of the Agency's mission-critical objective of translating research into practice. But now it is time to take our efforts a step further.

While each of the above studies shows how evidence was used to inform policy, and assess and improve the quality of care, many involved the identification of knowledge utilization, or impact, after the research had been conducted or the data developed. Such examples only reinforce expectations that we can do this systematically and strategically, rather than rely on serendipity. Today, there is a growing expectation among policymakers and other funders for evidence upfront for the return on investments in research. Again, it is an obligation not faced by AHRQ alone, but by all Federal agencies. Further, it is a concern that must be addressed at the outset of research—even as a grant application is being developed—and it must continue throughout the cycle of research.7

The challenge for health services researchers, then, is more immediate—one of identifying beforehand who will use the research, instead of waiting to see after the research is conducted who is using it. Therefore, in addition to turning what we know into what we do, we must first consider who needs to know what and why. Table 1 outlines some of the ways researchers can help translate research into practice (thereby anticipating impact), even before that research is conducted. And, of course, we encourage you to continue sending Impact Case Study leads to AHRQ's Division of Public Affairs Division at kmurray@ahrq.gov.

Table 1.

Researcher Roles in Tracking Impact

1. Ask users what they want to know.

2. Partner with users in research design phase.

3. Include in research all the people who will be affected.

4. Tell your user partners what you found.

5. Know why the findings matter, and say so.

6. Tell funders who's using your research.

7. Ask those users what they need to know next.

Looking Forward

To understand our newest challenge more clearly, we need only remember the words of one who valued his public service role and his health services researcher role equally and who experienced the task of balancing Federal accountability with the Agency's essential mission as the ultimate in career achievement: John Eisenberg.

When John was able to use research to inform a discussion, his joy in his work was readily apparent. Always willing to take a phone call or go to Capitol Hill to help policymakers think through complicated issues, John would often remark, ”Isn't this great? Can you believe we get paid to do this?”

We aspire to the high standards John set for us and will carry his work forward. But we need your help. Continuing John's legacy of supporting research that builds the evidence base for quality health care is a shared responsibility. Further, the Agency's ability to fully reflect back to Congress the richness and strength of your research in this regard depends on the Agency's having a growing portfolio of user-driven stories to tell. We've moved beyond serendipity; now we need to look forward.

The authors gratefully acknowledge the research assistance of Louise Arnheim, Karen Migdail, and Jane Steele of the Agency's Office of Health Care Information.