Evaluation of the AHRQ Patient Safety Initiative: Synthesis of Findings

Abstract

Objective

To present overall findings from the 4-year evaluation of the national patient safety initiative operated by the Agency for Healthcare Research and Quality (AHRQ).

Data Sources

Interviews with AHRQ staff, grantees, and other patient safety stakeholders; published materials; and internal AHRQ documents.

Study Design

The evaluation was structured to address a system framework of five components involved in improving safety. The initiative's contributions to improving each system component were assessed qualitatively, comparing results from three separate analyses—AHRQ's achievement of its patient safety goals, our own assessment of the initiative's activities, and independent stakeholder ratings of AHRQ's contributions.

Findings and Conclusions

AHRQ has faced a daunting challenge for improving patient safety, given the complex problems of the U.S. health care system and the limited resources AHRQ has had to address them. The patient safety initiative achieved strongest progress for its contributions to knowledge of patient safety epidemiology and effective practices, where AHRQ has considerable experience, and to strengthening infrastructure to support adoption of safe practices. Progress was slower in establishing a national monitoring capability and dissemination of safe practices for adoption. AHRQ needs to expand efforts to apply new knowledge for stimulating use of safe practices in the field.

Stimulated in large part by the publication in January 2000 of the Institute of Medicine report, To Err Is Human: Building a Safer Health System, the U.S. Congress established patient safety as a national priority in FY 2001 and gave AHRQ the mandate to lead federal patient safety improvement activities. In a multifaceted approach, AHRQ funded projects to develop new knowledge on patient safety epidemiology and practices, developed infrastructure components to strengthen support for patient safety improvements, and pursued dissemination activities to encourage adoption of effective practices.

To understand how the initiative was progressing, AHRQ contracted with RAND in September 2002 to serve as its Patient Safety Evaluation Center (evaluation center) and to perform a longitudinal, formative evaluation of the patient safety initiative. This 4-year evaluation, completed in September 2006, examined a broad range of components of the initiative. As described in the first article in this issue (Farley and Battles 2008), the evaluation design was based on the CIPP model, which encompasses the full spectrum of factors involved in the operation of a program (Stufflebeam et al. 1971; Stufflebeam, Madaus, and Kellaghan 2000), including the four evaluation components represented by the CIPP acronym: context, input, process, and product.

Collectively, the articles in this issue address the CIPP evaluation components that we performed in evaluating the patient safety initiative. The context and input evaluations, which are summarized in the introductory article (Farley and Battles 2008), examined the strategic aspects of the initiative. The process evaluation focused on its more operational aspects. Examples of how our process evaluation assessed specific activities of the initiative are presented in the second through sixth articles (Sorbero et al. 2008; Taylor et al. 2008; Damberg et al. 2008; Teleki et al. 2008; Mendel et al. 2008) We examined many more such activities in the evaluation, with results reported in four evaluation reports (Farley et al. 2005, 2007a, b, 2008). The seventh article (Greenberg et al. 2008) summarizes our product evaluation analyses, which was developmental work to help AHRQ prepare for monitoring impacts of the initiative on various stakeholders.

In this paper, we focus on process evaluation results, synthesizing our overall findings regarding AHRQ's progress in implementing its patient safety initiative over the 7-year period of 2000–2006. We also present suggested next steps for the initiative based on collective results from our context, input, process, and product evaluations.

The purpose of a process evaluation is to assess the evolution of activities undertaken in a program, successes or failures in executing those activities, and factors contributing to its progress. The process evaluation serves an important role in its own right by providing information about which aspects of the program are working well and what changes are needed to improve aspects that are not as successful. In addition, process evaluation results can be used to help interpret program outcomes, to better understand the dynamics of the activities that led to either positive or negative outcomes (Rossi and Freeman 1993; Stufflebeam, Madaus, and Kellaghan 2000).

A Framework for the Process Evaluation

To guide the process evaluation, we developed a conceptual framework that defines a five-component system for improving patient safety. These five components (which we call “system components”) work together to achieve improved practices in the health care system. We assessed how well AHRQ's patient safety initiative had succeeded at contributing to each component:

Monitoring Progress and Maintaining Vigilance. Establishment and monitoring of measures to assess performance improvement progress for key patient safety processes or outcomes, while maintaining continued vigilance to ensure timely detection and response to issues that represent patient safety risks and hazards.

Knowledge of Epidemiology of Patient Safety Risks and Hazards. Identification of medical errors and causes of patient injury in health care delivery, with a focus on populations that are vulnerable because they are compromised in their ability to function as engaged patients during health care delivery.

Development of Effective Practices and Tools. Development and field testing of patient safety practices to identify those that are effective, appropriate, and feasible for health care organizations to implement, taking into account the level of evidence needed to assess patient safety practices.

Building Infrastructure for Effective Practices. Establishment of the health care structural and environmental elements needed for successful implementation of effective patient safety practices, including an organization's commitment and readiness to improve patient safety (e.g., culture, information systems), hazards to safety created by the organization's structure (e.g., physical configurations, procedural requirements), and effects of the macroenvironment on the organization's ability to act (e.g., legal and payment issues).

Achieving Broader Adoption of Effective Practices. The adoption, implementation, and institutionalization of improved patient safety practices to achieve sustainable improvement in patient safety performance across the health care system.

Three-Part Method for Assessing Overall Progress

The process evaluation was designed to assess the contributions of the patient safety initiative to strengthening each of the five system components in the framework presented above. The summary results developed from these analyses are qualitative. To obtain as rich an understanding as possible of its contributions, we used three separate analyses and compared their results in summary assessments. By drawing upon three sets of results, we could interpret findings with more confidence than would have been possible if we used only one set of analyses.

The first analysis (analysis 1) was an assessment of how well the initiative was performing in meeting the goals and targets that AHRQ had established for itself, relative to each system component. The second one (analysis 2) was our own comprehensive assessment of how the key activities of the patient safety initiative collectively contributed to each of the five system components. The third one (analysis 3) was assessment of ratings by national stakeholders involved in patient safety, for which we gathered data in individual interviews on their views regarding progress in improving patient safety in the United States, AHRQ's contribution to progress, and the need for future efforts to make further gains.

In preparing for analyses 1 and 2, the first data collection step was to identify the various activities AHRQ was pursuing in the patient safety initiative, through review of written documents, lists of funded projects, and interviews with AHRQ staff. Some of the key activities are addressed in five articles in this issue—funding research and development projects (Sorbero et al. 2008), patient safety implementation projects (Taylor et al. 2008), health information technology (health IT) projects (Damberg et al. 2008), Patient Safety Improvement Corps training program (Teleki et al. 2008), and partnership formation with other organizations (Mendel et al. 2008). Examples of other activities are development of the Patient Safety Indicators (PSI), patient safety evidence report, PSNet Web site, patient safety culture survey, and the TeamSTEPPS package. See Farley et al. (2008) for detailed listings of activities in the initiative.

We then collected and analyzed data for each key activity, through which we documented and assessed progress, successes, challenges, and effects. In most cases, we used a combination of semistructured interviews for primary data collection and data from available publications or documents.

Analysis 1: Achievement of AHRQ's Goals and Targets

AHRQ established a patient safety strategy consisting of four elements: identifying threats to patient safety; identifying and evaluating effective patient safety practices; teaching, disseminating, and implementing effective patient safety practices; and maintaining vigilance (AHRQ 2003). Specific performance goals and related fiscal year targets were defined for the first three of these elements (Table 1). We mapped each of AHRQ's strategy elements to one of the five system components in our evaluation framework. Then we used the data we developed on the initiative's key activities to examine AHRQ's performance in meeting its goal(s) and targets relevant to each component.

Table 1.

Current Status of AHRQ on Its Patient Safety Performance Goals and Fiscal Year Targets

	Goals and Fiscal Year Targets*	Status as of End of FY 2006
Identify the Threat
Performance Goal: By 2010, patient safety events reporting will be standard practice in 90 percent of hospitals nationwide		As of 2005, an estimated 98 percent of hospitals report they have centralized, internal patient safety event reporting systems, but only 12 percent of them are fully computerized. None are reporting full safety information to external reporting systems
FY 2005	Continue reporting on patient safety events and begin to analyze the number and types	Cannot proceed with external reporting until regional or national reporting systems are in place
FY 2004	Pilot the system at 50 hospitals and begin reporting on patient safety adverse events	Not yet met as of FY 2006; may be achieved when external reporting occurs in the patient safety organization (PSO) program under the Patient Safety and Quality Improvement Act
FY 2003	Develop reporting mechanism and data structure through the National Patient Safety Network†	Not yet met as of FY 2006, although the national data network for the PSO program has potential to achieve this
Identify and Evaluate Effective Practices
Performance Goal: By 2010, double the number of patient safety practices that have sufficient evidence available and are ready for implementation (use the Evidence-based Practice Center [EPC] report for baseline data)		On track to achieve goal. Patient safety grants addressed many practices for which evidence was insufficient or lacking, but findings not yet assessed to determine strength of evidence
FY 2005	Five health care organizations/units of state/local governments will evaluate the impact of their patient safety best practices interventions	Accomplished in second year of the seven implementation Challenge grants; also Partnerships Implementing Patient Safety grants
FY 2004	Six health facilities or regional initiatives to implement interventions and service models on patient safety improvements will be in place	Accomplished with first year of work for the seven implementation Challenge grants
FY 2003	Awards to be made to at least six facilities or initiatives	Accomplished on schedule. Awarded 13 Challenge grants, of which 7 grants were to implement and evaluate new practices
Educate, Disseminate, and Implement to Enhance Patient Safety
Performance Goal: By 2010, successfully deploy hospital practices such that medical errors are reduced nationwide		Too early to assess this goal because impacts of the patient safety initiative are likely to lag several years from date of initial funding for research and development
FY 2005	15 additional states or major health care systems will have on-site experts in patient safety	Accomplished on schedule. Trained staff from another 15 states and hospitals in FY 2005
FY 2004	10 states or major health care systems will have been trained through the Patient Safety Improvement Corps (PSIC) program; five health care organizations or units of state/local government will implement evidence-based proven safe practices	Accomplished on schedule and exceeded target. Trained staff from 12 states and 12 health care organizations through FY 2004
FY 2003	Establish a PSIC training program; award up to five grants to health care organizations or units of state/local government for implementing evidence-based proven safety practices	Accomplished on schedule
FY 2002	Conduct a planning study	Completed. This study developed the initial design for PSIC

Sources: AHRQ Justification for FY 2005 Budget (AHRQ 2004). Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

^*A goal and set of annual targets are set for each of the four elements of AHRQ's patient safety strategy.

^†This is the term used by AHRQ to describe a national data capability that integrates data from multiple databases.

AHRQ's elements of identifying threats and maintaining vigilance were linked to the monitoring system component, and identifying and evaluating patient safety practices was linked to the epidemiology and effective practice system components. We linked AHRQ's element of teaching, disseminating, and implementing safety practices to two system components—infrastructure and dissemination of practices, assigning each of AHRQ's annual targets for this element to the system component to which it most directly applied.

Analysis 2: Evaluation Center Assessment of Activities

To assess the collective contributions of the activities in the patient safety initiative to each system component, we first developed a list of evaluation questions to be addressed for each component. Some questions address actions by AHRQ; others address actions or issues related to the larger health care system within which the initiative operates (Tables 2–5). For example, for knowledge on patient safety epidemiology, we asked how much information has been published over time, and we also asked how ARHQ-funded projects contributed to this information.

Table 2.

Evaluation Questions and Assessments for Monitoring and Vigilance, 2006

Evaluation Question	Summary Assessment
What progress has been made in establishing and using generally accepted sets of measures for patient safety events or outcomes in a range of health care service settings?	A modified Delphi consensus process was run by the evaluation center in 2006, which identified important outcome measures and assessed validity and measurement issues for them. The panelists concluded that work remains to validate most of these measures and to address definitions, measurement issues, and specifications
What progress has been made in establishing a consistent set of standards for patient safety reporting systems, public or private, for use by both government agencies and health care providers?	AHRQ began work on this with funding to IOM for preparation of its Data Standards report (IOM 2004). With passage of the PSQIA, AHRQ started work in 2006 to identify candidate standards for the data content of a national data network and reach consensus on which standards to use. These standards should complement the information technology standards being developed by Department of Health and Human Services
What actions have been taken to establish a national-level patient safety data repository?	With passage of the PSQIA, AHRQ has been able to move forward with design of a data capability, under the data network provisions of the Act. Plans are to begin full implementation of the Act in 2007, including progress on the national network of databases
To what extent are national-level data available regarding the performance of our health care system on patient safety measures, and how has this changed?	Data availability continues to be extremely limited, with heavy reliance on the HCUP data. State reporting systems have grown in number but vary widely in the measures and data they collect. No national-level data are available for ambulatory care, except Medicare data, which has not yet been used for patient safety measurement
What is the status of the use of generally accepted patient safety measures for assessing performance as part of accreditation or other credentialing processes?	The primary organization using patient safety measures is Joint Commission, which has had relevant policies in place for several years, specifically its Sentinel Events reporting policy. The National Quality Forum consensus process has established measure sets, but none of them has been used consistently across the country
What steps need to be taken to enhance the capability for effective monitoring of patient safety performance?	Priority issues to be addressed include standardization of measures and standards, increasing data availability at the national level, and measurement issues such as use of rates versus counts

PSQIA, Patient Safety and Quality Improvement Act of 2005.

Source: Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

Table 5.

Evaluation Questions and Assessments for Broader Adoption of Effective Practices, 2006

Evaluation Question	Summary Assessment
To what extent is new evidence on effective practices and implementation methods being disseminated to the broader health care system?	Results from patient safety projects are being published in journals and compendia. The only systematic synthesis and use of this information thus far has been for the updating of the National Quality Foundation safe practices
What actions has AHRQ undertaken for disseminating information and products from the patient safety grants and contracts it has funded?	AHRQ has highlighted results and products from individual projects in communications outreach using multiple media, including Web sites (e.g., PSNet), press releases, newsletters, etc. Most information has not yet been packaged for ready use by providers
What systematic programs are in place or under development by AHRQ or other organizations to provide technical support to health care providers for implementing tested patient safety practices?	Two major contributions AHRQ has made to tools have been the patient safety culture surveys and newly released teamwork improvement package. AHRQ has provided some technical support for the survey. The 100,000 Lives Campaign and SCIP have provided primary support for their participants
How are AHRQ's dissemination strategy and activities contributing to the growing number of patient safety implementation initiatives being initiated by others?	AHRQ is a partner in all of the major field-based safe practices initiatives, providing a support role as needed, with other organizations taking on the lead roles. These initiatives likely will seek updated information on practices and newly developed tools from AHRQ projects
Are patient safety practices and outcomes improving as a result of the cumulative efforts of generating knowledge, testing new practices, building an infrastructure, disseminating knowledge, and providing technical support for implementation?	Providers appear to be starting to implement safer practices, stimulated by field-based initiatives and supported by information and tools generated by AHRQ and its funded projects. The extent of actions is still too small to be having observable effects on outcomes at the national level

Source: Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

To address the evaluation questions developed for each system component, we assigned each of the initiative's specific key activities to one of the system components, and we examined the collective results of our assessments of the set of activities assigned to that component. A summary assessment was prepared for each question that addressed progress made, resulting status, and relevant issues.

Analysis 3: Ratings by National Patient Safety Stakeholders

In mid 2006, we conducted interviews with national stakeholders in the public and private sectors, which provided data for the third analysis. We interviewed 18 stakeholders who were national experts and thought leaders in patient safety from a diversity of organizations—government agencies (n=2), policy-making organizations (n=4), standards setting and accrediting organizations (n=1), purchasing groups (n=1), consumer groups (n=2), provider organizations (n=5), and research organizations (n=3).

The interviews were approximately 30–45 minutes in length. We developed a written interview protocol to ensure that consistent data were collected from all the stakeholders interviewed. Before the interviews, we provided interviewees with a one-page description of the five system components in the process evaluation framework. In the interviews, we asked them to assess overall progress in improving patient safety to date, and to identify areas in which the greatest gains have been achieved and areas with the greatest deficits. We also asked them to perform two ratings (on a scale of 1–5) for each of the five system components, one for progress made by the country as a whole and the other for AHRQ's effectiveness in providing leadership for these efforts.

We summarized the stakeholders‘ responses to the questions about progress in improving patient safety and issues to address in the future, identifying common themes and variations in responses. We calculated the means and standard deviations of their ratings of progress on the five system components, for the country and for AHRQ. We did not use medians because we wanted the full weight of all ratings, which fell within the 5-point range and were not skewed.

Results

Our process evaluation findings regarding the progress of AHRQ's patient safety initiative reflect the status of the patient safety initiative as of September 2006, which was the end of our 4-year evaluation. The goals and targets established by AHRQ and our summary assessments from analysis 1 are listed in Table 1. Summary assessments for the evaluation questions considered for each of the system components, developed in our comprehensive evaluation (analysis 2), are given in Tables 2–5. Stakeholder assessments of AHRQ's progress for each system component (analysis 3) are presented in Table 6.

Table 6.

Assessment by National Stakeholders of Progress for the Five Patient Safety Components, 2006

	Mean Scores (SD) by National Stakeholders Interviewed (Scale of 1 to 5 where 1=lowest; 5=highest)
System Component	Progress Made Nationally	AHRQ Effectiveness in Providing Leadership
Building and sustaining a national monitoring system	2.1 (1.0)	2.7 (1.1)
Expanding, conducting epidemiology research	2.5 (0.7)	3.2 (0.8)
Development of safe practices and tools	2.7 (0.9)	3.2 (0.9)
Building the infrastructure for safety	2.4 (1.1)	2.6 (1.4)
Adoption and diffusion of safe practices at the community, provider, or system levels	2.4 (1.1)	2.3 (0.9)

Source: Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

In general, we found that AHRQ's goals and fiscal-year targets were focused on activities that represented only a small portion of the total activities that AHRQ actually had undertaken. Many of AHRQ's other activities also were relevant to achieving the goals laid out in its strategy, a finding that highlights the importance of evaluating its progress in multiple ways.

Limited Progress in Monitoring and Vigilance

In analyses 1 and 2, we found that AHRQ had made limited progress in establishing a monitoring and vigilance capability, whereas the stakeholders rated its efforts somewhat more positively. AHRQ did not meet its goal or targets for identifying threats to patient safety, which requires a national monitoring capability that AHRQ had not yet developed (Table 1).

In analysis 2, we found that, although AHRQ has pursued several strategies for building a national data capability, these efforts had been frustrated by differing views among stakeholders on approaches, absence of consensus on data elements and standards, and disagreement on how best to develop a national-level capability. These differences likely reflect differing needs and priorities, highlighting the challenges involved in this endeavor (Table 2).

By contrast, the stakeholders gave AHRQ an average rating of 2.7 (on a 5-point scale) for its leadership in building and sustaining a national monitoring system. However, they gave the overall nation's progress a 2.1 rating, the lowest rating given (Table 6).

Strong Progress in Knowledge of Patient Safety Epidemiology

All three analyses identified strong progress for this component. We found that AHRQ met all its annual targets for the evaluation of patient safety epidemiology, which focused on reporting of safety events. In addition, our process evaluation found that AHRQ's contributions to the development of knowledge for patient safety epidemiology was one of its areas of greatest progress. This was achieved through the large number of patient safety projects it funded from 2000 through 2006, which yielded published papers that strengthened information on medications, diagnostic or treatment errors, and other issues (Table 3). Stakeholders also gave AHRQ the highest rating for its contributions in this area: 3.2 on a 5-point scale.

Table 3.

Evaluation Questions and Assessments for Patient Safety Epidemiology and Practices, 2006

Evaluation Question	Summary Assessment
Patient Safety Epidemiology
What information has been published over time about patient safety epidemiology, and how have the AHRQ-funded research projects contributed to this new information?	Numbers of articles published on patient safety epidemiology have grown steadily in the past four years. AHRQ projects have been important contributors, producing one-third of these articles
To what extent has the additional research strengthened the evidence regarding epidemiology of errors?	Published research has strengthened evidence for medications, diagnostic or treatment errors, general patient safety, working conditions, and other areas
What additional work is needed to strengthen the evidence regarding epidemiology and priorities for interventions to reduce adverse outcomes?	Work is needed on development of an ongoing national surveillance system, which would yield data needed to build evidence on epidemiology and identify emerging safety issues
Effective Patient Safety Practices
What do we know from evidence reports and other sources about which practices are or have the potential to be effective in improving patient safety?	The original patient safety evidence report documented evidence for many practices, which the National Quality Forum (NQF) used to develop its first list of safe practices in 2003
How are research and field tests on patient safety practices funded by AHRQ contributing new knowledge regarding practices for which further scientific evidence is needed?	AHRQ-funded projects addressed many practices in need of additional evidence. There results were used by the NQF to update its list of safe practices in 2006. More evidence synthesis is needed for other practices not addressed in the NQF update
What range of health information technology (IT) applications are being tested and studied by the health IT projects funded by AHRQ, and to what extent are they addressing patient safety issues specifically?	A wide range of health IT applications are being developed by AHRQ-funded projects. A high 90% of projects are addressing patient safety, all but three of which also addressed other quality issues
What are the field tests learning about the factors and issues that need to be managed to introduce tested new practices effectively?	The same implementation issues have been identified across many groups of projects. Through a literature review, the evaluation center systematically identified “success factors” that need to be in place to successfully implement practice improvements
What progress has been made by the AHRQ-funded projects in documenting effects of new patient safety practices on safety outcomes and the costs, cost effectiveness, and return on investment of the practices being tested?	Projects have been examining effects on outcomes and publishing those results. They have done limited work on assessing costs, cost effectiveness, or return on investment. Further work is needed to make the business case for new practices
To what extent are implementation methods and tools being developed and applied to support expanded use of tested practices across provider organizations?	Some projects have developed products or tools to support use of practices; many have not. AHRQ has packaged several tools, such as TeamSTEPPS (with DoD) and the hospital safety culture survey. More work is needed in this area

Source: Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

Strong Progress in Developing Effective Practices

All three analyses also identified strong progress for this component. We found that AHRQ met all its annual targets for the implementation of practices, which focused on work by its funded projects. AHRQ also appeared to be on track to achieve its goal of doubling, by 2010, the number of patient safety practices that have sufficient evidence available and are ready for implementation. This was indicated by our finding that the patient safety projects addressed a large number of practices for which evidence had been lacking (Sorbero et al. 2008, in this issue).

Our process evaluation found that AHRQ made strong contributions to the development of knowledge for patient safety practices through the patient safety projects it funded from 2000 through 2006 (Table 3). Projects have yielded published results, lessons, and tools for effective implementation for a variety of practices, including health IT projects.

Stakeholders also gave AHRQ the highest rating for its contributions in this area: 3.2 on a 5-point scale. They credited AHRQ with introducing effective practices and tools, developing indicators for measurement, pushing for transparency, and supporting research and development of practices and tools.

Moderate Success in Infrastructure to Support Effective Practices

Results from the three analyses were generally positive, but mixed, for this system component. Under AHRQ's goal for dissemination and implementation of practices, all of its annual targets addressed its Patient Safety Improvement Corps (PSIC) training program, and it met those targets. The absence of targets for other aspects of infrastructure, however, suggested a limited commitment to action in this area.

Our process evaluation found that AHRQ implemented a number of successful activities that strengthened infrastructure to support adoption of safety practice, despite having set few targets for this component. These include the health IT projects, PSIC, expansion of patient safety partnerships with other organizations, and culture survey products (Table 4). AHRQ had not yet addressed other aspects of this component, such as consumer engagement, building provider leadership commitment. The stakeholders gave AHRQ a rating of 2.6 for this component.

Table 4.

Evaluation Questions and Assessments for Infrastructure for Effective Practices, 2006

Evaluation Question	Summary Assessment
How are AHRQ and its funded patient safety projects contributing to establishment of infrastructure to support patient safety in health care organizations across the country?	Significant contributions have been made in several areas, including the hospital patient safety culture survey, funding of projects to develop information systems and reporting systems, growth in partnerships at the national and local levels, and measurement capability
What have we learned from existing research and practice networks funded by AHRQ about how to establish infrastructures that stimulate and support effective practices?	Funded projects have yielded lessons regarding factors necessary for successful implementation of practice improvements, some of which involve infrastructure (e.g., safety culture, data systems)
To what extent is there an infrastructure of inter-organization partnerships that is pursuing collaborative approaches to improving patient safety practices?	Between 2004 and 2006, the number of national-level, inter-organizational partnerships focused on patient safety grew substantially, as did AHRQ's involvement in these partnerships
How well are payment systems structured to provide appropriate incentives for safe delivery of safe health care?	There has been growing use of financial incentives by insurers for adoption of safe practices in hospital settings, with little information on their effects yet available. Such financial incentives are largely absent in ambulatory settings, in part due to the lack of readily available safety measures
What additional research or development work is needed to strengthen effective infrastructures for patient safety practices in the health care system?	Future efforts should focus on how to implement health IT to improve safety, increasing the role of leadership in establishing a strong safety culture, and supporting front-line staff in their efforts to improve safety. Additional training is still needed to expand the network of individuals with safety knowledge and the ability to apply that knowledge

Source: Assessment of the National Patient Safety Initiative: Final Report (Farley et al. 2008).

Limited Progress in Achieving Adoption of Effective Practices

Because AHRQ had not developed annual targets for practice dissemination or diffusion actions, we could not assess how well it was doing in this area relative to its own goal.

Our process evaluation found that AHRQ had made limited progress in disseminating knowledge and products to support adoption of effective safe practices by health care providers. At the end of our evaluation, AHRQ was still in the early steps of its dissemination activities, and several aspects of the process were underdeveloped or lacking (Table 5). These include lack of a systematic synthesis of the published results of funded projects to update the previous evidence report (Shojania et al. 2001), limited work to package products and tools for providers‘ use, and limited technical support for users in implementing new practices.

The stakeholders also gave AHRQ a low rating of 2.3 for its activities related to diffusion of safe practices through adoption by health care providers.

Stakeholders Highlighted Achievements and Future Directions

The stakeholder interviews yielded other insights on patient safety issues as a whole. There was general agreement among the stakeholders that much work remained to be done to advance patient safety in the United States. Although awareness of patient safety issues and the need to improve had increased, progress in achieving improvements had been limited. In addition, several individuals were concerned that it was hard to know just how much progress had been made because we lacked definitive measures and data needed to track possible improvements.

Specific examples of successes identified by stakeholders included enactment of the Patient Safety and Quality Improvement Act of 2005 (PSQIA), an increase in hospital reporting systems, enhanced attention to systems approaches for reducing medical errors and hospital-acquired infections, and training of individuals on safety concepts. They also highlighted specific project successes that had been highly visible and could serve as examples for others (e.g., 100,000 Lives Campaign, Surgical Care Improvement Project, Keystone ICU project with Michigan hospitals). Some of the identified drivers of the gains included leadership by the Institute of Medicine, AHRQ, the Joint Commission, the Leapfrog Group, and the Institute for Healthcare Improvement. Also cited were the broader movements toward transparency in health care and pay-for-performance.

The stakeholders also identified numerous factors having negative effects on patient safety progress. These included, for example, lack of engagement of health care leadership, lack of ongoing training structures, insufficient attention to quality at the systems level, poor coordination of policies among organizations guiding safety quality, and continued denial of safety issues by the provider community.

Virtually all the stakeholders interviewed expressed solid appreciation for the work that AHRQ had done to date in the area of patient safety, particularly in light of the limited resources AHRQ had to spend on safety relative to the magnitude of the problem. AHRQ was credited with introducing effective practices and tools, developing indicators for measurement, pushing for transparency, and supporting research and development of practices and tools.

The stakeholders indicated that AHRQ needed to work more aggressively in disseminating the results from the patient safety projects, to ensure that evidence-based practices were adopted by front line health care workers. For effective and quick dissemination, they felt that AHRQ needed to partner with other organizations working on safety to achieve synergy in these dissemination efforts.

Discussion

Given the size and decentralized nature of the U.S. health care system, and the significant patient safety problems it has been documented to have, AHRQ has faced a daunting leadership challenge for effecting systems change. The nature of activities undertaken in its patient safety initiative has shifted gradually over time. The initiative began with the funding of a large number of patient safety projects selected to generate new knowledge on epidemiology and practices for a diversity of patient safety issues. Then AHRQ's emphasis shifted toward testing methods to implement patient safety practices and to dissemination of project results for encouraging adoption by health care providers.

The perspectives shared by the national stakeholders interviewed might be characterized as cautious optimism. Most of them felt that the only significant progress to date had been to create awareness across the health system that safety is a real issue that must be addressed. The stakeholders emphasized that a great deal of work remained before the singular successes achieved thus far could be turned into broad-spread safety improvements. This sentiment also was reflected in the findings from our own process evaluation.

AHRQ Has Made Progress in Some Areas with More Work Needed in Others

Overall, AHRQ made observable progress in implementing its patient safety initiative from FY 2001 through FY 2006. The consistency in findings across the three separate analyses suggests a robustness in the evaluation results. Although results differed for some system components, most of them were in accordance.

It is not surprising that the areas of strong progress have been in the development of new knowledge on patient safety epidemiology and practices. AHRQ already had a strong history of developing knowledge through research, which it brought to the patient safety initiative. At the start of the initiative, AHRQ elicited priorities from stakeholders on which topics are most important to address, which paid off in a portfolio of funded projects that addressed a breadth of important patient safety issues and practices. Its contributions toward building infrastructure to support patient safety practices in the field also are encouraging. More work remains to be done, however, in the areas of monitoring and vigilance, and in supporting the adoption of safe practices by health care practitioners and providers.

Our finding that AHRQ's fiscal-year targets were focused on only a small portion of what AHRQ actually had undertaken in the patient safety initiative suggests that AHRQ was conservative in its goal setting. With the experience gained over the past few years, AHRQ should be well positioned to set future targets across the range of system components. By setting targets for monitoring and practice dissemination, in particular, it can strengthen progress in the two areas for which further work was found to be needed.

Future Directions and Patient Safety Priorities

As we have been aware, and stakeholders have highlighted in our interviews, one of the constraints on AHRQ's progress has been the limited funding appropriated for its patient safety work, relative to the sheer size of the U.S. health care system and the patient safety problems it has been documented to have. We expect that, with more substantial funding, AHRQ would be able to pursue a broader range of activities, which could accelerate progress toward achieving a safer health care system.

We presented in our evaluation reports numerous specific suggestions for AHRQ actions to strengthen its contributions to each system component. These suggestions were based on results of our assessments of AHRQ's patient safety strategy, current activities of its grantees and field organizations, and feedback from stakeholders. We also identified the following general priorities that we consider the most important items for ARHQ to address as its initiative moves forward:

• Continue working toward establishing a national data network for use in monitoring trends in patient safety status, including collaboration with health care organizations to establish a core set of national patient safety measures and consistent data standards for measuring them.

• Update the patient safety evidence report to incorporate recently published results from the patient safety projects, applying standards of evidence that ensure rigorous assessment of study designs for testing patient safety practices that cannot be tested effectively using randomized control study designs.

• Select key patient safety practices to be national priorities for dissemination to health care providers, using syntheses of results from the AHRQ-funded patient safety projects.

• Working closely with the health care provider community, package and disseminate patient safety products and tools for the key practices identified from the synthesis of project results, including development of generic, “off-the-shelf” products that providers can use readily.

• Continue to engage actively in field-based partnerships to leverage finite resources toward stimulating broad adoption of patient safety practices by health care providers.

Evaluation Limitations and Considerations

In evaluating this large program, we had to choose where to apply our finite funding to capture most effectively the key activities, experiences, and progress of its activities. We also had to work within the limitations of the 4-year term of the evaluation, which limited our ability to track some multiyear activities and to monitor effects on many outcomes. In particular, we could not obtain field data on the extent to which health care providers were actually adopting the practices that AHRQ-funded research was finding to be beneficial for safer care. AHRQ had the foresight to anticipate the need for such data. It funded a 2-year addition to the evaluation contract, in which we have collected some data on adoption and have developed instruments that AHRQ can use to do so in the future.

Conclusion

By 2006, after 7 years of the patient safety initiative, the country was still at a point of low rates of safe practice adoption. We could not know then whether we were standing at the threshold of increasing adoption rates for safer practices, or whether adoption would continue to be slow. To encourage growth in adoption, AHRQ will need to continuously reinforce adoption activities through dissemination of information and tools to support practices and continued active partnerships with organizations that are leading related initiatives in the field.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.