Assessment of Contributions to Patient Safety Knowledge by the Agency for Healthcare Research and Quality-Funded Patient Safety Projects

Abstract

Objective

To characterize the activities of projects funded in Agency for Healthcare Research and Quality (AHRQ)' patient safety portfolio and assess their aggregate potential to contribute to knowledge development.

Data Sources

Information abstracted from proposals for projects funded in AHRQ' patient safety portfolio, information on safety practices from the AHRQ Evidence Report on Patient Safety Practices, and products produced by the projects.

Study Design

This represented one part of the process evaluation conducted as part of a longitudinal evaluation based on the Context–Input–Process–Product model.

Principal Findings

The 234 projects funded through AHRQ' patient safety portfolio examined a wide variety of patient safety issues and extended their work beyond the hospital setting to less studied parts of the health care system. Many of the projects implemented and tested practices for which the patient safety evidence report identified a need for additional evidence. The funded projects also generated a substantial body of new patient safety knowledge through a growing number of journal articles and other products.

Conclusions

The projects funded in AHRQ' patient safety portfolio have the potential to make substantial contributions to the knowledge base on patient safety. The full value of this new knowledge remains to be confirmed through the synthesis of results.

Background

One of the central activities of the Agency for Healthcare Research and Quality (AHRQ) patient safety initiative has been the use of its appropriated funding to support research and development work to expand knowledge on patient safety epidemiology and practices. At the start of the initiative, AHRQ identified a set of priorities to guide its funding choices (Keyes et al. 2005). AHRQ was informed in that task by two sources. The first was a report by the Quality Interagency Coordination Task Force (QuIC), which described >100 actions that the QuIC and its participating agencies planned to take to improve patient safety, including expansion of the knowledge base about the causes and prevention of errors (QuIC 2000a).1 The second was a research agenda generated from the first National Summit on Patient Safety Research convened by the QuIC in September 2000 (QuIC 2000b).

The history of AHRQ funding for patient safety grants is presented in Table 1 (see Appendix A for project group summaries). Each group of projects addressed different aspects of the priority topic areas and methodological issues identified by the QuIC and Summit research agenda. The first project group funded was the systems-related best practice grants, for which the Request for Application (RFA) was released in late 1999 (AHRQ 1999), as the Institute of Medicine (IOM) report To Err Is Human (2000) was being published. RFAs for another six groups of projects were released in fiscal year (FY) 2001 as AHRQ launched the patient safety initiative (AHRQ 2000a, b, 2001a–d). Subsequent groups of grants differed from these first two groups in their greater focus on testing the implementation of patient safety practices and requirements for cost sharing by the grantee organizations. These included the Challenge grants (FY 2003), several groups of health information technology (IT) grants (FY 2004), and the “Partnerships in Implementing Patient Safety” grants (FY 2005) (AHRQ 2003a–d, 2004a, b, 2005). In FY 2006, AHRQ funded a set of projects to examine use of simulation techniques to improve patient safety (AHRQ 2006).

Table 1.

History of AHRQ Funding for Groups of Patient Safety Grants, FY 2000–2005

Fiscal Year	Type of Grant	Number of Grants Funded	Grant Period	Annual Funding Amount
FY 2000	Systems-related best practices	6	Up to 3 years	$2 million
FY 2001	Six groups of patient safety grants			$50 million
	Centers of excellence	3	5 years
	Developmental centers	18	3 years
	Reporting demonstrations	16	3 years
	Dissemination and education	6	Up to 3 years
	Clinical informatics	11	Up to 5 years
	Working conditions	5	Up to 3 years
FY 2001	Working conditions grants	16	Up to 3 years	$7 million
FY 2003	Challenge grants			$4 million
	Implementation	7	2 years
	Risk assessment	6	1 years
FY 2004	Health information technology grants			$60 million
	Planning	38	1 years
	Implementation	40	3 years
	Demonstration of value	26	Up to 3 years
	State and regional demonstrations	5	5 years
FY 2005	Partnerships in implementing patient safety	17	Up to 2 years	$3 million
FY 2005	Health information technology implementation grants	14	3 years	$7 million
FY 2006	Improving patient safety through simulation research	19	Up to 2 years	$2.4 million

Reflecting their central role in the patient safety initiative, the funded projects were a core focus of the work of the patient safety evaluation center, which AHRQ funded in 2002. Within the five-component system framework established to guide our process evaluation (see Farley and Battles 2008, in this issue), these projects should be contributing to the two knowledge development components, with the resulting expanded knowledge then informing and supporting practice improvements in the field. Throughout the 4-year term of the evaluation (2002–2006), we examined the potential for each new group of projects funded through 2005 to make such contributions to knowledge on patient safety issues and practices.

We use the term “potential to contribute to knowledge” because actual contributions could not be documented or assessed until after the project results were published in the scientific literature and could be tested against standards of evidence. This long process would not be completed for these projects within the 4-year evaluation timeline. Using information from the project descriptions and proposals, however, we could characterize their potential contributions by examining the mix of patient safety issues and health care settings being addressed by the projects, as well as the practices being tested, and their attention to special populations.

Purpose of This Evaluation Task

To assess the potential contributions of the patient safety projects funded by AHRQ, we characterized the nature of the projects, and we assessed whether the practices they addressed were among those requiring additional evidence on effectiveness. Specifically, we sought to address the following evaluation questions:

1. What patient safety issues, practices, and settings are being examined by the projects funded by AHRQ through its patient safety initiative?

2. To what extent are these projects contributing new knowledge regarding practices for which additional scientific evidence is needed?

3. What progress has been made by the projects in documenting effects of patient safety practices and tools on patient safety outcomes?

We focused separately on the projects' potential contributions to epidemiological knowledge and to knowledge on the effectiveness of patient safety practices. We also examined trends in the generation of products from these projects to assess their progress in documenting their findings (question 3).

Methods

We used three types of data in this evaluation task: information abstracted from proposals for the patient safety projects, information on evidence for safety practices from the AHRQ Evidence Report on Patient Safety Practices (Shojania et al. 2001), and published products from the projects through June 2006. An Access database was developed containing data on each project in AHRQ' patient safety portfolio, including projects funded through FY 2005.

Characterization of the AHRQ-Funded Patient Safety Projects

The patient safety portfolio contained a total of 234 projects funded by AHRQ in FY 2000 through FY 2005. AHRQ provided us copies of the proposals for all of these projects.

Two types of information were abstracted from the proposals: (1) basic project information and (2) categorical variables characterizing the nature of the projects.2 Basic project information, including the project title, AHRQ project number, RFA number, principal investigator (PI), institution, and project start and end dates, was obtained directly from the project proposals or AHRQ master file. The evaluation center leadership team defined the coding categories for the categorical variables that characterized the projects (e.g., patient safety issue addressed, health care setting). For the health IT projects, additional variables were defined to characterize the types and purposes of the health IT that the projects were addressing, which were adapted from IT taxonomies used in previous RAND work (Girosi, Meili, and Scoville 2005). Projects could be coded for more than one category for a given variable (e.g., a project could examine care in both inpatient and outpatient settings).

To provide data on projects' potential contributions to patient safety epidemiology, we included coding categories in the variable for patient safety actions for “studying the epidemiology of patient safety risks and hazards” and “studying the underlying causes of medical errors or adverse events.”3 If a project was studying epidemiological issues or causes of medical errors, it was categorized as generating knowledge in patient safety epidemiology, even if this was not its primary focus.

Variables for the 234 projects that were not provided from the AHRQ master file were coded manually from information in the proposal narrative. A data abstraction guide with coding instructions for each variable was developed and used to guide the abstraction. Abstraction was performed by a small team of coders, with a task leader supervising the process.

To calibrate coding by new abstractors, a sample of three proposals was coded by the task leader and abstractors. Responses were compared and discrepancies resolved. This process was repeated until no discrepancies occurred. Questions that arose during the coding process were resolved by the task leader. Coding questions were brought to the PI of the evaluation as necessary for a final determination. The task leader reviewed abstracted data from a random sample of proposals to ensure accuracy.

Potential Contribution to Evidence on Effective Patient Safety Practices

Information presented in the 2001 evidence report, Making Health Care Safer: A Critical Analysis of Patient Safety Practices (Shojania et al. 2001), on the status of evidence on the effectiveness of patient safety practices4 was used to assess the extent to which projects in the patient safety portfolio have the potential to contribute new evidence on the effectiveness of patient safety practices. A total of 79 patient safety practices were evaluated in the evidence report. The report categorized practices into five groups based on the strength of existing evidence regarding impact and effectiveness (greatest, high, medium, low, and lowest). The report also organized the practices into two research groups: “research likely to be highly beneficial” and “research likely to be beneficial” (Shojania et al. 2001). Some practices addressed in the body of the evidence report were not categorized by its authors on one or both of the dimensions because of a lack of evidence.

For each of the patient safety practices addressed by a given project, we used the evidence report' assessment of the practice to code (1) the strength of existing evidence on that practice and (2) the value of additional research for it. Projects that addressed more than one practice were coded for each practice.

Generation of Project Results and Products

The first step in expanding knowledge in the field about patient safety epidemiology and practices is the publication of results and generation of other products from the funded patient safety projects. Our analysis assessed the extent to which the projects were taking that first dissemination step.

We collected and summarized the products produced over time by the AHRQ-funded patient safety projects. Products were defined to include reports, journal articles, conference presentations, tools such as survey instruments, press releases, newspaper articles, and interviews. We identified products using information from: (1) the website of the Patient Safety Research Coordinating Center (Coordinating Center), (2) a review of the published literature, and (3) a manual review of the four-volume compendium, Advances in Patient Safety: From Research to Implementation (AHRQ 2005) (which were not listed in the standard health literature databases).

In the Coordinating Center website search, we collected all posted products that pertained to a particular project, primarily using the name of the project PI. Occasionally, the PI was not an author or otherwise linked to a product, but we were able to link the product to the PI' project using names of other key investigators.

In the literature review, we performed separate searches for each PI, identifying all articles published from 1997 to June 2006 on which the PI was an author. Multiple years of data were used to document publication trends, including years that predated the patient safety initiative. Search terms used were the PI' last name, first name or initial, and middle name or initial as appropriate for the database searched. The databases searched were BIOSIS, CINAHL, National Library of Medicine LocatorPlus, NTIS, PapersFirst, PubMed, and WorldCat.

In each year, a single project member reviewed the citations for each PI to eliminate duplicates, and then coded the patient safety relevance of the papers and other products. We assumed that any product found on the Coordinating Center website was related to patient safety. Products were also coded for type of product, including journal articles, AHRQ publications, conference presentations, editorials, reporting systems, bibliographies, book chapters, newspaper articles, software products, taxonomies, and government reports.

As we started the coding process, five projects were coded independently by the coder and the evaluation center PI. Their codings were compared, and refinements were made to the screening and coding logic to ensure consistency. The same person coded all products for the first three evaluation years, and she trained a second person who performed the coding for the final year. The product coding and the source of the citation were entered into ProCite.

Across the patient safety projects, 11 PIs had two funded projects each and four PIs had three funded projects. Thus, the 234 projects correspond to 215 unique PIs. For PIs who had multiple projects, we could infer that the product was a patient safety product, but we could not always determine the project to which it belonged. If a product was authored by a PI who led two projects, or was the product of collaboration between two PIs on separate projects, we gave all possibly relevant projects “credit” for that product. Thus, the unit of analysis in this assessment is “product–project” pair.

Analyses Performed

Descriptive tables were constructed to answer each of the questions addressed in this part of the evaluation. All analyses were performed using Stata. The analyses were performed for each of four categories of patient safety projects: (1) projects funded in FY 2000–2001, (2) Challenge grant projects, (3) health IT projects, and (4) Partnerships in Patient Safety projects (PIPS). We characterized the projects in each category by tabulating the number and percentage of projects coded as addressing several aspects of patient safety.

To assess the potential contribution of the funded projects to expanding the evidence base regarding effective safety practices, for each project group, we tabulated counts of the projects based on our coding for the strength of evidence behind the practices addressed in projects, and the extent to which further research would be beneficial as determined by the safety practices evidence report (Shojania et al. 2001). To assess the generation of project results and products, we tabulated the identified project products.

Results

Profiles of the Areas and Issues Addressed by the Funded Projects

This first set of results addresses the first evaluation question: What patient safety issues, practices, and settings are being examined by the projects funded by AHRQ through its patient safety initiative?

As shown in Table 2, an estimated 68 projects (29 percent) examined the epidemiology of patient safety or contributing causes of medical errors or adverse events. Projects funded in the early years of the patient safety portfolio were more likely to examine epidemiological issues. Fifty-six of the FY 2000–2001 projects (69 percent) and eight of the Challenge projects (62 percent) included epidemiology or root causes as part of their projects, while only three of the health IT projects (3 percent) and one of the PIPS projects (6 percent) did so. The most common patient safety issues addressed by these projects were general patient safety (54 percent) and medication ordering or administration (43 percent). Issues rarely addressed by the funded projects included nosocomial infections, falls, pressure ulcers, problems related to restraint use, handoffs, and equipment failures. One of our first recommendations to AHRQ was that additional funding on patient safety epidemiology should focus on the less studied issues (Farley et al. 2005). The focus of projects funded subsequently, however, shifted to implementing and evaluating health IT and safety practices in part due to Congressional mandates.

Table 2.

Patient Safety Issues Addressed by AHRQ-Funded Projects for Epidemiology or Root Cause Analysis

	Number of Issues by Type of Grant
Patient Safety Issue	FY 2000–2001	Challenge	Health IT	PIPS
Medication ordering/administration	21	5	2	1
Nosocomial infections	3
Falls, pressure ulcers, restraint related	3
Nurse staffing	9	1
Provider fatigue/working conditions	12
Surgical/invasive procedure errors	5	2
Diagnostic or treatment errors	11
Equipment/device failure	2	2
Ordering/administering blood	1
Care procedures and coordination*	1	1
Wrong patient/procedure/test
General patient safety	34	1	2
Handoffs
Other issues	4	2
Total number of issues studied	106	14	4	1
Number of projects studying epidemiology or root cause	56	8	3	1
Average number per project	1.9	1.8	1.3	1.0

^*Care procedures and coordination include errors in the admitting process, such as applying the wrong patient identification bracelet, misplaced documentation such as “lost” medical records, failure to notify patients of a positive test result, failure to register a patient in the emergency department resulting in delayed care and adverse outcome, etc.

AHRQ, Agency for Healthcare Research and Quality; FY, fiscal year; IT, information technology; PIPS, Partnerships in Patient Safety projects.

The AHRQ-funded projects addressed epidemiological issues in several health care settings, including settings for which little was known about patient safety epidemiology at the start of AHRQ' patient safety initiative. Over a third of the projects included outpatient settings, six included nursing homes, and one included home health care (data not shown). Settings identified by the evaluation as warranting additional attention were hospital ancillary services, long-term care, and home care (Farley et al. 2005).

Presented in Table 3 are tabulations of the number of AHRQ-funded patient safety projects that addressed each of a list of patient safety issues, including patient safety for special populations. We found variation in the issues addressed across groups of projects, but collectively, the projects addressed all but a small number of the issues listed. The FY 2000–2001 grants and Challenge grants addressed the broadest range of patient safety issues. General patient safety received the most attention by the FY 2000–2001 grants, but subsequent groups of grants focused on more specific issues. Medication ordering and administration was a primary focus for all four groups of grants. While handoffs (transitioning patients from one health care provider to another) and care procedures and coordination received little attention by the FY 2000–2001 grants, they were among the most common areas of focus by the health IT grants and PIPS grants. The PIPS projects focused on fewer issues than the previously funded projects. Issues receiving the least attention included nosocomial infections, falls and pressure ulcers, equipment or device failure, and blood ordering or administration.

Table 3.

Patient Safety Issues and Special Populations Addressed by the AHRQ-Funded Patient Safety Projects

	Number of Issues by Type of Grant
	FY 2000–2001	Challenge	Health IT	PIPS
Patient safety issue
Medication ordering/administration	29	7	60	9
Noscomial infections	4	2		1
Falls/pressure ulcers	6		2
Nurse staffing	12	2
Provider fatigue, working conditions	16	2		1
Surgical/invasive procedure errors	11	2	1
Diagnostic/treatment errors	18		37	3
Equipment/device failure	5	1
Ordering/administering blood		1
Care procedures and coordination*		2	60	3
Wrong patient/procedure/test		1
General patient safety	40	3	6
Handoffs	2	2	62	3
Other issues	4		2
Total number of issues studied	147	25	230	20
Average number per project	1.8	1.9	2.2	1.2
Special populations
Elderly	17	3	17	4
Minority populations	17	1	16	3
Low income	15	1	25	2
Health vulnerable	10	4	34	3
Other vulnerable	5	1	13	1

^*Care procedures and coordination include errors in the admitting process, such as applying the wrong patient identification bracelet, misplaced documentation such as “lost” medical records, failure to notify patients of a positive test result, failure to register a patient in the emergency department resulting in delayed care and adverse outcome, etc.

AHRQ, Agency for Healthcare Research and Quality; FY, fiscal year; IT, information technology; PIPS, Partnerships in Patient Safety projects.

Table 3 also shows that many of the projects in each of the groups addressed issues for special populations. These included the elderly, minority populations, low-income populations, the health vulnerable, or other vulnerable individuals.

Table 4 presents the patient safety actions undertaken by the grants in the four groups as well as the health care settings in which the projects conducted their work. Each of the project groups engaged in a wide variety of patient safety actions, although the most common types of actions also varied by project group. For example, the most common actions among the FY 2000–2001 projects were health professional education and awareness, monitoring and reporting adverse drug events, and provider proficiency or training to prevent errors, while the most common actions among the health IT projects were altering physical environment/infrastructure and the use of technology to prevent errors (diagnostic, medication, or other types of errors). The most common actions among the PIPS projects were increasing patient or consumer awareness of patient safety, protocols to prevent errors (medication errors or other types of errors), and provider proficiency or training to prevent errors. The projects also take place in a variety of settings, with hospitals (either inpatient acute care or the entire hospital) and outpatient clinics/providers' offices being the most common.

Table 4.

Patient Safety Actions and Settings Addressed by the AHRQ-Funded Patient Safety Projects

	Number of Actions and Settings by Type of Grant
	FY 2000–2001	Challenge	Health IT	PIPS
Patient safety action
Administrative actions to prevent error	2	1		2
Altering physical environment/infrastructure	12	3	67	1
Altering staffing/work conditions/scheduling	14	2		3
Effecting change in patient safety culture	15	2	3	2
Health professional education/awareness	33	1	3	3
Monitoring/reporting adverse drug events	27	4	7	1
Monitoring/reporting adverse events	24	2	4	1
Patient/consumer awareness of patient safety	4	1	1	6
Patient/family communication of errors	11
Protocols to prevent nonmedication errors	13	4	13	4
Protocols to prevent medication errors	15	6		5
Provider proficiency/training to prevent errors	25	3	4	4
Use of technology to prevent diagnostic errors	4		22	1
Use of technology to prevent medication errors	17	6	47	3
Use of technology to prevent other errors	6	2	34	3
Risk assessment—prospective		9		1
Risk assessment—retrospective		4
Other	5
Unclear			4
Total number of actions	227	50	209	40
Average number per project	2.8	3.8	2.0	2.4
Health care setting
Outpatient clinic, provider' office	28	2	74	4
Inpatient acute care	28	6	10	9
Hospital ancillary (e.g., laboratory, radiology)	3	1
Hospital outpatient diagnosis or treatment	9		8	2
Entire hospital	16	3	54	2
Community-based diagnosis or treatment	2		1
Nursing home or inpatient rehab care	10	3	16
Home care	3		9
Health system	15	2	9	2
Health profession educational setting	14		2
Behavioral health			2
Hospice			2
Other	1		3
Total number of settings	129	17	190	19

AHRQ, Agency for Healthcare Research and Quality; FY, fiscal year; IT, information technology; PIPS, Partnerships in Patient Safety projects.

New Contributions of AHRQ-Funded Projects to Evidence on Practices

These results address the second evaluation question: To what extent are these projects contributing new knowledge regarding practices for which additional scientific evidence is needed?

We found that many of the patient safety projects addressed practices that the evidence report identified as requiring additional scientific evidence to determine their effectiveness. We list in Table 5 the levels of strength of evidence rated by the evidence report and, for each level of evidence, we report the number of projects (by type of grant) that were coded as addressing practices at that level. We also present the number of projects that addressed practices that the evidence report identified but did not categorize due to a lack of evidence. Some projects addressed more than one practice listed in the evidence report and so are counted more than once.

Table 5.

AHRQ-Funded Projects Covering Evidence Report Chapters^*

	Number of Practices by Type of Grant
	FY 2000–2001	Challenge	Health IT	PIPS
Evidence report: impact and effectiveness
Greatest strength of evidence	0	4	1	3
High strength of evidence	15	2	4	3
Medium strength of evidence	10	5	37	3
Lower impact or strength of evidence	15	5	8	4
Lowest impact or strength of evidence	0	1	10	4
Evidence report: further research
Likely to be highly beneficial	23	11	37	4
Likely to be beneficial	15	6	16	8
Evidence report: not rated, but covered in projects	104	5	53	5
Practice not addressed in evidence report	6	3	2

^*Many projects are addressing health information technology that falls into more than one category.

AHRQ, Agency for Healthcare Research and Quality; FY, fiscal year; IT, information technology; PIPS, Partnerships in Patient Safety projects.

Across all four projects groups, 102 of the projects addressed practices that the evidence report identified as having only medium, lower, or lowest “impacts or strength of evidence” for effectiveness. In addition, 120 projects addressed practices for which the evidence report indicated that further research would be beneficial or highly beneficial.

Early in the evaluation, we recommended that AHRQ focus future research funding on testing patient safety practices that are promising, but for which evidence was lacking or insufficient (Farley et al. 2005). We also recommended that studies go beyond examining effectiveness to also consider the barriers or unintended outcomes involved in implementing a specific patient safety practice, as well as cost and return on investment assessments to make the business case for their adoption (Farley et al. 2005, 2007c). As the knowledge base on safety practices grew, we recommended AHRQ update its safety practices evidence report (Farley et al. 2005, 2007a, c).

Products Generated from Patient Safety Grantees

These results take the first step toward addressing the third evaluation question: What progress has been made by the projects in documenting effects of patient safety practices and tools on patient safety outcomes?

We found that the projects have generated growing numbers of papers and other products from their work over time, which depending on the strength of their research, have the potential to contribute to knowledge on the effects of practices. This question cannot be answered fully, however, until these results are synthesized and projects assessed for methodological strength and the integrity of their results.

Across the 4 years of the evaluation, we found a total of 1,846 documents related to patient safety authored by AHRQ-funded patient safety grantees. A total of 437 of these documents were deemed to be related to the grantee' AHRQ-funded projects as of July 2006, which corresponds to 446 product–project pairs (table available on web).

The number of products emerging from the patient safety portfolio increased substantially over time as earlier projects were completed or drew closer to completion. Through July 2006, the project-related products represented 23.7 percent of all safety-related products generated by the projects' PIs, having grown from 8.9 percent of the total for 2001–2004. Among the 446 product–project pairs identified, four were published in 2001, 41 in 2002, 74 in 2003, 123 in 2004, 162 in 2005, and an additional 30 in January through July 2006. Of the 437 unique patient safety products identified as funded by AHRQ, 273 were journal articles, 40 were AHRQ publications, 65 were conference presentations or proceedings, and the remainder was small numbers of each of the other types of products coded.

The predominance of journal articles and conference presentations or proceedings as products from the funded projects raised concerns about the extent to which information generated by the projects would reach front-line practitioners and be translated into practice. We recommended that AHRQ develop and implement a strategy, including collaborative efforts with other organizations, to synthesize and disseminate newly developed knowledge from its funded projects, and to develop implementation tools and guides to reduce barriers in the implementation process (Farley et al. 2005, 2007a,c).

Discussion

In its report, To Err Is Human: Building a Safer Health System, the IOM recommended three key roles for AHRQ in leading a patient safety initiative (Institute of Medicine 2000). Through the wide variety of projects it has funded, AHRQ has progressed considerably toward fulfilling each of these roles. The first role identified was to fund research that assessed the magnitude of medical errors and identified the role of human error in causing medical errors. Our evaluation results suggest that the projects examining the epidemiology of patient safety and underlying causes will provide new knowledge on a wide variety of patient safety issues, ranging from medication ordering and administration to working conditions, and nurse staffing to equipment failures. The projects also extend their work beyond the hospital setting into settings for which there was little evidence at the start of the patient safety initiative.

Most of the focus on patient safety epidemiology was very early in the patient safety initiative and declined as later rounds of funding shifted focus to health IT and practice implementation. Some level of ongoing monitoring is important to identify shifts in trends for patient safety epidemiology, and to identify emerging patient safety risks and hazards that need to be addressed.

The second AHRQ role identified by the IOM was to test and evaluate approaches to prevent errors (Institute of Medicine 2000). The funded projects also made strides toward this goal through their work with patient safety actions such as educating providers to increase their awareness, implementing protocols to reduce the risk of errors, and implementing health IT to alter the physical work environment. Furthermore, many of the funded projects implemented and tested practices for which the patient safety evidence report (Shojania et al. 2001) identified a need for additional evidence regarding their effectiveness, and for which it identified that additional research would be beneficial or highly beneficial.

The identification of the practices being addressed by the funded projects is, however, only the beginning of the story. It provides no information on the quality of the projects or their experiences, which were topics addressed by other evaluation tasks. In this issue, Taylor et al. (2008) discuss the experiences of the FY 2000–2001, Challenge, and PIPS projects that implemented safety practices, which were assessed from interviews with project PIs that were conducted early on and at the end of their projects. Also in this issue, Damberg et al. (2008) present the early implementation lessons learned by grantees implementing health IT. An assessment of the study designs used by the 45 FY 2000–2001 grantees that implemented interventions, the implementation Challenge grants, health IT value grants, and PIPS grants found that only 37 percent of the evaluations used in these projects had characteristics of a controlled trial, such as using a control group and either population or randomized samples. This highlights the need for the creation of standards of evidence for study designs beyond controlled trials (Farley et al. 2007b).

The third AHRQ role suggested by the IOM was the dissemination of information to improve patient safety, which is consistent with AHRQ' stated function as a “science partner” that assists the health care system to translate research into practice (Clancy, Slutsky, and Patton 2004). In our process evaluation, we identified three basic steps that AHRQ needs to take to be successful in disseminating new knowledge on patient safety epidemiology and effective practices: (1) synthesis of the aggregate results of grantees' research and field tests; (2) development of packaged products and tool kits; (3) dissemination of this information to providers and policy makers (Farley et al. 2007a). Through the assessment reported in this paper, we confirmed that the projects have been generating a growing body of new patient safety knowledge comprised of written papers and other products from their work. The full value of this new knowledge, however, will depend on the integrity of their research methods and findings, which remains to be confirmed through the synthesis of results—the first step of the dissemination process defined above.

In separate evaluation tasks, we tracked AHRQ' development and execution of its dissemination activities, which are intended to bring the results of the funded projects' work to health care providers in the field as well as to other stakeholders in patient safety. As of the end of the evaluation in September 2006, AHRQ was in the early stages of preparation to perform the syntheses of project results. At the same time, the National Quality Form used the published findings from the projects as an important part of the scientific foundation for updating its list of safe practices, which has been published in its report, Safe Practices for Better Healthcare: 2006 Update—A Consensus (National Quality Forum 2007). See Chapter 6 of the final patient safety evaluation report for details on these dissemination activities (Farley et al. 2007b).

Another important use of the information on the funded projects is to guide decisions by AHRQ regarding priorities for types of projects to be funded in the future. Our evaluation has identified several patient safety issues, practices, or settings for which limited work has been done by existing projects. Some examples include behavioral health, hospital ancillary services, community-based diagnostic services, and consumer/patient awareness and roles in enhancing safety. More knowledge on safety issues and practices in these settings remains to be developed.

Conclusions

The projects funded in AHRQ' patient safety portfolio have the potential to make substantial contributions to the knowledge base on patient safety. While they address many important patient safety issues and safety practices, there are still areas where additional work is needed. The full value of this new knowledge remains to be confirmed through the synthesis of results, along with dissemination of the knowledge gained for use by the health care community.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Please note:Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.