Incorporating evidence review into quality improvement: meeting the needs of innovators

Abstract

Background

Achieving quality improvement (QI) aims often requires local innovation. Without objective evidence review, innovators may miss previously tested approaches, rely on biased information, or use personal preferences in designing and implementing local QI programmes.

Aim

To develop a practical, responsive approach to evidence review for QI innovations aimed at both achieving the goals of the Patient Centered Medical Home (PCMH) and developing an evidence-based QI culture.

Design

Descriptive organisational case report.

Methods

As part of a QI initiative to develop and spread innovations for achieving the Veterans Affairs (VA) PCMH (termed Patient Aligned Care Team, or PACT), we involved a professional evidence review team (consisting of review experts, an experienced librarian, and administrative support) in responding to the evidence needs of front-line primary care innovators. The review team developed a systematic approach to responsive innovation evidence review (RIER) that focused on innovator needs in terms of time frame, type of evidence and method of communicating results. To assess uptake and usefulness of the RIERs, and to learn how the content and process could be improved, we surveyed innovation leaders.

Results

In the first 16 months of the QI initiative, we produced 13 RIERs on a variety of topics. These were presented as 6–15-page summaries and as slides at a QI collaborative. The RIERs focused on innovator needs (eg, topic overviews, how innovations are carried out, or contextual factors relevant to implementation). All 17 innovators who responded to the survey had read at least one RIER; 50% rated the reviews as very useful and 31%, as probably useful.

Conclusions

These responsive evidence reviews appear to be a promising approach to integrating evidence review into QI processes.

Introduction

Context-responsive local innovation is often required for achieving successful organisational and provider behaviour change. Such innovation can provide the tools and locally appropriate policies and procedures that enable national policies or guidelines to succeed. However, these QI innovation efforts, typically carried out through quality improvement (QI) initiatives, are often expensive and may not be successful.^{1 2} Access to relevant prior evidence could potentially improve QI effectiveness and success. There is little documentation, however, showing that systematic approaches to assessing prior evidence are routinely integrated into QI efforts.^3–5 One reason may be a mismatch between traditional evidence reviews and QI innovator needs.

Traditional evidence reviews, such as those published in journals, often do not meet the specific needs of, and consequently may be underused by, QI innovators. Undertaking new reviews of this type is expensive and time-consuming; and use of existing reviews is limited by their goals. Traditional evidence reviews are primarily directed at establishing whether an innovation is effective. The yes/maybe/no answers provided by these reviews are extremely useful for a wide variety of purposes, such as policy making, guideline development, and large-scale organisational decision making. These reviews, however, include few of the intervention, evaluation, or contextual details needed by local QI teams (who generally work within established organisation guidelines and policy) to determine whether and how to apply the interventions to their own situations. Some teams are at the earliest stages of project conception and require a topic overview, while others may have a specific area of uncertainty for a project they have been working on for some time. Additionally, even if relevant reviews are available, innovation teams may lack the resources and expertise to find or assess them. Teams may have little prior experience with using evidence review, formulating QI questions, planning and implementing multidisciplinary interventions and evaluating outcomes. Finally, the limited duration typical of QI projects demands a more rapid review result than can be achieved using traditional evidence review methods. The Responsive Innovation Evidence Review (RIER) project aims to provide a reasonably objective and rapid approach to evidence review to help innovators access evidence that is relevant and tailored to their needs.

The use of rapid reviews in healthcare is increasing and audiences for these reviews are diverse. They include policy makers, healthcare providers and managers, and patients and their representative organisations; all of whom desire evidence-based recommendations on emerging issues provided in a timely manner.^6–10 Methods for rapid review are evolving and are not yet standardised.^6–9 One approach being used to support policy makers involves the use of a database of policy-relevant systematic reviews with or without additional services.^{11 12} Rapid reviews may limit literature searches by years, databases, or language; and may use sources beyond electronic searches.⁸ Rapid reviews also frequently emphasise user-friendliness and end-user focus.⁹ Although brief reports have been found to be a useful component of a health technology assessment programme,¹³ there is no agreement as to the impact of the strategies used to expedite evidence reviews on quality and risk of bias.^6–9 Additionally, few rapid review efforts have sought to address the unique needs of local QI innovators directly. The RIER project adds to the evolving literature on rapid review methods by emphasising the specific needs of QI innovators.

This paper describes the process and tools used to introduce responsive evidence review into innovation projects; discusses the approaches considered and the search strategies used; and reports on a survey of innovation team members regarding uptake and usefulness of the evidence review programme with suggestions for improvement. Our evaluation questions are: (1) are review results perceived as useful by QI innovators? and (2) does the new rapid review approach show promise as a feasible method for delivering timely and relevant reviews?

Methods

Project

The RIER project is a component of the Veterans Affairs (VA) VAIL-PACT initiative. PACT (Patient Aligned Care Team) represents the VA effort to implement the Patient Centered Medical Home (PCMH).^14–17 VAIL (VISN 22 Veterans Assessment and Improvement Laboratory) is a large VA research/clinical effort in the Southern California region that aims to promote evidence-based QI innovation and to develop an evidence-based QI primary care culture. VAIL involves (1) six primary care QI demonstration practices in three different VA medical systems and (2) four interdisciplinary, cross-site workgroups on specific topics (eg, homelessness). Each demonstration practice is overseen by an interdisciplinary quality council that reviews and monitors local QI efforts. A regional steering committee prioritises innovations submitted by quality councils or workgroups.

Evidence review workgroup

The Evidence Review Workgroup was created to explore and test a way of introducing evidence review into ongoing and future innovation projects that are part of the VAIL-PACT project. The workgroup includes Southern California Evidence-based Practice Center (EPC) and VA researchers, the EPC reference librarian, and the EPC project assistant. The Southern California EPC is part of the EPC Program established by the Agency for Healthcare Research and Quality (AHRQ) to produce the AHRQ Evidence Reports.

Innovator–evidence review workgroup interaction

To help focus and formulate the evidence review requests, the Evidence Review Workgroup developed a request form (see online supplementary appendix A). The request form has undergone several revisions over time to maximise its effectiveness in practice. The current version requires the innovation teams to specify which problem, area of concern, or outcome measure the team aims to improve. The form also asks teams to formulate specific questions they want to have answered. Finally, the form requires the team to specify which innovations are being considered.

Often, innovation teams have already identified a particular area or intervention category, or are limited by system constraints. In some cases, this narrows the list of possible QI interventions reviewed. In other cases, based on clinical judgment, Evidence Review Workgroup staff may determine it is necessary to broaden the review beyond the identified interventions.

Search strategy

The overviews employ a number of search strategies selected for their effectiveness in identifying relevant information. The approach and features that are now in RIERs were developed over the course of the project. Searches are performed by the EPC reference librarian, and studies are typically selected by one reviewer; data are abstracted by a graduate student and checked by an EPC systematic reviewer; and the literature flow is managed by the EPC project assistant.

Every RIER includes a search for systematic reviews on the topic of interest. For this, we use the Systematic Review Clinical Query function in the PubMed database. We also search the Database of Abstracts of Reviews of Effects (DARE) which covers published systematic reviews, Cochrane reviews and AHRQ Technology Assessment (TA) reports. If exploratory searches reveal key articles, we use these as seed articles for additional searches. For this, we employ the PubMed Related Citations function for automated relevance ranking, and we conduct Forward Searches in the Web of Science by identifying articles that cited the seed articles.

Since innovators are often interested in research information stemming from comparable systems, we may restrict searches to the VA setting using the author affiliation field coding in PubMed. Additionally, some of our searches target research in settings with established PCMH models. We, therefore, developed a PCMH search filter using an article set maintained by the VAIL-PACT project. This article set contains publications considered relevant for the development of medical home-based QI innovations. We used it as the reference standard and iteratively tested and improved the filter. The PCMH search strings for the PubMed, CINAHL and EMBASE databases are presented in box 1.

Box 1. PCMH Search filter.

• Database PubMed:

• “patient centered medical home” OR “patient centered medical homes” OR “medical home” OR “medical homes” OR “advanced medical home” OR: “advanced medical homes” OR “patient-aligned care team” OR “patient aligned care team” OR “patient-aligned care teams” OR “patient aligned care teams” OR pcmh

• NOT

• letter[pt] OR editorial[pt] OR editorial* OR letter*

• Database CINAHL:

• “patient centered medical home” OR “patient centered medical homes” OR “medical home” OR “medical homes” OR “advanced medical home” OR “advanced medical homes” OR “patient-aligned care team” OR “patient aligned care team” OR “patient-aligned care teams” OR “patient aligned care teams” OR pcmh

• NOT

• letter[pt] OR editorial[pt]

• Search modes—Phrase Searching (Boolean)

• Database EMBASE:

• “patient centered medical home” OR “patient centered medical homes” OR “medical home” OR “medical homes” OR “advanced medical home” OR “advanced medical homes” OR “patient-aligned care team” OR “patient aligned care team” OR “patient-aligned care teams” OR “patient aligned care teams” OR pcmh

To identify the ‘How to’ information for design and implementation of the innovations, we employ Google searches, consult content experts, or identify resources referenced in research publications (reference mining).

In deciding which articles or ‘How to’ sources to include in the evidence overviews, we typically use the following priority criteria: (1) systematic reviews on relevant topics, (2) articles published by established content experts in peer-reviewed journals and (3) manuals and tools found on websites created by recognised organisations conducting relevant QI studies.

Format for RIERs overviews

RIERs use a standard format aimed at enabling a diverse innovator group to access and interpret the information. Overviews include the review questions or objectives, the methods employed to answer the questions, the results of the review, the references cited, and an evidence table summarising the key articles. The use of evidence tables is an established method in systematic reviews to enable comprehensive literature overviews. The RIERs provide sufficient detailed information to allow innovators to obtain more details if desired. The three primary components of the overviews are: (1) an effectiveness assessment of the likelihood that the general approach proposed for the innovation could result in the desired outcomes; (2) ‘how to’ information on designing and implementing the innovations with access to online sources; (3) a summary of key articles in an evidence table. The focus and scope of the reviews determines the weight given to each of the components.

Dissemination

To introduce the evidence review programme, we presented 11 RIERs orally at a project collaborative (a meeting of leaders from the quality councils, workgroups and innovation teams) at which progress for each innovation project was presented. The pertinent evidence overviews were presented together with the progress reports. Written copies of the RIERs were also available. Later, RIERs were emailed or distributed in written form, and all RIERs were placed on a SharePoint site available to members of the quality councils, workgroups and innovation teams.

Evaluation survey

To test the response of members of the quality councils, workgroups and innovation teams to the evidence overviews, we conducted an online survey. The survey asked respondents to assess and comment on the usefulness of RIERs and on how both the content and the process of obtaining the RIERS could be improved. The survey was anonymous and was designed to capture participant experiences over the first 16 months of the programme.

Results

Evidence reviews

In the first 16 months of the project, the team provided 13 RIERs covering the following topics: Advanced Access, Homelessness, Interactive Communication, Motivational Interviewing, PCMH Evaluation Measures, Patient Registries, PCMH and Mental Health, Pharmacists, Primary Care Team Functioning, Readmissions, Relational Coordination, Secure Messaging, and Self-Management (table 1).

Table 1.

Description of responsive innovation evidence reviews (RIERs)

RIER	Project (Innovation team)	Number of pages (text word count)	Questions/topics addressed	Number of citations	Articles in evidence table
Advanced access	VISN system redesign open access project (VAIL leadership)	9 (1390)	▸ What is advanced access? ▸ What is the evidence that advanced access works? ▸ Resources for advanced access implementation	7	7
Homelessness	Home telehealth and vet-to-vet in HUD-VASH (VA supportive housing (VAIL homelessness workgroup, supporting access to primary care for homeless veterans)	6 (961)	▸ Elements of programmes directed toward the homeless ▸ Evidence of outcomes ▸ Resources	5	5
Interactive communication	Mental health integration into primary care PACT (primary care/mental health provider communication workgroup)	15 (1159)	▸ What are some effective and efficient evidence-based strategies to allow for two-way communication between primary care and mental health? ▸ What are the known barriers and facilitators of communication among primary care and mental health providers?	34	31
Motivational interviewing	Informing project selection, no project resulted from the topic (VAIL affiliated staff)	13 (2057)	▸ Is there an abbreviated method for training providers in this technique? ▸ What are the essential elements necessary to successfully motivate a patient? ▸ What is the time commitment needed to motivate individuals to change their behaviour using this technique? ▸ Are there any gender differences requiring modifying intervention and/or affecting outcomes? ▸ Costs of the intervention	60	23
PCMH evaluation measures	VAIL project evaluation (VAIL evaluation team)	6 (504)	▸ Evaluation measures used in PCMH interventions	3	3
Patient registries	RN disease managers role in PACT (Oceanside quality council)	7 (925)	▸ Use of registries ▸ Resources	7	7
PCMH and mental health	Mental health integration into primary care PACT (Sepulveda ambulatory care clinic quality council)	8 (1125)	▸ Mental health and primary care integration—A spectrum of models ▸ Situation in the VA ▸ Other resources	21	18
Pharmacists	Reducing walk-in visits for pharmacy refills (sepulveda ambulatory care clinic quality council)	10 (2094)	▸ Prescription refills ▸ The bigger picture: medication therapy management ▸ Role of the pharmacist in primary care ▸ Summary ▸ Resources	15	8
Primary care team functioning	Team based communication mini-TEX (teamlet evaluation exercise) (VAIL education workgroup)	8 (1409)	▸ How do we measure team functioning? ▸ What do we know about team effectiveness within primary care? ▸ An example of care team assessment within the VA ▸ How can team development be assessed? ▸ Examples of questions from various care team functioning instruments	14	5
Readmissions	PACT posthospitalisation telephone intervention (Redlands Boulevard outpatient clinic quality council)	9 (697)	▸ Readmission reduction strategies ▸ Readmission rate as an outcome—Some concerns ▸ Elements of a readmission reduction intervention	11	11
Relational coordination	Team based communication Mini-TEX (teamlet evaluation exercise) (VAIL education workgroup)	10 (578)	▸ What is relational coordination? ▸ Measuring and analysing relational coordination ▸ Relational coordination and primary care ▸ Resources	13	11
Secure messaging	Point-of-care enrolment of veterans in MyHealtheVet (VA online personal health record) (redlands boulevard outpatient clinic quality council)	7 (960)	▸ Secure messaging ▸ Patients using secure messaging ▸ Evidence for outcomes ▸ Implementation challenges ▸ Resources	9	9
Self-management	RN disease managers role in PACT (Oceanside quality council)	8 (1370)	▸ Does self-management support work? ▸ Twelve evidence-based principles for self- management support implementation in primary care ▸ Implications for PACT	7	6

Note: Excluding search terms, tables, figures, references.

Note: VISN (Veterans Integrated Service Network); VAIL (Veterans Assessment and Improvement Laboratory); PACT (Patient Aligned Care Team); HUD-VASH (US Dept. of Housing and Urban Affairs—VA Supportive Housing).

PCMH (Patient Centered Medical Home); VA (Veterans Affairs); RN (registered nurse). The abbreviation TEX is already in the text of the line where it is found and stands for (teamlet evaluation exercise).

All but two of the topics were selected based on innovation projects already approved by the VAIL Steering Committee and at various stages of planning, implementation and evaluation at the time the RIERs were initiated. The reviews varied based on the goals and questions of those requesting the reviews. Some of the reviews were primarily topic overviews, some were ‘how to’ reviews, and some addressed specific items. For example, the Homelessness review focused on homelessness only in the context of the PCMH for which the literature available was very limited.

The 11 reviews presented at the collaborative were initiated to support innovation projects, introduce the evidence review resource to the innovation teams, and encourage development of an evidence-based QI culture among the team members. These reviews were requested by the VAIL leadership based on discussions with the innovation teams, quality councils and workgroups. The quality councils oversee innovation activities within the participating demonstration sites and are composed primarily of clinicians with patient care and/or administrative responsibilities; workgroups include members from one or more demonstration sites and are generally composed of both clinicians with direct patient care responsibilities and academic researchers. The other two topics were requested directly by innovators after the collaborative meeting. Reviews were produced in 2–6 weeks depending on the project timeframe and requirements.

RIERs ranged in length from six to 15 pages with a mean of nine pages. The text was purposefully kept short with about 1000 words per topic (500–2100 words, mean 1171). The number of citations varied between three and 60 (mean 16, median 11) according to the review questions and purpose. Evidence tables described 3–31 key articles (mean 11 articles, median eight). Citations and key articles were included to provide sources that the innovation team members could refer to for additional information or guidance. In some reviews, citations and key articles were also used to illustrate the conclusions. We included systematic reviews where possible. For example, in the Self-Management review with six items in the evidence table, one article was a review of 83 reviews and meta-analyses; another, a review of 30 Cochrane systematic reviews; a third, a Cochrane review of 17 studies; and a fourth, a Cochrane review of 14 studies.

Two examples of RIERs are shown in online supplementary appendix B. The evidence review on Advanced Access was requested by the VAIL leadership to support a VISN system redesign project. The objective of the review was to provide an overview for innovators on what is being done and to generate ideas for new interventions. The evidence review on Interactive Communication was requested by the Primary Care/Mental Health Provider Communication Workgroup to provide information on effective and efficient ways of enhancing two-way communication between mental health and primary care providers and on the associated barriers and facilitators. The workgroup itself had 11 members—nine clinicians, two of whom had research experience, and two academic researchers. The clinicians consisted of primary care physicians, psychiatrists, nurse practitioners and a nurse manager. The workgroup also had an advisory group consisting of one primary care physician, three clinical social workers, and one researcher. The workgroup received additional input from the Sepulveda Primary Care/Mental Health Integration Workgroup.

Evaluation survey

Respondents

Seventeen out of 28 invited innovators (61%) responded to the rapid review survey. Seventy-five per cent of respondents were members of the site-specific management teams (quality councils), 31% were members of the functionally based workgroups, and 50% were members of individual innovation teams. At the time the RIERs were first introduced, 63% of respondents were involved with innovations in the implementation stage; 31% in the planning stage; only 6% in the evaluation stage; and no respondent was working on spread or dissemination of an innovation.

Over 80% of respondents had some experience with performing literature or information searches to support project planning or activities: of these, 71% had used PubMed or other academic databases, and 57% had performed Google or other internet searches. Close to 60% of respondents had requested or received help with literature reviews from project staff prior to the creation of the Evidence Review Workgroup.

Uptake

Most (88%) of the respondents were present for the RIER presentations at the project collaborative meeting when they were first introduced; and all respondents had read at least one of the write-ups. About 24% indicated that they had requested a rapid review following the May 2011 meeting. In all, 53% of respondents indicated that they would request a rapid review from the Evidence Review Workgroup in the future.

Satisfaction

Overall, 50% of respondents rated the RIERs as very useful, and 31% as probably useful. About 50% of respondents thought the RIERs were very useful in helping them to think more clearly and broadly about their areas of concern, and 44% of respondents stated that they were very helpful in helping them to identify next steps in the innovation process. Finally, 56% thought the RIERs were helping them to gain confidence in how the innovation project fits into established evidence from the literature. Examples of the qualitative comments regarding the RIERs are shown in box 2.

Box 2. Satisfaction with the service.

• I was very impressed by the scope of the reviews and the excellent summaries

• I think the presentations and summaries are fine. I would like to start using them during the planning stage because I really think they could be helpful in further developing our innovations.

• The evidence review is good already. I didn't fully understand what they provided until I saw the product at the May 2011 VAIL Collaborative in Costa Mesa. Now that I see what they do, I am more inclined to communicate with them and bounce ideas back and forth starting in the beginning phases of an innovation.

• The rapid reviews are very helpful. The ability to request a rapid literature review is an important asset in expediting project or programme development.

• Thank you for this wonderful resource. We will try to use it more effectively

Although respondents were generally positive about the content of the reviews, 26% indicated that they did not know how the process of requesting and receiving evidence overviews worked or felt it was not working very well. The only negative comments that we received stated that the process of requesting the reviews was confusing.

Suggestions for improvement

The respondents had a number of suggestions for improving the rapid review process. Most respondents felt that they would like to have more contact with the Evidence Review Workgroup staff to discuss the problem (87%) before the RIER is started. A substantial number would like to have contact both during the RIER production process (67%) and after the RIER is completed (60%). Examples of improvement suggestions are shown in box 3. Several respondents indicated that reminders about the service would be helpful since the resource may be forgotten in the day-to-day activities of clinical practice and working on the project.

Box 3. Example suggestions for improvement.

• It would be helpful to have a conversation with a VAIL Researcher prior to putting in a request to ensure that the review request is narrowly and adequately focused.

• Speak with the people developing the innovation and ask them what they are struggling with. As projects develop, it would be nice to have direct contact with the Evidence Review Workgroup, since other issues (especially related to evaluation) come up.

• Have the innovation team talk directly to those conducting the rapid review in order to get a sense of context for the overall innovation project. It would eliminate time wasted researching areas that are not within the scope of the innovation.

• Require the requestors to narrow the focus of the request and articulate clearly what they desire to know. Provide a mechanism for communication between the requestors and those conducting the review.

Discussion

Currently, routine local QI methods do not emphasise the integration of systematic evidence review and innovation design and evaluation. Efforts to bring evidence into practice, however, could be strengthened and magnified if evidence reviews were used by local QI teams. Additionally, the success of the QI projects themselves might be enhanced. We propose that further efforts to tailor evidence review approaches to the needs and constraints of QI innovators are needed. This paper reports on our ongoing efforts to develop such an approach to enable innovators to effectively use available information to guide innovation activities.

Preliminary results, based on experience with 13 RIERs, produced during the project's first 16 months, appear promising in terms of both user acceptability and the time frame within which the reviews were generated. Our innovator survey revealed that all respondents had read at least one of the RIERs. Fifty per cent rated the RIERs as very useful, and 31% as probably useful. Innovator process improvement comments focused on requests for more interaction with the Evidence Review Workgroup, suggesting that future efforts should explore how to facilitate this type of communication. The review team also met its goal of generating each review in a 1–2-month time frame.

The RIER approach also appears promising in terms of the strategy used to provide relevant and timely information while minimising bias. Our strategy first involves a search for systematic reviews, and then uses key seed articles to identify additional articles looking backward and forward in time. By using terms such as ‘VA’, and predeveloped search strategies such as ‘PCMH’, we can maximise the proportion of articles with relevant contextual information in the search results. The RIER approach may provide a useful foundation for additional investigation and methodological development.

As the programme evolved, we found there was a need to support the innovators in formulating review questions and deciding what kind of information would be most useful to obtain. We addressed this by developing a RIER request form. The RIER request form helps to focus the scope of the review and facilitate interaction between the review group and the innovators to ensure that the review meets innovator needs. In providing evidence summaries for regional managers and stakeholders, one study⁹ reported on an approach that also required collaborative development of a clear and effective research question and proposal.

Support for QI requires information about advances in evidence-based medicine as well as information about how to implement advances in routine practice.^{18 19} Traditional reviews typically emphasise the former. We found that the implementation information often desired by members of the innovation teams belongs to the latter. Frequently sought information included (1) whether an innovation had been implemented in a similar context²⁰; (2) how much designing and implementing the innovation would cost and (3) exactly how the innovation was carried out. Although the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines recommend including enough information about an intervention that it can be reproduced,²¹ most journal articles do not include or reference this type of information. We, therefore, searched resources such as Google Scholar and other relevant web sites to identify manuals, tools and other information from tested innovations that we identified in our searches. Links to these types of information are included in the reviews. Since innovators often need to extrapolate design and implementation strategies from one context to another, we facilitated contact with content experts by including relevant links in the reviews.^{10 13 22}

Questions to be explored differ among various types of evidence review stakeholders, including innovators, policy makers, managers and individual clinicians.^{14 15} Different needs among stakeholders requiring different evidence review methodologies suggest that rather than attempting to formalise the components of rapid reviews, emphasis should be placed on clearly describing what was done and on discussing potential bias and impact on validity of results.^{10 13 14} In this paper, we attempt to explore and document our experience in meeting the needs of one stakeholder group, rather than in developing a generic approach to rapid evidence review.

A limitation of our evaluation is the lack of formal cost assessment. By leveraging the resources of an evidence synthesis centre, we were able to carry out the reviews using limited amounts of experienced reviewer and librarian time. We do not know to what extent this would be feasible under other circumstances. Even with an experienced EPC librarian and experienced reviewers, we initially tried a variety of approaches in developing our search strategies, and ‘a learning period’ should be expected for organisations trying this type of review for the first time. Searches related to QI are often harder to capture due to the diverse nature of projects.²³ It should be noted that we did not keep track of the literature flow (eg, number of articles screened), and did not record the reasons for exclusion/inclusion of articles in order to adhere to rapid turnaround times for reviews. Another limitation of our approach is that we only assessed yield, uptake and satisfaction with the service; the accuracy of the RIERs was not formally tested. The Evidence Review Workgroup staff relied on their systematic review experience to produce valid and unbiased reports. Broad overviews, however, may miss issues that comprehensive systematic reviews will uncover. In a published comparison between rapid and full reviews, substantial differences were found; however, the authors also highlighted that the essential conclusions of the reviews did not differ extensively.^{10 13} Finally, we have not measured the impact of the RIERs on innovator design and implementation activities. Generally, literature on the impact of systematic evidence review and the interventions that encourage healthcare policy makers and managers to use evidence review is sparse.²⁴

In summary, based on the first 16 months of an ongoing VA evidence-based QI project aimed at primary care redesign, RIERs showed promise for increasing the impact of evidence review on QI initiatives.¹⁴ Additional research and development of systematic approaches for integrating prior evidence into QI are critical for maximising the sophistication and impact of QI efforts in healthcare organisations.