Abstract
Background
There is an often-held assumption that the engagement of clinicians and healthcare organizations in research improves healthcare performance at various levels. Previous reviews found up to 28 studies suggesting a positive association between the engagement of individuals and healthcare organizations in research and improvements in healthcare performance. The current study sought to provide an update.
Methods
We updated our existing published systematic review by again addressing the question: Does research engagement (by clinicians and organizations) improve healthcare performance? The search covered the period 1 January 2012 to March 2024, in two phases. First, the formal updated search ran from 1 January 2012 to 31 May 2020, in any healthcare setting or country and focussed on English language publications. In this phase two searches identified 66 901 records. Later, a further check of key journals and citations to identified papers ran from May 2020 to March 2024. In total, 168 papers progressed to full-text appraisal; 62 were identified for inclusion in the update. Then we combined papers from our original and updated reviews.
Results
In the combined review, the literature is dominated by papers from the United States (50/95) and mostly drawn from the Global North. Papers cover various clinical fields, with more on cancer than any other field; 86 of the 95 papers report positive results, of which 70 are purely positive and 16 positive/mixed, meaning there are some negative elements (i.e. aspects where there is a lack of healthcare improvement) in their findings.
Conclusions
The updated review collates a substantial pool of studies, especially when combined with our original review, which are largely positive in terms of the impact of research engagement on processes of care and patient outcomes. Of the potential engagement mechanisms, the review highlights the important role played by research networks. The review also identifies various papers which consider how far there is a “dose effect” from differing amounts of research engagement. Additional lessons come from analyses of equity issues and negative papers. This review provides further evidence of contributions played by systems level research investments such as research networks on processes of care and patient outcomes.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12961-024-01187-7.
Keywords: Clinical trials, Clinicians, Health equity, Healthcare organizations, Networks, Oncology, Patient outcomes, Processes of care, Research engagement, Systematic review
Background
There is an often-held assumption that the engagement of clinicians and healthcare organizations in research improves healthcare performance at various levels. This assumption contributed to policy documents from various health organizations promoting research engagement by healthcare providers as a way of improving healthcare, for example, in the United Kingdom [1–3]. Therefore, it was believed that policy-makers who make relevant decisions, such as on the allocation of resources for health and health research systems, should have access to evidence on the validity of the assumption. In the United Kingdom, two programmes of the National Institute for Health Research (now called the National Institute for Health and Care Research) (NIHR) decided to commission reviews of the global evidence on this [1–3].
The wide-ranging brief provided for the second review, which was the original review by the authors of this present paper (published in full as Hanney et al. in 2013 [3] and more succinctly as Boaz et al. in 2015 [2]), included the additional aim of conducting a theoretically grounded synthesis to explore the mechanisms by which research engagement might improve healthcare [3]. The protocol for that study considered pertinent global literature, including on accelerating the adoption of evidence in health systems, and ways to enhance the relevance of the research conducted to the needs of health systems. The final protocol published as part of the Hanney et al. report [3] then used these ideas to identify possible mechanisms that would be worth analysing to help understand the processes that might be at work when research engagement leads to improved health. Among these was the idea that engaging in conducting research increases the ability and willingness of clinicians to use research findings from the global pool of knowledge, and here the concept of “absorptive capacity” was expected to be useful [3].
Some analyses focussed on the importance of exploring the relationship between research engagement and improved healthcare to contribute towards understanding of the benefits for healthcare performance in the context of a strong research culture. These papers were reviewed in Australia by Harding et al. in 2017 [4].
As far as we are aware, these three systematic literature reviews published in the second decade of this century provided the first analyses of the empirical evidence available to support the assumption of improved healthcare from research engagement [1–4]. Their differing scopes and approaches are summarized briefly in Table 1.
Table 1.
Summary of the three systematic reviews conducted prior to 2020 on whether research engagement by healthcare organizations and staff improves healthcare
| Review | Objective | Scope of the formal review | Findings |
|---|---|---|---|
|
Clarke and Loudon [1] United Kingdom A Cochrane methodology review |
To examine the influence of “trial active” practitioners or institutions on patient outcomes and/or care processes |
Focus limited to RCTs 15,000 studies identified 13 papers included in the review |
Care processes (e.g. guideline adherence) might be better when service providers take part in trials, but the magnitude of this effect and the consequences for patient health are uncertain |
|
Hanney et al. [3] United Kingdom NIHR funded |
To provide a theoretically and empirically grounded assessment of the relationships, if any, between research engagement and performance |
Focus is on empirical research studies (not limited to clinical trials) 10 495 studies identified 33 papers included in the review |
Majority of studies (28/33) identified a positive association between research engagement and improvements in healthcare Most improvements were in processes of healthcare; only a quarter were in health outcomes |
|
Harding et al. [4] Australia |
To examine the potential benefits of a “strong research culture” on health service performance |
Focus is on fewer databases than Clarke or Hanney and search targeted different organizational metrics (e.g. staff satisfaction and retention) 3015 studies identified 8 papers included in the review |
The review found an association between health service performance and activities representing organizational investment in research culture – in which the authors claimed being “research active” was a component |
All three of the reviews reported some evidence of a positive association between research engagement and healthcare performance, but the available evidence was not mature enough to support statements about causality [2]. Our review [2, 3] had the widest scope of the three, reflecting the broad brief given by our NIHR funder. It included an extensive initial mapping exercise, a formal focussed review, and a wider review which drew on the earlier stages to explore, as noted above, the mechanisms by which research engagement might improve healthcare [3]. Our review identified 33 papers from 9 countries (15 from the United States), 28 of which reported positive findings [5–37]. Even our review concluded, however, that there did not appear to be a well-structured, steadily accumulating body of knowledge about the benefits associated with research engagement.
In the succeeding years, we have identified a continuing and growing interest in this general topic, therefore an updated review seemed desirable to gather more evidence about how far research engagement might lead to improved healthcare and the mechanisms involved. In addition to these general questions, our original review had identified two specific issues that could usefully be considered further. These were research networks as potentially important mechanisms through which research engagement might improve healthcare, and whether greater amounts of research engagement would have a larger beneficial effect. Our original review also covered some aspects of a third issue (health equity) that has subsequently become increasingly important [2, 3].
The growing development of research networks has been associated with efforts to move towards more formalized attempts to boost the role of health research systems in accelerating science and facilitating the translation of research into practice [2, 3]. However, at the time of our original review, the evidence was still emerging and its availability was heavily skewed by the different timing of the establishment of formal research networks in different countries.
In the United States, various research networks had been set up in the second half of the last century, and most of the early papers on networks and their role came from there [2]. These networks in the United States are described in the “Glossary of the United States of America and United Kingdom Research Organizations and Networks Discussed in the Papers” (see Additional file 1). They include the National Cancer Institute (NCI)-funded Cancer Community Oncology Program (CCOP), established to encourage outreach and improve equity by bringing the advantages of clinical research to cancer patients in their own communities [37]. In the United Kingdom in contrast, national research networks were not formally created until this century, too late for any potential benefits to patient outcomes to be fully researched and reported prior to our review which started in 2011. However, even in our original review we were aware of concurrent United Kingdom work to measure those outcomes and to improve patient access to clinical research, and identified a need for further evaluations (see Hanney et al. [3], pp. 48, 83).
Subsequently, we also became increasingly aware of new studies on the effects of the developing research networks, especially in the United Kingdom, and Boaz et al. identified a promising approach in statistical analysis that could help further analysis [2]. As set out in the Glossary, there have been policy shifts and organizational changes in the United States and the United Kingdom, and there have been further ones elsewhere, which are designed to promote research networks to address the time lag between the production of research and its use in practice, including various efforts to strengthen links between academic centres and community services. There has also been an increasing emphasis, including within research networks, on the potential research contribution of healthcare professionals other than medical professionals.
Our original review had also noted a partly related second issue as worthy of further attention. This is the question of whether the association between research engagement by healthcare providers and improved healthcare outcomes increases with greater amounts of research participation. There was early evidence that it did. This came, in particular, from the 2008 paper by Majumdar et al. [26] that compared outcomes for patients with angina in hospitals in the United States having a high level of angina research activity with hospitals with low research activity, and those with no research activity. Other papers compared centres with different levels of research activity within a research network [23]. However, there was little certainty about extent and implications around this issue at that time, although it has become increasingly important with the development of the comprehensive research networks that we summarize in the Glossary. It also has theoretical implications for the exact nature of the association between research engagement and improved healthcare: in our original review we argued that further data on this effect, and on the time an institution was research active, “are needed to provide evidence of causation” (p. 12) [2].
These findings also have implications for health equity, the third unresolved issue. More outreach by research networks means more access to clinical research and its benefits for more patients. The United States CCOP has been rightly lauded for achieving this [37], but can that be squared with the emerging finding that higher levels of research participation in specific provider institutions bring greater benefit to the patients in those centres?
Reflection on these uncertainties further strengthened the argument that with all the developments since our original review, it seemed timely in 2020 to revisit this topic to explore and collate what additional understanding had been gained. While conducting the resulting update, we became aware of some more recent developments. A United Kingdom qualitative systematic review was published in 2021 that explored the impact of research activity by healthcare professionals other than medical professionals [38], and another UK review published in 2023 focussed on research engagement by allied health professionals (AHPs) [39]. With few exceptions, the papers specifically on nursing and AHPs in these reviews were typically smaller scale than the papers included in our formal review, and/or usually did not include the quantifiable comparisons that featured in most of our included papers. Nevertheless, these reviews usefully illustrate the growing interest in the contribution of these healthcare professionals in countries such as Australia, Canada and the United Kingdom.
In addition, we identified a large-scale study from the United States by Shahian et al. [40] that was published in 2022 and examined the link between research engagement and improved healthcare performance in 5 major medical fields across 1604 Medicare-participating hospitals. A noticeable facet of the paper by Shahian et al. was their referencing of a large number of papers that we had identified either in our original review, or in the first phase of our updated review [40].
To ensure our updated review adequately reflected all such developments since May 2020, we conducted a further search in March 2024. The review presented here is based on papers identified in both phases of the updated review, the findings of which are then combined with those from our original review.
Review question
To identify studies, the primary research question used the same approach as Boaz et al. [2, 3].
Does research engagement (by clinicians and organizations) improve healthcare performance?
By research engagement, we mean, as in our original review, engagement in research rather than the broader concept of engagement with research, and we are referring to participation in research by healthcare organizations and staff rather than patient participation in trials. Engagement in research is taken to mean, “a deliberate set of intellectual and practical activities undertaken by healthcare staff (including conducting research and playing an active role in the whole research cycle) and organizations (including playing an active role in research networks, partnerships or collaborations)” (p. 2) [2].
Methods
Design
The 2020 decision to complete an update of the previous review [2, 3] was informed by a published decision framework for updating systematic reviews [41]. After completion in 2024 of the comprehensive initial phase of the updated review, including the two searches and considerable subsequent analysis, we recognized, as noted above, that while we had been conducting the review some important further papers had been published. We wanted to incorporate such papers, and so decided to conduct a further search for papers. The design of this final phase (which included a third search) was informed both by the fact that we had already identified a considerable number of papers for the updated review, and by the way new papers in this field were by now much more likely to cite earlier papers, with Shahian et al. [40] being a prime example. Therefore, we thought it was reasonable to rely to a much greater extent on checking citations to the papers already identified, as explained below.
Search strategy and information sources
Search 1 (update)
The first step in syntax development used the Medline Ovid strategy published by Boaz et al. [2].
Initial diagnostic testing indicated issues preventing code execution. Due to the syntax comprising several nested terms and Boolean operators, it was rebuilt using recommendations for “single-line” optimization for debugging complex code [42].
Search 2 (modified)
The syntax for Search 2 was a term modification to capture papers that more explicitly indexed research networks and collaborations. Search 2 necessitated a deeper dive into the full-text content of papers. The decision to search full-text articles reflected observations that the sensitivity of Search 1 was potentially affected by the variable quality (and relevance for our review) of abstracts, a consistent challenge for reviewers [43]. As a second search also adapted published syntax, the Preferred Reporting Items for Systematic reviews and Meta-Analyses Literature Search—Extension Checklist (PRISMA-S) reporting protocol was followed [44]. (The full text for search strategies is provided in “Search Strategy and Syntax Sensitivity”; see Additional file 2).
Electronic databases
Nine electronic records collections were used in Search 1: Medline (OVID and EBSCO), EMBASE, PsycInfo (OVID and EBSCO), CINAHL, Web of Science, Health Management and Information Consortium and British Nursing Institute. The mix provided parity with previous reviews and mitigated risk of missed papers by combining general and specialized databases. Different interfaces (e.g. OVID, EBSCO) for the same collection were also included to offset variations due to platform [45]. Grey literature was not searched: these collections failed to uniquely identify papers in previous reviews on this topic. Search 2 was restricted to the Medline EBSCO Full Text records, which was the collection which yielded the highest hit ratio for relevant papers (see Additional file 2).
Other sources
Manual and snowball searching were used in three ways. Firstly, a range of search engines (Google Scholar, PubMed, ProQuest Central, Scopus, the Web of Science Cited Reference Search) were used to track citations for (a) prior reviews as whole papers, (b) the individual studies within these reviews and (c) article reference lists. Secondly, key journals that published studies shortlisted in the previous reviews were hand-checked, including: Implementation Science, PLOS One, BMJ Open and BMC Health Services Research. Thirdly, topic experts suggested papers for consideration.
Search 3 (final phase)
As explained above, we subsequently conducted a further search covering May 2020–March 2024. This consisted of: a hand-search of three of the journals in which papers from the first phase of the updated review had been published (Health Research Policy and Systems, Implementation Science and Medical Care); a check of papers in the two reviews published in this period [38, 39]; and a check of citations in this period to all the papers identified both in our original review and in the update’s initial phase.
Eligibility criteria
The following limiters were applied:
Timeframe: 1 January 2012 to 20 March 2024 (inclusive of eprint)
Population: Human (any setting)
Language: English (any country)
Paper type: Academic Journals (scholarly works). Conference papers were admitted as flags for accessible peer-reviewed works (e.g. pre-print) or key teams.
Three criteria were defined, guided by definitions from the original review [3].
Criterion A: study design
Empirical studies using method/s aligned with health services research, including clinical trials, retrospective cohort and survey methods. Studies with only patient reported outcomes (e.g. satisfaction) were excluded.
Criterion B: healthcare performance
Studies must report an outcome indexing performance assessment for a care process or healthcare improvement. The following were excluded: staff-specific reports alone, (e.g. job satisfaction or morale), policy impacts alone (no flow through to healthcare), descriptions of networks without outcomes data.
Criterion C: research engagement
Explicit demonstration of engagement in research including: agenda-setting, conducting research, participation in action research or in networks where the research involvement is noted. This criterion also allowed engagement implicitly through research network membership, even if a specific study was not recorded, but there was a comparison of healthcare between member and non-member settings. More details about examples that were in scope can be found in Hanney et al. [3, p. 2].
Records management
To efficiently manage the export of the large records for the first two searches, Endnote X9 (Clarivate) was used to combine downloads from different databases and discard software detected duplicates. The endnote library was imported into Rayyan, a free multi-collaborator online screening tool [46]. Study selection procedures for Searches 1 and 2 followed the same screening/eligibility check sequence.
Screening and eligibility/quality checking
In Rayyan, titles were scanned to exclude papers that were irrelevant, did not meet criteria or were non-exact duplicates. Abstracts of retained records were then screened and classified as “include”, “exclude” or “maybe”. A third screening of “maybe” classifications forced a binary coding of “include” or “exclude”, with comment flags on issues. A final records’ sweep with the Rayyan query function checked for misclassified studies. This four-step screening process was completed by a single reviewer (BG).
Full-text for each provisionally included study was uploaded into Rayyan. The initial eligibility check was completed by three experts who were involved in article screening for Hanney et al. [3]. As a criterion check and to orient reviewers to the Rayyan platform, a practice phase used 10 randomly sampled records. The abstract was the primary source for expert reviewers, with full-text also available. After the practice task and consensus discussion of criteria, a batch of records (alphabet determined) was assigned to each expert reviewer, to rate each paper as “include”, “exclude” or “maybe” (ratings were unblinded). If the rating pair (i.e. B.G. and an expert from the original review) were both “include”, the paper was progressed to full-text appraisal. If there was disagreement, papers rated as “maybe” were reassigned to another expert reviewer for an opinion, and those rated as “exclude” by an expert reviewer were marked for discard. If consensus for a “maybe” paper could not be reached by discussion, it was progressed to a full text appraisal, conducted by a single reviewer (B.G.) using all available information sources and reviewer ratings.
A final review of all potential “includes” was jointly conducted by team members, including a few papers identified by other sources such as continued manual snowballing from key papers. The study selection procedures for Search 3 mirrored this final step, and so consisted of a review of all potential “includes” conducted jointly by team members.
Study quality was assessed using the mixed-methods appraisal tool (MMAT v2018), on a scale of 1 (low) to 5 (high) [47]. The MMAT accommodated all designs in the paper set. The majority of the papers have a design which fitted into the MMAT category of quantitative non-randomized. All papers scored good to high quality on the five questions in their relevant MMAT subscale. The lower end of ratings (good) was typically due to lack of information in the article, such as whether and/or how confounding factors may have been identified or managed. Quality ratings were not used to exclude papers, but formed part of the discussion about the quality and contribution of the papers.
Data extraction, coding and ethics
As Rayyan is only a screening platform, a data extraction sheet was created in Excel (v2016) for each included paper. A university research ethics committee deemed the project as not requiring formal ethical approval, due to secondary data mining on anonymized aggregated records.
Analysis
A large and methodologically diverse mix of papers was identified with a range of different outcomes and outcome measures. The papers were combined through a process of critical interpretive synthesis inspired, as in our original review, by the approach outlined by Dixon-Woods et al. [48]. This involves adopting an iterative approach to refining the research question, searching the literature and defining and applying codes and categories. It enables the generation and development of theory with strong explanatory power and uses relevance as one measure of quality. Following analysis of the papers in the updated review, we collated the results from the updated review with those from our original review to create one combined set of papers for overall analysis.
Results
Figure 1 summarizes the review literature flow. The two formal searches identified 66 901 records, with 68 further papers coming from other sources, including the March 2024 extension. From these, 168 papers progressed to full-text appraisal, and 62 were identified for inclusion [40, 49–109].
Fig. 1.
Flow diagram for literature search
This review updates the previous review conducted by the team [2]. Table 2 outlines the 95 papers in our combined review: the 62 additional papers in the updated review along with the 33 papers in our original review. The latter 33 papers are shown in italics in Table 2, which includes details about the study characteristics of all 95 included papers as well as key dimensions of the findings. To complement Table 2, brief notes on the development and scope of key United States and United Kingdom research networks/organizations discussed in the papers are provided in the “Glossary of the United States of America and United Kingdom Research Organizations and Networks Discussed in the Papers” (see Additional file 1).
Table 2.
Combined review’s 95 papers on whether research engagement by healthcare organizations and staff improves healthcare
| 1st author, year | Location of study | Clinical area | Sample: including comparators, such as network membership, or not. (Data type: including, comparators if not apparent from sample, that is, if bibliometric and/or sometimes if “dose” paper) | Level of analysis: organization or clinician | Findings: positive or negative | Improvement (scope of impact) |
|---|---|---|---|---|---|---|
| Abraham 2010 [5] | United States | Substance use disorder (SUD): alcohol disorders’ medication | 127 programmes in the Clinical Trials Network (CTN) of the National Institute on Drug Abuse (NIDA) compared with 244 not in the network. (Data: repeat interviews, including on treatments used.) | Organization | Positive | Processes (broad) |
| Abraham 2014 [49] | United States | SUD: 35 treatment services | 78 programmes in NIDA’s CTN that participated in at least one protocol compared with 78 CTN programmes that had not participated in any. (Data: interviews, including on treatments used.) | Organization | Positive | Processes (broad) |
| Alder 1978 [6] | United Kingdom | Inguinal hernia/varicose veins | A health district that hosted a randomized controlled trial (RCT) on early discharge of inguinal hernia and varicose veins patients compared with two adjacent districts that had not. (Data: retrospective, length of stay [LoS].) | Organization | Positive | Processes (specific) |
| Alotaibi 2017 [50] | Canada led; international | Intracranial aneurysm neurosurgery | 3307 patients of 147 surgeons participating in trials: 42% from the United States; 12% Canada (total 21 countries including in Africa and Central America). (Data: bibliometric [publication “quality”, as measured by H-index, not quantity] compared with retrospective analysis of trial outcomes data.) | Clinician | Positive-mixed | Outcomes (broad) |
| Andersen 2006 [7] | Denmark | Asthma: medication in general practice | 10 general practices conducting a trial on asthma medication compared to 165 non-trial practices. (Data: observational cohort study on medication used.) | Organization | Negative | N/A |
| Bennett 2012 [51] | United Kingdom | Hospital care | 147 National Health Service (NHS) hospital Trusts. (Data: bibliometric [citations/admission] compared to hospital mortality data.) | Organization | Positive | Outcomes (broad) |
| Birkmeyer 2004 [52] | United States | Cancer: surgery in various cancer sites | 27 021 patients at 51 National Cancer Institute (NCI) cancer centres compared with 36 839 patients at 51 high-volume non-NCI centres. (Data: retrospective – Medicare databases for surgical mortality and survival.) | Organization | Positive-mixed | Outcomes (broad) |
| Brown 1996 [53] | United States | Hospital care | 81 Department Veterans Affairs (VA) hospitals. (Data: retrospective – databases and VA reports on research programme size compared with LoS, costs.) | Organization | Positive | Processes (broad) |
| Carpenter 2011 [8] | United States | Cancer: breast | 3874 patients: those treated at organizations affiliated with a cancer research network, such as NCI Cooperative Groups, compared with those treated at non-affiliated organizations. (Data: retrospective, Surveillance, Epidemiology and End Results [SEER]-Medicare database – treatments, surgical volume, teaching affiliation.) | Organization | Positive | Processes (broad) |
| Carpenter 2012 [54] | United States | Cancer: colon | 1029 patients treated at NCI’s Community Clinical Oncology Program (CCOP) practices compared with 3025 at treated at non-CCOP practices. (Data: retrospective, SEER-Medicare database, for 2003–2006 data on four treatments options, see also Penn [87], who analysed same data but for 2003–2005 for just two treatment options.) | Organization | Positive | Processes (broad) |
| Chaney 2011 [9] | United States | Depression: collaborative care (in primary care) | 288 patients in 7 Veterans Affairs (VA) primary care sites that implemented a collaborative care intervention compared with 258 patients in 3 VA non-intervention primary care sites. (Data: interviews, including on evidence-based care and symptom improvement.) | Organization | Positive-mixed | Processes (specific) |
| Chen 2006 [10] | United States | Cancer: laryngeal | 31 044 patients at teaching/research facilities (including most NCI Comprehensive Cancer Programs) compared with 29 854 at community cancer centres and 13 447 at community hospitals. (Data: retrospective – National Cancer Database on use of chemoradiation.) | Organization | Positive | Processes (broad) |
| Clark 2003 [11] | Canada | Apheresis for various conditions | 22 physicians and 19 major medical centres in Canada – some involved in 3 trials of the experimental apheresis therapy, some not. (Data: retrospective, registry data on adoption of the intervention prior to and during the trials—including outside trials; repeat surveys of physicians.) | Clinician | Negative-mixed | N/A |
| Corrigan 2005 [55] | United States | New drugs in various clinical fields | 2108 physicians involved with specific phase-3 trials compared with same number not involved. (Data: retrospective, databases—matched case control on prescribing.) | Clinician | Positive | Processes (specific) |
| Das 2008 [12] | United Kingdom | Barratt’s oesophagus | 228 specialists at 107 sites (57 already in a trial): comparisons over time and between specialists at sites already enrolled and those waiting to enter. (Data: repeat surveys of specialists, including on guideline adherence.) | Clinician | Positive | Processes (specific) |
| de Arriba-Enriquez 2021 [56] | Spain | Psychiatry | The top 50 hospitals with psychiatric wards. (Data: bibliometric [range of quantitative and quality indicators per number of beds] compared with system records on quality-of-care outcomes such as readmissions.) | Organization | Positive | Outcomes (broad) |
| de Lange 2015 [57] | The Netherlands | Preterm labour: tocolysis | 6489 patients treated in 8 hospitals from 2006 to 11 – comparisons between time before, during and after some patients entered into a trial. (Data: retrospective, LoS data—Obstetric High Care registry; costs.) | Organization | Positive | Processes (specific) |
| Downing 2017 [58] | United Kingdom | Cancer: colorectal (CRC) | 209 968 patients, 150 NHS Trusts. (Data: retrospective on extent and length of participation in National Cancer Research Network CRC randomized controlled trials [RCTs] compared with outcomes, patient volume and other data.) | Organization | Positive | Outcomes (broad) |
| Du Bois 2005 [13] | Germany | Cancer: ovarian | 275 patients in 80 trial study group hospitals compared with 201 patients in 85 non-trial study group hospitals (see also Rochon [34, 94]). (Data: surveys; survival data collected up to 2 years after diagnosis, hospital patient volume data also used in multivariable analysis.) | Organization | Positive | Outcomes (broad) |
| Ducharme 2007 [14] | United States | SUD: Buprenorphine and behavioural treatments | 1006 centres’ treatment compared: some were in the NIDA’s CTN and participated in the trials; some in NIDA’s CTN and did not participate in these trials; some not in NIDA’s CTN. (For contrast—see Rieckmann [91]). (Data: panel longitudinal design, repeat interviews on treatments.) | Organization | Positive-mixed | Processes (specific) |
| Eaton 2016 [59] | United States | Cancer: lung | 495 patients in a trial by NCI’s Radiation Therapy Oncology Group (RTOG) (which became part of a wider NCI Cooperation Group): 300 in high accrual volume centres compared with 195 in low accrual centres. (Data: secondary analysis of trial data – including overall survival.) | Organization | Positive | Outcomes (broad) |
| Fanaroff 2019 [60] | United States | Cardiovascular: myocardial infarction (MI) | 453 821 patients in 430 hospitals enrolling at least 1 patient in a clinical trial compared with 138 923 patients in 336 non-trial hospitals. (Data: retrospective – registry data on outcomes, procedures, patient and hospital characteristics.) | Organization | Positive | Outcomes (broad) |
| Farquhar 2021 [61] | United States | Cancer: head and neck | 463 patients who had been in a 2002–2006 population case–control study; 39 centres: 5 NCI/VA designated academic centres and 9 community cancer centres compared with 25 non-designated community hospitals. (Data: retrospective, on volume; deaths from National Death Index.) | Organization | Positive-mixed | Outcomes (broad) |
| Fernández-Domínguez 2022 [62] | Spain | Multiple fields: physiotherapists | 419 practicing physiotherapists across Spain – survey responses on their career and role, including time spent on research, compared with their responses on evidence, including its use in practice. (Data: survey – the Health-Sciences Evidence-Based Practice questionnaire.) | Clinician | Positive | Processes (broad) |
| Fields 2016 [63] | United States | SUD: Network for the Improvement of Addiction Treatment | 458 SUD treatment centres compared: 153 NIDA CTN programmes formed one group to compare with others. (Data: retrospective—used interview data from two earlier studies including on innovations used.) | Organization | Positive | Processes (specific) |
| Fong 2020 [64] | United States | Cancer: pancreatic | 989 patients treated at NCI-designated centres compared with 2070 at Commission on Cancer (CoC)-accredited centres and 2059 at non-accredited centres. (Data: retrospective – SEER-Medicare database on treatment and survival.) | Organization | Positive | Outcomes (broad) |
| García-Romero 2017 [65] | Spain | Hospital care | 189 public hospitals – both medical and surgical specialities. (Data: bibliometric [quantity and quality] compared with retrospective hospital data including LoS through detailed statistical/econometric analysis.) | Organization | Positive | Processes (broad) |
| Gilbert 2013 [66] | United States led; international | Dental: caries | 565 practitioners in an international dental practice-based research network (PBRN) – United States led with three Scandinavian countries. (Data: repeat surveys to assess impact of level of research network participation, full or partial, on increasing evidence-based treatment.) | Clinician | Positive | Processes (broad) |
| Goldberg 2002 [15] | United States | Diabetes: primary care management | 185 patients at one family medical centre divided into 2 parallel groups for an intervention study to improve routine care. (Data: computerized database, including processes and % of patients inadequately controlled.) | Organization | Negative-mixed | N/A |
| Hall 2010 [16] | United States | Veteran rehabilitation: family care | Four sites of the VA’s Family Care Collaborative’s intervention – part of its Quality Enhancement Research Initiative; 226 rehabilitation team members – interdisciplinary. (Data: cross-site, mixed-method evaluation; surveys; practice change – before/after care scores at sites.) | Organization | Positive | Processes (specific) |
| Hébert-Croteau 1999 [17] | Canada | Cancer: breast | 682 patients admitted to hospitals involved in multicentre clinical trials compared to 577 patients admitted to non-trial hospitals. (Data: retrospective – medical charts, oncology registries: guideline adherence.) | Organization | Positive | Processes (broad) |
| Janni 2005 [18] | Germany | Cancer: breast | 198 medical centres in a very large trial which provided support such as a newsletter and study meetings to a network of participating trial centres. (Data: survey of care provided by centres before and after trial.) | Organization | Positive | Processes (specific) |
| Jha 1996 [19] | Canada | Cardiovascular: acute myocardial infarction (AMI) | 32 468 patients in two trials: comparisons between trial patients, non-trial patients at trial hospitals, and external hospitals. (Data: retrospective datasets on complicated pattern of treatments and survival.) | Organization | Positive-mixed | Outcomes (specific) |
| Jones 2000 [20] | United States | Appendectomy | 200 patients treated by 25 surgeons in a hospital where a trial had previously been conducted. (Data: retrospective on type of surgery conducted – review of patient records, survey.) | Clinician | Negative | N/A |
| Jonker 2015 [67] | United Kingdom | Hospital care | 155 English NHS Hospital Trusts. (Data: retrospective, 1 year’s data [2012–2013] of National Institute for Health Research (NIHR) research activity compared with hospital performance on the overall Care Quality Commission (CQC) risk rating.) | Organization | Negative | N/A |
| Jonker 2018 [68] | United Kingdom | Hospital care | 129 English NHS Hospital Trusts. (Data: retrospective, 5 years’ data [2012/13–2016/7] on extent of NIHR research activity per clinician compared with a hospital mortality index and overall CQC rating.) | Organization | Positive | Outcomes (broad) |
| Jonker 2021 [69] | United Kingdom | Hospital care | 129 English NHS Hospital Trusts. (Data: retrospective, 2018–19 data on extent of NIHR research activity per clinician and hospital geographical location, compared with hospital performance on mortality and CQC rating.) | Organization | Positive-mixed | Outcomes (broad) |
| Karjalainen 1989 [21] | Finland | Cancer: leukaemia | 1978 patients, 21 health districts: 17 in study area exposed to protocol, compared with 4 not exposed. (Data: retrospective – Finnish Cancer Registry for 5-year cumulative relative survival rates.) | Organization | Positive | Outcomes (specific) |
| Kirkby 2020 [70] | United Kingdom | Cardiac arrest: Emergency Medical Service providers | 498 paramedics participated in an online survey about the effect on them of being part of an RCT of interventions to manage out-of hospital-cardiac arrest. Before and after comparisons. (Data: online survey, and 19 respondent interviews, including on interventions used.) | Clinician | Positive-mixed | Processes (specific) |
| Kirwan 2014 [71] | United Kingdom | Cancer: breast | 153 patients in 5 previously high trial-recruiting hospitals compared with 43 patients in 5 low/non-recruiting hospitals in same NHS Trust. (Data: retrospective: consecutive computed tomography planning images.) | Organization | Positive | Processes (specific) |
| Kizer 1999 [22] | United States | Cardiovascular: AMI medications | 8386 patients received into 7 trials (1978–1995) compared in terms of use of medications at enrolment (therefore prescribed by physicians in routine practice) and at discharge (prescribed by physicians in RCTs). (Data: retrospective – case reports on medications prescribed.) | Clinician | Positive | Processes (broad) |
| Knudsen 2009 [23] | United States | SUD: buprenorphine | 206 treatment centres of the NIDA’s CTN compared for buprenorphine adoption: if participated in drug’s trials or not. (Data: repeat interviews with administrators, 24 months apart, including on treatments used.) | Organization | Positive | Processes (specific) |
| Laliberte 2005 [24] | United States | Cancer: breast | 16 600 patients in 423 facilities: comparing 77 facilities in two or more NCI networks, with 67 in one NCI network, and with 279 in no network. (Data: retrospective, SEER-Medicare database – guideline concordance.) | Organization | Positive | Processes (broad) |
| Levan 2014 [72] | United States | Neonate: intubation | 3849 infants, in 11 centres in a trial by National Institute of Child Health and Human Development Neonatal Research Network: the three centres that had been in a feasibility study were compared with the eight that had not. (Data: retrospective, Institute’s registry – pre-/post-trial intubation data.) | Organization | Positive-mixed | Processes (specific) |
| Lin 2018 [73] | United Kingdom | Cancer: head and neck | 465 patients in 60 hospitals who had been in an NIHR Clinical Research Network (CTN) RCT: compared those treated in hospitals recruiting low numbers to cancer trials with those treated in higher recruiting hospitals. (Data: retrospective – trial recruitment, hospital throughput, survival.) | Organization | Positive | Outcomes (broad) |
| Litjens 2013 [74] | The Netherlands | Obstetrics/gynaecology | 83 sites, 202 staff (gynaecologists, residents, nurses, midwives) who had variously participated in up to nine trials. (Data: survey of knowledge and implementation of trial findings in each site.) | Organization | Positive-mixed | Processes (specific) |
| Ljunggren 2022 [75] | Sweden | Cancer: colorectal | 3168 metastatic patients with CRC treated at university hospitals (i.e. research-active) compared with 6800 treated at non-university hospitals. (Data: retrospective – Swedish CRC database and registers data on hospital volume and type, treatment [metastatic surgery] and survival.) | Organization | Positive | Outcomes (broad) |
| Majumdar 2002 [25] | Canada led; North America | Cardiovascular: AMI medication | 25 886 patients in 659 hospitals in a trial: the 1415 patients in the 22 of those hospitals that were also in an earlier trial compared to the rest. (Data: retrospective – trial data on treatments given.) | Organization | Negative | N/A |
| Majumdar 2008 [26] | United States (led from Canada) | Cardiovascular: unstable angina | 174 062 patients in 494 hospitals; comparisons between: 123 high trial enrolment hospitals, 226 low trial enrolment, 145 no trial enrolment. (Data: retrospective – data from an observational data collection and improvement initiative on processes, in-hospital mortality, size.) | Organization | Positive | Outcomes (broad) |
| Manes 2019 [76] | United States (led from Israel) | Hospital care: 3 fields | 50 United States university hospitals: same three departments/wards in each. (Data: bibliometric [see Tchetchik [100]] compared with range of retrospective data on survival, health performance, size, staffing levels, costs, LoS.) | Organization | Positive | Outcomes (broad) |
| Marmor 2015 [77] | United States | Hematopoietic cell transplantation | 12 993 transplants conducted in 162 centres: compared 24 non-certified centres; 106 certified by a Foundation; and 32 certified by the Foundation and the Blood and Marrow Transplant clinical trials network. (Data: retrospective, from relevant databases: processes and outcomes.) | Organization | Positive | Outcomes (broad) |
| McBride 2013 [78] | United States | Dental treatments | 950 dental practitioners, most members of one of three dental PBRNs: 378 full network participants compared with 346 partial network participants and 226 inactive or non-members. (Data: survey of practice patterns.) | Clinician | Positive | Processes (broad) |
| McCarthy 2015 [79] | United States | Cardiovascular: aortic valve | 5009 patients who underwent transcatheter aortic valve replacement at 68 hospitals that had participated in clinical trials compared with 1392 patients treated at 140 non-trial hospitals. (Data: retrospective, Medicare data including on mortality.) | Organization | Negative | N/A |
| McDaniels-Davidson 2022 [80] | United States | Cancer: cervical | 1395 women in California treated at NCI designated cancer centres compared with 2855 treated other locations in California. (Data: retrospective, California Cancer Registry data – death, guideline-concordant treatment.) | Organization | Positive | Outcomes (broad) |
| Meineche-Schmidt 2006 [27] | Denmark | Gastro-oesophageal reflux: primary care treatment | 1206 patients enrolled by 122 General practitioners (GPs) (some in earlier trial, some not) in study on the impact of participation in first trial. (Data: prospective – observational study including of treatments given.) | Clinician | Positive | Processes (specific) |
| Merkow 2014 [81] | United States | Cancer care | 56 NCI-designated cancer centres compared with 1112 CoC-accredited centres and 2395 non-accredited hospitals. (Data: retrospective, Medicare and American Hospital Association – range of process, outcome and cost measures.) | Organization | Negative | N/A |
| Meyer 2013 [82] | United States | Cancer: breast surgery | 874 patients of CCOP physicians compared with 16 303 patients of non-CCOP physicians. (Data: retrospective: SEER-Medicare database on level of sentinel lymph node biopsy.) | Clinician | Positive | Processes (broad) |
| Mold 2014 [83] | United States | Renal: chronic disease | 711 patients in 31 primary care practices in 4 primary care PBRNs in first wave; 1179 patients in 58 practices in the PBRNs in second wave. (Data: prospective intervention study; before/after measures of guideline use.) | Organization | Positive | Processes (specific) |
| Morton 2006 [28] | Australia | Sexual health | 100 patients of 21 clinicians: high recruiters into a relevant trial compared with low recruiters. (Data: retrospective – case notes of clinicians’ practice.) | Clinician | Positive-mixed | Processes (specific) |
| Murimwa 2023 [84] | United States | Cancer: pancreatic | 3300 patients at four NCI-designated cancer centres in Texas compared with 8969 patients at CoC centres, 1087 at safety net hospitals and 3715 no-designation hospitals; high/low volume hospitals also compared. (Data: retrospective, Texas cancer registry – guideline concordance; survival.) | Organization | Positive | Outcomes (broad) |
| Ozdemir 2015 [85] | United Kingdom | Acute care | 2 349 160 patients, 140 NHS Trusts: 35 low mortality Trusts, 63 expected level of mortality Trusts, 42 high mortality Trusts. (Data: retrospective, NIHR [Comprehensive] CRN funding/bed compared with outcomes, size, staffing, teaching, etc.) | Organization | Positive | Outcomes (broad) |
| Pancorbo-Hidalgo 2007 [29] | Spain | Pressure ulcer nursing care | 713 nurses responded to the research activity part of a wider survey: the 138 research active nurses compared with the 575 who were not. (Data: survey, including of level of guideline implementation.) | Clinician | Positive | Processes (broad) |
| Paulson 2008 [86] | United States | Cancer: colorectal surgery | 1299 patients treated at NCI designated cancer centres compared with 41 262 patients at non-NCI designated hospitals – some high volume. (Data: retrospective: SEER-Medicare database for outcomes, volume.) | Organization | Positive | Outcomes (broad) |
| Penn 2015 [87] | United States | Cancer: colon | 875 CCOP patients, 2096 non-CCOP; four groups compared: African Americans or Caucasian Americans; treated by CCOP or non-CCOP. (Data: retrospective, SEER-Medicare database, for 2003–2005 data on the use of two drugs, rather than the 2003–2006 use of the four treatment options in Carpenter, 2012 [54].) | Organization | Positive-mixed | Processes (broad) |
| Pons 2010 [30] | Spain | Cardiovascular: AMI; congestive heart failure | 50 acute hospitals voluntarily participating in a quality initiative and with more than 5 citable heart disease papers 1996–2004. (Data: bibliometric [citation measures] compared with retrospective hospital morality data.) | Organization | Positive | Outcomes (broad) |
| Prendergast 2018 [88] | South Africa-led; international | Rheumatic heart disease | 22 project sites across Africa (including in South Africa, Egypt, Ethiopia, Kenya, Mozambique, Nigeria, Rwanda, Sudan, Uganda, Zambia) and Yemen in a prospective, multi-centre project. (Data: survey and interview—trial participation impact on patient management.) | Organization | Positive | Processes (specific) |
| Puoane 2004 [31] | South Africa | Severe child malnourishment: participatory research, nurse-led in hospitals | 98 admissions in 2 rural hospital settings for a participatory “pre- and post-intervention descriptive study”. (Data: retrospective record review, including of fatality rates; structured observations; in-depth interviews; focus group discussions with nursing and medical staff.) | Organization | Positive-mixed | Outcomes (specific) |
| Rai 2017 [89] | United States | Cancer: lymphoma | 3490 patients of NCI cooperative group-affiliated institutions compared with 2222 patients of non-affiliated institutions. (Data: retrospective: SEER-Medicare database on treatment patterns.) | Organization | Positive | Processes (broad) |
| Rhyne 2011 [32] | United States | Diabetes: acanthosis nigricans | 145 clinicians who participated in two PBRN trials. (Data: practice change since trials: short-term post-trial survey and 16 interviews for one trial; long-term post-trial surveys for other trial.) | Clinician | Positive | Processes (specific) |
| Rich 2011 [33] | United Kingdom | Cancer: small-cell lung | 7845 patients in trusts in National Cancer Research Network: 2524 patients in trusts recruiting higher numbers into trials compared to 5321 patients in lower recruiting ones. (Data: retrospective, databases including research network; data on use of chemotherapy and survival.) | Organization | Positive-mixed | Processes (broad) |
| Rieckmann 2014 [90] | United States | SUD: buprenorphine | 922 counsellors from 172 NIDA CTN-affiliated programmes compared with 1203 counsellors from 251 non-CTN-affiliated programmes. (Data: survey of counsellors, interviews with administrators on the drug’s use.) | Clinician | Positive | Processes (broad) |
| Rieckmann 2016 [91] | United States | SUD: motivational interviewing | 156 CTN affiliated programmes compared with 345 non-affiliated programmes for a different type of behavioural treatment to one where Ducharme et al. [14] reported no benefit from CTN affiliation. (Data: interviews on the use of a behavioural treatment.) | Organization | Positive | Processes (broad) |
| Rindal 2014 [92] | United States | Dental: caries | 103 235 treatments between 2005 and 11 provided by 35 dentists in one multi-clinic dental group who had variable levels of engagement in the National Dental PBRN, or no involvement. (Data: natural experiment, retrospective – dental database for use of evidence-based treatments.) | Clinician | Positive | Processes (broad) |
| Robinson 2019 [93] | United Kingdom | Stroke: thrombolysis | 774 patients recruited in the United Kingdom to an international trial: 502 by 9 high-volume, well-resourced NIHR CRN Hyperacute Stroke Research Centres compared with 272 patients recruited to 24 other United Kingdom NIHR CRN sites. (Data: retrospective, trial endpoint on death or disability at 90 days.) | Organization | Positive-mixed | Outcomes (broad) |
| Rochon 2011 [34] | Germany | Cancer: ovarian | 275 patients in 80 cooperative trial study group hospitals compared with 201 patients in 85 non-trial study group hospitals. (Data: surveys; third year survival data collected, adding to data in Du Bois, 2005 [13].) | Organization | Positive | Outcomes (broad) |
| Rochon 2014 [94] | Germany | Cancer: ovarian | 219 patients with advanced cancer treated at trial study group hospitals compared with 133 treated at non-trial study group hospitals. (Data: mediation analysis of retrospective data on survival – this subset of the patient data in Rochon [34] focusses solely on advanced cancer.) | Organization | Positive | Outcomes (broad) |
| Salbach 2010 [35] | Canada | Stroke: physical therapist rehabilitation | 270 physical therapists returned a completed survey that could be used: 86 had participated in research compared with 182 who said they had not. (Data: cross-sectional mail survey, including on implementing evidence.) | Clinician | Positive | Processes (broad) |
| Salge 2009 [95] | United Kingdom | Acute care | 173 NHS Trusts. (Data: bibliometric [quantity of publications per staff member] and research and development grants/projects [collectively called science-based innovativeness] compared with retrospective data from a range of sources on mortality, size, etc. [collectively called organizational performance].) | Organization | Positive | Outcomes (broad) |
| Seaburg 2016 [96] | United States | Internal medicine | 308 Mayo Clinic resident physicians. (Data: bibliometric [publications’ quantity] compared with multi-source clinical performance evaluations.) | Clinician | Positive | Processes (broad) |
| Shahian 2022 [40] | United States | Acute care hospitals: multiple care areas | 1604 acute care hospitals, which was a random 40% of 4008 Medicare-participating hospitals; the hospitals were divided into three groups by total number of staff publications: > 46, 1–46 or 0. (Data: bibliometric [quantity and quality] compared with retrospective data including on mortality in five major fields, size, teaching, region.) | Organization | Positive | Outcomes (broad) |
| Shen 2022 [97] | China | Hospital care | 111 965 hospitalizations (random 20% from the Insurance scheme data) treated by 5794 physicians in a Chinese city’s 38 teaching hospitals. (Data: bibliometric [various quantitative and other measures of types of publications] compared with retrospective 30-day readmission data.) | Clinician | Positive-mixed | Outcomes (broad) |
| Siegel 2006 [36] | United States | Ear, nose and throat (ENT): otitis media | 17 practitioners from 11 practices who had been involved in a PBRN study compared with 18 practitioners who had not been involved. (Data: survey of prescribing in the year before and year after the trial.) | Clinician | Positive | Processes (specific) |
| Siracuse 2021 [98] | United States | Chronic limb threatening ischemia (CLI) | 9 231 909 CLI patients in the four US regions: participation rates in a relevant ongoing major trial compared with those regions’ total amputation rates. (Data: retrospective – Medicare on number of amputations and ongoing trial database on participation rates.) | Organization | Positive | Outcomes (broad) |
| Tan 2015 [99] | United States | Cancer: kidney | 1578 CCOP treated patients compared with 4316 non-CCOP treated patients (some non-CCOP patients were treated at other NCI centres). (Data: retrospective: SEER-Medicare database on surgical technologies.) | Organization | Negative-mixed | N/A |
| Tchetchik 2015 [100] | United States (led from Israel) | Hospital care: three fields | 50 university hospitals – same three departments in each, 4330 physicians. (Data: bibliometric measures [quality more than quantity] [as used in Manes [76]] compared with retrospective data: survival, care quality, etc.) | Organization | Positive | Outcomes (broad) |
| Trusson 2019 [101] | United Kingdom | Healthcare: nurses, midwives and Allied Health Professionals | 67 members of the NIHR’s East Midlands Clinical Academic Practitioner network identified how research engagement led to patient benefits etc. – implicitly compared with time before their research engagement. (Data: survey and 16 interviews – self-selected and snowball sampling.) | Clinician | Positive | Processes (specific) |
| Tsang 2015 [102] | United Kingdom | Cancer: breast radiotherapy | 26 sites of trial where National Radiotherapy Trials Quality Assurance Group assisted with adherence to components of trial protocol. (Data: survey of 26 sites to compare before/after, plus earlier Royal College of Radiologists audit of 1386 patients – in 13 trial sites and 37 non-trial.) | Organization | Positive | Processes (specific) |
| Tsang 2022 [103] | Canada | Pneumonia: probiotics prevention trial | 2203 patients in 34 academic/research hospitals in a trial compared to 352 patients in 7 community, not research, hospitals. (Data: pre-planned observational study nested in main trial – data on patient demographics, interventions, outcomes [mortality] and trial metrics.) | Organization | Positive | Outcomes (broad) |
| van der Tuuk 2011 [104] | The Netherlands | Obstetrics: eclampsia | 24 709 women delivered at 38 hospitals participating in a trial compared with 18 839 women at 55 non-participating hospitals before, during and after the trial. (Data: retrospective, Perinatal Registry – patient-records on patient management and prevalence of eclampsia.) | Organization | Positive | Outcomes (specific) |
| Venables 2012 [105] | United Kingdom | Cancer: breast radiotherapy | 51 sites of 7 trials involving the National Radiotherapy Trials Quality Assurance Group over 15 years asked about trial-linked processes. (Data: series of pre-trial and other surveys to facilitate comparisons over time on how trials/protocols help implementation of new techniques.) | Organization | Positive | Processes (specific) |
| Warnecke 1995 [37] | United States | Cancer: breast | 11 450 records from two evaluations: CCOP physicians in 66 hospitals compared with 24 non-CCOP hospitals in first evaluation, fewer in second. (Data: retrospective, hospital records and SEER registry—treatments.) | Clinician | Positive | Processes (specific) |
| Wolfson 2015 [106] | United States | Cancer: 8 adult-onset cancers | 4428 patients with cancer at three NCI-designated centres in Los Angeles County compared with 65 151 at non-NCI sites. (Data: retrospective, county cancer registry – data on survival, case-mix and race/socioeconomic status, which used to assess access equity.) | Organization | Positive | Outcomes (broad) |
| Wuthrick 2014 [107] | United States | Cancer: head and neck | 471 patients in RTOG’s chemoradiation trial: 150 in 13 historically high accrual centres compared with 321 in 88 historically low centres. (Data: retrospective: secondary analysis of trial data on processes/outcomes.) | Organization | Positive | Outcomes (broad) |
| Yawn 2010 [108] | United States | Mental health: depression | 28 practices, 48 trial-site leaders (e.g. 27 family practitioners and 6 nurses) who participating in a PBRN postpartum depression study. (Data: interviews during trial comparing mid-trial processes to previous ones.) | Organization | Positive | Processes (specific) |
| Young 2016 [109] | Australia | ENT: child (speech therapy) | 653 children in New South Wales Aboriginal health services received enhanced clinical care compared with baseline (including 5822 speech-language pathology sessions) through a research collaboration. (Data: case study evaluation – prospective interviews/service delivery data.) | Organization | Positive | Processes (specific) |
Papers in italics mean they are an original review paper. Papers not in italics mean they are an updated review paper.
AMI acute myocardial infarction, CQC Care Quality Commission, CLI chronic limb threatening ischemia, CRN Clinical Research Network, CTN Clinical Trials Network, CRC colorectal cancer, CoC Commission on Cancer, CCOP Community Clinical Oncology Program, ENT ear, nose and throat, GPs general practitioners, LoS length of stay, MI myocardial infarction, NCI National Cancer Institute, NHS National Health Service, NIHR National Institute for Health (and Care) Research, NIDA National Institute on Drug Abuse, PBRN practice-based research network, RTOG Radiation Therapy Oncology Group, RCTs randomized controlled trials, SUD substance use disorder, R&D research and development, SEER Surveillance, Epidemiology and End Results, VA (Department of) Veterans Affairs
Study characteristics
Across the 95 papers, 12 countries are either the location for the research engagement described in a single-country study, or the location from which a multi-country study was led, with one paper led from South Africa having authors from a range of African countries (and Yemen) [88]. The 12 countries are: United States (50 papers), United Kingdom (17), Canada (7), Spain (5), Germany (4), the Netherlands (3), Australia (2), Denmark (2), South Africa (2), China (1), Finland (1) and Sweden (1).
Cancer was the most common field, with 32/95 papers overall. Next came hospital care in general/multi-field/acute care with 16 papers, cardiovascular/stroke (12), substance use disorder (7), dentistry (3), mental health/psychiatry (3) and obstetrics (3).
Main findings
As presented in Table 2, the key findings from the combined review are presented in terms of the four pairs of binary options, though inevitably some papers did not neatly fit into one category. The first categorization is in terms of the level of analysis explored in different papers; 23 papers compare clinicians, but 72 compare organizations. There is an even higher proportion in the updated review at the organizational level (50/62, 81%) than in our original review (22/33, 67%).
A total of 86 of the 95 papers report positive results, of which 70 are purely positive and 16 are positive/mixed meaning that there are some key negative elements in their findings, that is, important parts of the analysis where a lack of healthcare improvement is identified. Nine papers are negative, of which four are negative-mixed.
The final two pairs of binary options consider just the 86 positive papers. In total, 37/86 report improved health outcomes in terms of reduced mortality or morbidity. A higher proportion of the positive papers in the updated review (30/58, 52%) than in our original review (7/28, 25%) describe such improved health outcomes. There is a corresponding reduction from three quarters (21/28) to a half (28/58) in the proportion of papers solely describing improved processes in terms such as applying proven interventions.
Finally, in terms of the type of impact, 55/86 of the papers describe research engagement leading to a broader impact on healthcare performance. Broad impacts arise when the improved healthcare goes more widely than just being linked to clinicians or healthcare organizations implementing the findings, or processes, from their own research more rapidly/extensively than do others. When the improved healthcare is linked to the results or processes of their own research, that is categorized as specific impact, which is the case in 31/86 papers. Using these various categories, Fig. 2 outlines the findings from the combined review, alongside the findings from our original review, and the updated review. This highlights various trends in terms of the main findings.
Fig. 2.
Results from Boaz et al. systematic reviews of whether research engagement by health organizations and staff improves healthcare: analysis of original; updated; and combined reviews (and of the 86 positive papers). Green rows (top): original review; Brown rows (middle): updated review; blue rows (bottom): combined review
One further trend in terms of the type of analysis is seen in the 11/95 papers that used bibliometric analysis as an indicator of the extent, and/or quality, of research engagement compared with some measure of the healthcare performance, in terms of processes and/or outcomes [30, 40, 50, 51, 56, 65, 76, 95–97, 100]; 10 of these papers are in the updated review, with just 1 [30] from our original review. All these 11 papers are positive, but various types of bibliometric analysis are used. The broad categories of academic indicators applied include publication volume [95, 96], publication “quality” (for example, as measured by citations) [51] and a combination of volume and “quality” [30, 40, 50, 56, 65, 76, 97, 100]. Of the latter, five relatively small studies suggest that the association with “quality” was stronger than with volume. The bibliometric studies also illustrate the varying levels of analysis at which the included studies in the review are conducted; 4 of the 11 papers compare the academic outputs of clinicians [50, 56, 96, 97] and 7 make comparisons at an organizational level [30, 40, 51, 65, 76, 95, 100], focussing variously on academic outputs at ward, department or hospital/trust level.
The combined review allows for a range of issues to be analysed more thoroughly than they had been in our original review. These include issues highlighted in the background such as the role of networks and the “dose effect”. These are examined in turn below, followed by consideration of how far the included studies have addressed various aspects of health equity, and finally an analysis of lessons from the overall portfolio of positive and negative studies.
The role of research networks
The full significance of papers on research networks is seen in the combined review. Using the inclusive definition developed by Laliberte et al. [24], we have applied the term to various arrangements that, however loosely, give some measure of commonality to the research of multiple healthcare organizations that not only enhance science production, but also share a concern to transfer research findings into clinical practice. About half the papers in the combined review analysed research activity by clinicians or healthcare organizations who were part of research networks of various types.
In the United States, the NCI cancer research networks include the NCI-designated Comprehensive Cancer Centres, the NCI Cooperative Groups and collaborative groups of community hospitals affiliated to the NCI’s CCOP- see the Glossary for its new name. In various ways these networks all include outreach and the engagement of community physicians in their brief; see the Glossary for more details. Their potential was recognized early in the 2005 study by Laliberte et al. [24] that looked at these networks and concluded that network membership may influence compliance with treatment guidelines, and should therefore be taken into account in predictive models of compliance.
Seven included papers illustrated various aspects of this issue by comparing the processes and outcomes for patients treated at NCI-designated (comprehensive) cancer centres with those treated elsewhere, six of these studies showed better outcomes for patients treated at NCI centres [52, 64, 80, 84, 86, 106], while one paper suggested that despite better processes, patient outcomes were worse at NCI centres. This paper is considered in the section on negative papers below [81]. Of the positive papers, Paulson et al. showed how the NCI designation was “associated with lower risk of postoperative death and improved long-term survival” (p. 675) [86], identified possible factors such as better adherence to guidelines, and demonstrated that the better outcomes at NCI-designated centres remained even when compared with non-NCI designated centres with a similar high volume of cases [86]. Wolfson et al. identified the requirements that underpin the positive association between high-quality research and high-quality care [106]. These included the mandate NCI centres have to “lead clinical trials, exchange ideas, disseminate findings” (p. 3892), which showed how the centres could act as part of a network. Wolfson et al. continued: “The NCI operates on the belief that a culture of discovery, scientific excellence, transdisciplinary research, and collaboration yields tangible benefits extending far beyond the generation of new knowledge” [106].
Building on Laliberte et al. [24], Carpenter et al. demonstrated an association between CCOP membership and accelerated innovation adoption but added the important codicil that it was not possible to “definitively ascertain whether there is a direct causal relationship between the two” [54].
Improved healthcare has also been associated with membership of the United States practice-based research networks (PBRNs). These networks cover family practice/primary care, dentistry, mental health and substance abuse. Like the CCOP and its affiliates, PBRNs involve practising clinicians in the community who conduct research. The combined review includes seven PBRN papers covering primary care and dentistry, all of which are positive [32, 36, 66, 78, 83, 92, 108] and one of which describes an international dental PBRN led from the United States that includes three Scandinavian countries [66].
A total of seven papers from another PBRN, the National Institute on Drug Abuse’s Clinical Trials Network (CTN), also provided evidence of accelerated translation, identified mechanisms through which this might work, and discussed the theoretical frameworks within which those operated [5, 14, 23, 49, 63, 90, 91]. Thus, Ducharme et al. [14] and Knudsen et al. [23] explored Rogers’ notion of the “trialability” [110] of innovations, that is, how far an innovation may be experimented with on just a limited basis, and Abraham et al. [5] discussed the role of absorptive capacity [111, 112], which they summarized as an organization’s ability to assess and use information [5]. Rieckmann et al. noted that although the mechanisms involved were not fully understood they appeared “to be influenced by core experiences from network participation” (p. 894) [91], and Fields et al. [63] used insights from implementation science to explore the influence of a set of organizational characteristics (including network membership) on innovation adoption [113].
In an analysis of data on 12 993 transplants conducted in 162 US centres, the 32 centres in the Bone Marrow Transplant trials network were found to have significantly better survival rates than others [77]. Marmor et al. reported that there was not an association between procedure volume and survival. Rather, they suggested, the better outcomes for those treated in centres in this network could be linked to the nature of trials that required “higher levels of national clinical collaboration and standardization of protocols”, and such collaboration was “likely to generate higher levels of innovation and excellence among clinical colleagues” (p. 92) [77].
In Germany, one team produced three papers on the improved healthcare performance of hospitals that were part of clinical trials organizations [13, 34, 94]. Two papers described the improved outcomes for patients with ovarian cancer if they were treated in a hospital that belonged to one of two German ovarian cancer clinical trials organizations, in effect research networks [13, 34]. They noted that the improved outcomes were not related to patient volume, suggesting instead that possible factors may include hospitals’ participation in the study group’s quality assurance programs and team members attending regular and scientific and educational meetings [13]. In a follow-up study, the data were analysed in more detail using mediation analysis that showed not just that the research participation of a hospital contributed to superior patient survival, but also began to unpick how it happened, including through better use of surgery and chemotherapy [94].
Downing et al. noted that, following the 2006 establishment of the NIHR in the United Kingdom, the increase in research activity in networks throughout the English NHS also increased the scope for analysing the benefits of research engagement [58]. The role of NIHR networks in boosting research engagement, which is then linked to improved healthcare, also covers clinicians such as nurses and AHPs who had traditionally had limited research opportunities. Studies are now showing how they can play an important role by engaging in research because, according to Trusson et al. reporting on a research network for nurses and AHPs, people working in such roles “have opportunities to explore possible solutions to issues that they encounter in their clinical role through academic study” (p. 1) [101]. Such opportunities can also enhance their clinical skills. More broadly, Downing et al. claimed that, in relation to the NIHR’s clinical trials network, “this natural experiment, presented by the rapid expansion of trial activity across a whole national health system, is perhaps the best opportunity to address the subject though outcomes research” (p. 95) [58]. This development is discussed in the next section.
The “dose effect” of the extent of research engagement
Evidence indicating a link between the extent of research engagement and the degree of improved healthcare has been accumulating for some time. In the United States, the 1996 study by Brown and Griffiss found that the average acute length of stay (LoS) in Department of Veteran Affairs hospitals was inversely related to the size of research programmes [53]. Majumdar et al. [26] used a tertile approach to show that in-hospital mortality decreased as the rate of trial participation increased in the area of unstable angina. In the substance abuse field, early CTN studies also contributed: thus Knudsen et al. [23] noted that the adoption of buprenorphine therapy by practitioners within the trials’ network was much greater in those programmes in the network that participated in the specific buprenorphine trial than those that had not. In a 2006 study of a sexual health trial in Australia, Morton et al. [28] identified improved post-trial clinical practice by high-recruiting clinicians, but not by low-recruiting ones.
In our combined set of papers the first use of the specific term “dose effect” to describe the effects of differing amounts of research engagement occurred in Downing et al., who tested the hypothesis that for colorectal cancer (CRC) “high, sustained hospital-level participation in interventional clinical trials improves outcomes for all patients with CRC managed in those research-intensive hospitals” (p. 89) [58]. They found that high participation in such clinical trials was independently associated with better outcomes and that these effects were not restricted to academic centres or large institutions but were seen across all the NHS Trusts that conducted research on and treated patients with colorectal cancer. They extended their analysis to look at the effects of different levels of research participation and found that the highest levels of participation led to the highest levels of improved outcomes. However, in relation to these findings, Downing et al. were careful to say that, in the absence of the possibility of an RCT, caution was needed if attempting “to infer a causal contribution” (p. 89) from participation in research activity to improved healthcare [58].
Other United Kingdom database studies support the findings of Downing et al. For example, Ozdemir et al. [85] compared mortality with research funding per hospital bed in hospitals with high, medium and low levels of research funding and showed that not only was mortality lower in high-funded research hospitals than in other hospitals, but also, on average, hospitals in the middle category had a lower mortality rate than ones with the least research funding. In two studies using NIHR research study activity data from different years, Jonker and Fisher [68, 69] showed an inverse correlation between the number of clinical trials/patient participation levels in United Kingdom hospitals and the mortality rate. Lin et al. [73] used retrospective data to examine the survival rate of the 465 patients (recruited by 60 hospitals) who had participated in an RCT in the NIHR Clinical Research Network (CRN). While they identified a significant association between low trial recruitment and lower survival rates, looking at the volume of patients treated in the disease area by the respective hospitals they report that “no significance was found between hospital throughput and outcomes” (p. 40) [73].
Further support for the “dose effect” concept comes from the United States and elsewhere. According to Abraham et al., in the substance abuse field “treatment programs participating in a greater number of CTN protocols had significantly higher levels of treatment quality, an association that held after controlling for key organizational characteristics” (p. 232) [49]. Similarly, Gilbert et al. [66] reported that members of a dental PBRN who fully participated in the network were more likely to move evidence-based care into everyday practice than members who only partially participated. Seaburg et al. [96] showed an association between the quantity of resident physicians’ publications and their clinical performance scores during training, and García-Romero et al. claimed that increases in the scientific output of Spanish hospitals made a significant contribution to a reduction of hospital LoS [65].
In Canada, Tsang et al. [103] conducted a pre-planned observational study nested within a clinical trial to test how well traditionally non-research active community hospitals could participate in an RCT alongside the traditional RCT sites in academic hospitals. However, while that aspect of the study did show that, in terms of adherence to trial metrics, the community hospitals could successfully participate in studies, outcomes for patients in the trial were significantly better in the traditional research hospitals, although the full reasons for this will need further exploration [103].
Health equity
Various aspects of health equity are considered in the included papers, and some of these report attempts to improve health equity. Some population groups are particularly vulnerable. In the United States, for example, Wolfson et al. listed the following groups: “underrepresented minorities, those with low socio-economic status (SES), those with public or no insurance, and those with a significant distance to care” (p. 3886) [106]. On the basis of its long-held assumption that patient access to research active healthcare providers is beneficial, the NCI has attempted to reduce geographic inequalities in access. In a 1995 paper, Warneke et al. noted that the CCOP was established by the NCI in 1983 with the deliberate intention of spreading the benefits of the clinical research conducted in NCI centres: “The program was designed with the assumption that by participating as equals in the research process, community physicians would be more likely to accept and implement the results in their practices with non-protocol patients” (p. 336) [37].
Similar moves to encourage wider participation in clinical trials have recently been made in Canada in the nested study described above [103]. A recent analysis showing higher levels of research activity within the English healthcare system were associated with lower mortality, noted that although the NIHR CRN was established to promote research participation across England, there was still some way to go to ensure greater geographical equity [69].
Other initiatives, such as the United States minority-based CCOPs described in the Glossary, addressed racial inequalities in relation to access to research engagement and timely evidence-based healthcare. These sometimes overlap with geographic inequalities. Some of the papers on the NCI-designated cancer centres observed with concern that the proportion of certain racial/ethnic groups, including African Americans, who received treatment at these centres compared with non-NCI centres, was lower than for other racial groups [64, 80, 106]. Having noted that African Americans with colon cancer experienced worse outcomes than Caucasian Americans, and suggested that this was partly due to differential treatment, a study by Penn et al. found evidence that African Americans receiving treatment from CCOP providers had benefitted from a seemingly deliberate attempt to boost early access to a recently recommended innovative treatment [87]. In Australia, Young et al. [109] reported that the health services, and health research system, of the Aboriginal community work together to try to ensure health research is embedded into activities that improve health, and described a specific example in relation to ear, nose and throat surgery and speech-language pathology services.
Lessons from the overall collection of studies: positive and negative
A wide variety of papers contribute to the combined review’s overall finding that the included studies are overwhelmingly positive. As the section on the “dose effect” illustrates, throughout the time covered by the combined review, individual papers have contributed to a wider understanding that goes beyond specific issues about research networks. Many papers contribute to the analysis of both the strength of the association between research engagement and improved healthcare, and the mechanisms involved. For example, a 2019 US positive study by Fanaroff et al. [60] identified improved care and outcomes for patients with acute myocardial infarction who were treated at research active hospitals, even after accounting for potential confounders. The authors encapsulated some of the key thinking on research engagement with their conclusion that participation in clinical trials by hospitals “may be emblematic of a culture that embraces novel therapeutics, engages both clinicians and patients, and incentivizes continuous improvement in care” (p. 191) [60].
While overall the 95 studies included in the combined review are positive, about 10% are categorized as negative. These nine negative papers also provide important insights [7, 11, 15, 20, 25, 67, 79, 81, 99]. For example, existing widespread use of one proven intervention prior to a company-sponsored clinical trial exploring physicians’ adherence to international treatment recommendations meant that the trial had no significant impact on that adherence, although it did increase use of the trial sponsor’s drug [7]; physicians adopted another trial intervention before it was proven one way or another [11]; more positively, a unique policy and regulatory environment governing the adoption of another intervention ensured that all hospitals benefitted, not just those in the trial [79]. Two teams with negative results later conducted further, more comprehensive studies with positive conclusions [25, 26, 67–69]. Six of the seven papers examining whether NCI-designated cancer centres provided patients with better healthcare processes and outcomes are positive [52, 64, 80, 84, 86, 106]. However, one paper suggested that outcomes were worse in these accredited hospitals despite the better healthcare and, in seeking to explain this, drew attention to the factors considered in the accreditation processes used by different organizations and how far they accurately captured the most relevant data [81].
Discussion
Our original review set out to find whether there was empirical evidence that supported the often-held assumption that engagement by clinicians and healthcare organizations in research improves healthcare performance at various levels. It concluded that there was some positive evidence but that systematic analysis of the data related to this engagement was in its infancy [2]. The 62 papers in the updated review, 58 of which are positive, provide further empirical evidence to support the positive conclusions of the original review.
When the papers from both reviews are considered together, they provide a more complete dataset than previously available [1–4], and an updated picture of this literature in which the trends identified in our initial analyses [3] become more apparent. With more than a third of the papers in the combined review (32/95) focussing on aspects of cancer, this is the field overall in which there is the most comprehensive analysis of the link between research engagement and improved healthcare. While the individual cancer papers differ in the strength of the association identified, and most of the papers focus one or other of the main cancer sites, many of the cancer papers analyse the role of research networks – one of the main mechanisms through which it is claimed research engagement improves healthcare.
The combined review reflects policy shifts and organizational changes that occurred first in the United States and later in the United Kingdom and elsewhere, and were designed to address the time lag between the production of research and its use in practice. These include the development of research networks and their associated databases over several decades (accompanied by an improved understanding of their strengths and limitations [54, 64, 77, 106, 108]) and efforts to strengthen links between academic centres and community services [61, 87]. More recent developments, especially in the United Kingdom, encouraged further deliberate attempts to identify and explore the impacts of research engagement. Research teams were, for example, better able to study the real-world impacts of system-level mechanisms such as research networks as they became more formalized and embedded in national health and science structures [58, 67–69, 85, 93, 101].
Across the board, within and beyond networks, there is also further evidence about the mechanisms by which research engagement might improve healthcare, including the ones identified in our original review. The role of strong evidence-based protocols developed for RCTs, but contributing to improved healthcare more widely in research active healthcare sites, was highlighted in various studies [77, 98, 105]. Papers also identified the importance of providing evidence-based/guideline consistent care, which could also be linked to a culture of discovery, excellence and collaboration [40, 60, 62, 64, 77, 84, 86, 87, 106]. There were also more nuanced mechanisms at the speciality and clinician levels, such as the use of multi-disciplinary coordination of care in radiation therapy treatment [107] and practitioner skill development in substance abuse work [90]. Similar practitioner skill development was also reported among nurses and AHPs, including in the wider literature [31, 38, 39, 62, 70, 101].
In the combined review it also became easier to see connections across this diverse literature. It was possible to identify research teams that had worked together on multiple studies and to explore the extent of cross referencing. In the United States, for example, the CTN of the drug abuse institute had been created to emulate the CCOP, and a centre was established to assess the CTN’s impact [114]. Analysis of this research network highlighted its role both in conducting research that was relevant to the “real-world” needs of clinical settings, and in enhancing evidence-adoption by healthcare organizations and staff [114]. Many of the papers from this substance abuse CTN [23, 49, 90] referenced each other and also cross-referenced key cancer papers [8, 24, 54], and there was common use of the same early sources [110, 112, 115, 116]. These interactions prompted ongoing methodological development, strengthened understanding of theoretical concepts, and supported shared learning across the specialities. Additionally, themes that had been recognized in the original review, including concepts such as absorptive capacity [5, 111], were further explored and tested in new contexts, even if the same literature was not always drawn upon [40, 65].
In the combined review, the nature and strength of the association found between research engagement and improved health varies enormously among the 86 positive papers, even among those that describe the role of research networks. One approach that begins to identify where evidence might be strongest was noted in the original review as being the important concept of the “dose effect”, even if it was not specifically labelled as such [26]. However, the combined review can now more fully consider the concept because evidence about this greatly increased as the scope of the papers included has increased. There are many more studies where all the clinicians or organizations compared are engaged in research but to varying extents and/or with different levels of resources, for example within a trial [28, 50, 59, 73, 93, 103, 107] or within a network [23, 33, 49, 51, 66, 68, 69, 85, 95]. The inclusion of papers regarding differences within trials, and the emergence of the importance of the “dose effect”, have implications for both (a) how the issue of research engagement is analysed and (b) how far efforts to enhance research engagement should be concentrated or spread widely across a system.
In relation to the first of these issues, when considering how research engagement is analysed, the key question morphs somewhat: it is no longer simply whether research engagement improves healthcare performance compared with no research engagement, rather, it is whether a larger amount of research engagement improves healthcare performance by more than a smaller level of engagement (and, if so, by how much). Answers to these questions could then feed back to strengthen the evidence for a positive association between research engagement and improved healthcare performance.
In relation to the second question, about the concentration or wide distribution of research funding, analyses might have to consider the context and trade-offs in terms of benefits for improved health and health equity. The widespread distribution of research funding across the health system could maximize the number of patients who might benefit, but a more concentrated approach, with a higher dose of research engagement in a smaller number of hospitals, could maximize the benefit for patients in such centres.
Research infrastructures in countries such as the United States and United Kingdom have been developed to enhance the relationship between health and health research systems, and the evidence from our combined review suggests that these changes have been positive. In both systems, but particularly in the United Kingdom, there have been deliberate attempts to fund major centres of research in leading healthcare facilities, as well as to spread research funding more widely to healthcare organizations across the country, but this impetus needs to be maintained if the full benefits of research engagement are to be realized.
Such an argument is reinforced by the conclusions of a major recent analysis of progress in the United Kingdom in engaging healthcare staff in research and building research capacity. The findings from the study suggest that many healthcare staff in the United Kingdom are interested in being involved in research, there are supportive national policies and strategies in place and there has been some important progress. However, achieving widespread involvement “will only be possible by focusing more on how healthcare organizations embed and support research activity through organizational policies which are supported by the wider research support and funding infrastructure. This is an essential part of a system-based approach to developing and supporting research engagement” (p. 356) [117]. The progress possible, and the potential benefits of trying to build a health research system embedded into a healthcare system, but also the full range of substantial challenges, have also recently been explored in a hospital and regional healthcare system in northern Queensland, Australia [118, 119]. Studies such as these indicate that this combined review could provide timely evidence to further the challenging task of improving healthcare by boosting engagement in health research.
Strengths and limitations
The combined review contains a considerable number of papers from diverse perspectives, but the literature is drawn predominantly from the United States and the Global North, thus the conclusions may not be appropriate in different contexts, including in the Global South. This, perhaps, partly reflects the inclusion criteria of papers in English only. While the increasing use of bibliometrics as an indicator of research engagement has widened the range of positive studies available, differing claims as to the most appropriate measure of research publications challenge consistent interpretation of the data and indicate there is more work to do. Furthermore, it is important to recognize that the national policy, noted in one paper, of attaching promotion and bonuses for clinicians to publish in journals with an impact factor of at least three [97] runs contrary to the internationally widely endorsed Declaration on Research Assessment [120].
The complexity of this literature (with many generic terms such as “research” and “engagement”), and the tangential approach of some papers to the broad question of whether research engagement improves performance, posed considerable challenges. It helped enormously that this time around, we were able to build on our experience in the original review. We adopted a somewhat more extensive approach to the formal search in the updated review, and we identified some papers that we had missed in the original review. We were aided by the generally greater clarity in later papers. We are now able, therefore, to present a more nuanced understanding of this field, building on our experience in the original review. In particular, we have found considerably more evidence on two topics identified as important in our original review, and on their implications for health equity: the role of research networks and consideration of how far there is a dose effect with regard to the degrees of research engagement. On both topics the combined review has strong papers showing important healthcare improvements even after considering potential confounders such as patient volume [8, 13, 26, 40, 58, 73, 77, 84–86]. However, the failure of some papers to address such confounders [59, 107] means some weaknesses in the overall analysis remain, and we are still not able to undertake any meta-analysis as the included literature remains very diverse.
We have now included a significant range of largely positive papers in the combined review. However, lack of resources meant we were not able to replicate our original review’s [2, 3] structured analysis of the wider range of papers identified as making many relevant and illuminating points related to the topic, but not meeting the review’s inclusion criteria. For example, while the combined review does include some consideration of health equity issues, there were papers taken to full paper review that were not in the end included but which provide considerably more evidence [121, 122].
Future possible work
The system-based approaches for expanding the amount of research in healthcare systems that are mentioned above continue to provide important opportunities for further work on exploring the relationship between research engagement and improved healthcare, including the implications for health equity. Likewise, improvements in the identification and collection of relevant data and developments in statistics have prompted increasingly sophisticated analyses, sometimes using approaches developed in other fields, and could continue to do so [65, 90, 94]. There has also been increasingly sophisticated use of bibliometrics, and there are likely to be continuing opportunities to apply such approaches to more countries. However, the warning from Downing et al. that caution is needed if attempting “to infer a causal contribution” from research participation to improved health outcomes [58], as well as frequent mention of similar disclaimers in other papers [8, 40, 54, 59, 69, 74], is a reminder that more work is needed.
While some of our papers have claimed that the costs of research engagement are broadly covered by the associated reduced LoS [53, 65], further research might be useful around the costs associated with research engagement and how these relate to reported benefits. Such studies could add to the existing large-scale studies showing the considerable monetary value of the health and economic gains resulting from health research [123].
The insights revealed by the negative papers, particularly in relation to the contexts in which research and research networks operate [99], could usefully be further explored. Merkow et al. [81], the one negative paper out of seven papers included on the NCI-designated centres, raises issues about the accuracy, or perhaps appropriateness, of the measurement used by various organizations to accredit cancer centres. These issues have also been explored by various teams [122, 124, 125] but could perhaps be worth further examination because the findings from Merkow et al. are so starkly different from those of other papers included in our review.
Finally, there are increasing opportunities, as well as a growing need, to address the limitations identified above (and also noted in the review by Chalmers et al. [39]) and go beyond the formal inclusion criteria of this review. A major area that could usefully be incorporated into an overall analysis of the field relates to the impact of the growing interest in research engagement strategies [126]. This includes the efforts to enhance research roles for healthcare professionals other than medical professionals [38], and the increasing number of organizational arrangements within health and health research systems for partnerships that seek to boost the production and use of relevant evidence [127, 128].
Conclusion
Previous reviews [1–4] have investigated the association between research engagement and improvements in healthcare performance. This study updates and extends the most comprehensive of these reviews [2, 3], and combines its findings with those from that original review to produce a more substantial pool of studies, which are largely positive in terms of the impact of research engagement on processes of care and patient outcomes. Of potential mechanisms, the combined review highlights the important role played by research networks and further identifies the various ways the research engagement facilitated by them operates to improve healthcare. The review also draws together a set of papers which consider how far there is a research engagement “dose effect”. Given the difficulty of conducting randomized controlled trials of large-scale research engagement initiatives, studies of the dose effect offer another approach to understanding the potential contribution and complexities of research engagement, including the implications for health equity. This review provides further evidence of the important contribution played by systems-level research investments such as research networks on processes of care and patient outcomes.
Supplementary Information
Acknowledgements
The authors would like to thank Teresa Jones for her expert advice on the search strategy. The review was completed as partial fulfilment of Belinda Goodenough’s Masters dissertation at King’s College London. In the original full report for our first review [3], we gratefully acknowledged the valuable help we had received from our expert advisory group. The members included two patient representatives who were consulted at various stages throughout the project, especially around the necessity of having our systematic review focus on the complexities of benefits from research engagement by healthcare organizations and staff, while separate reviews and analyses focussed on the benefits of PPI in health research.
Abbreviations
- AHP
Allied health professionals
- CRN
Clinical research network
- CTN
Clinical trials network
- CCOP
Community Clinical Oncology Program
- LoS
Length of stay
- MMAT
Mixed-methods appraisal tool
- NCI
National Cancer Institute
- NHS
National Health Service
- NIHR
National Institute for Health (and Care) Research
- PBRN
Practice-based research network
- R&D
Research and development
- RCTs
Randomized controlled trials
Author contributions
All authors were involved in planning the study. B.G. conducted the searches, with all authors involved in screening and analysis. B.G. produced an initial draft of the paper. The final version of the paper was produced collaboratively by all the authors.
Funding
None.
Availability of data and materials
No datasets were generated or analysed during the current study.
Declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Clarke M, Loudon K. Effects on patients of their healthcare practitioner’s or institution’s participation in clinical trials: a systematic review. Trials. 2011;12:16. 10.1186/1745-6215-12-16. 10.1186/1745-6215-12-16 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Boaz A, Hanney S, Jones T, Soper B. Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review. BMJ Open. 2015;5: e009415. 10.1136/bmjopen-2015-009415. 10.1136/bmjopen-2015-009415 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hanney S, Boaz A, Jones T, Soper B. Engagement in research: an innovative three-stage review of the benefits for health-care performance. Health Serv Deliv Res. 2013;1:8. 10.3310/hsdr01080. 10.3310/hsdr01080 [DOI] [PubMed] [Google Scholar]
- 4.Harding K, Lynch L, Porter J, Taylor NF. Organisational benefits of a strong research culture in a health service: a systematic review. Aust Health Rev. 2017;41:45–53. 10.1071/AH15180. 10.1071/AH15180 [DOI] [PubMed] [Google Scholar]
- 5.Abraham AJ, Knudsen HK, Rothrauff TC, Roman PM. The adoption of alcohol pharmacotherapies in the clinical trials network: the influence of research network participation. J Subst Abuse Treat. 2010;38:275–83. 10.1016/j.jsat.2010.01.003. 10.1016/j.jsat.2010.01.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Adler MW. Changes in local clinical practice following an experiment in medical care: evaluation of evaluation. J Epidemiol Community Health. 1978;32:143–6. 10.1136/jech.32.2.143. 10.1136/jech.32.2.143 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Andersen M, Kragstrup J, Sondergaard J. How conducting a clinical trial affects physicians’ guideline adherence and drug preferences. JAMA. 2006;295:2759–64. 10.1001/jama.295.23.2759. 10.1001/jama.295.23.2759 [DOI] [PubMed] [Google Scholar]
- 8.Carpenter WR, Reeder-Hayes K, Bainbridge J, Meyer A-M, Amos KD, Weiner BJ, et al. The role of organizational affiliations and research networks in the diffusion of breast cancer treatment innovation. Med Care. 2011;49:172–9. 10.1097/MLR.0b013e3182028ff2. 10.1097/MLR.0b013e3182028ff2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chaney EF, Rubenstein LV, Liu C-F, Yano EM, Bolkan C, Lee M, et al. Implementing collaborative care for depression treatment in primary care: a cluster randomized evaluation of a quality improvement practice redesign. Implement Sci. 2011;6:121. 10.1186/1748-5908-6-121. 10.1186/1748-5908-6-121 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Chen AY, Schrag N, Hao Y, Flanders WD, Kepner J, Stewart A, et al. Changes in treatment of advanced laryngeal cancer 1985–2001. Otolaryngol Head Neck Surg. 2006;135:831–7. 10.1016/j.otohns.2006.07.012. 10.1016/j.otohns.2006.07.012 [DOI] [PubMed] [Google Scholar]
- 11.Clark WF, Garg AX, Blake PG, Rock GA, Heidenheim AP, Sackett DL. Effect of awareness of a randomized controlled trial on use of experimental therapy. JAMA. 2003;290:1351–5. 10.1001/jama.290.10.1351. 10.1001/jama.290.10.1351 [DOI] [PubMed] [Google Scholar]
- 12.Das D, Ishaq S, Harrison R, Kosuri K, Harper E, Decaestecker J, et al. Management of Barrett’s esophagus in the UK: overtreated and underbiopsied but improved by the introduction of a national randomised trial. Am J Gastroenterol. 2008;103:1079–89. 10.1111/j.1572-0241.2008.01790.x. 10.1111/j.1572-0241.2008.01790.x [DOI] [PubMed] [Google Scholar]
- 13.du Bois A, Rochon J, Lamparter C, Pfisterer J, for the Organkommission OVAR. Pattern of care and impact of participation in clinical studies on the outcome in ovarian cancer. Int J Gynecol Cancer. 2005;15:183–91. 10.1111/j.1525-1438.2005.15202.x. 10.1111/j.1525-1438.2005.15202.x [DOI] [PubMed] [Google Scholar]
- 14.Ducharme LJ, Knudsen HK, Roman PM, Johnson JA. Innovation adoption in substance abuse treatment: exposure, trialability, and the clinical trials network. J Subst Abuse Treat. 2007;32:321–9. 10.1016/j.jsat.2006.05.021. 10.1016/j.jsat.2006.05.021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Goldberg HI, Neighbor WE, Hirsch IB, Cheadle AD, Ramsey SD, Gore E. Evidence-based management: using serial firm trials to improve diabetes care quality. Jt Comm J Qual Improv. 2002;28:155–66. [DOI] [PubMed] [Google Scholar]
- 16.Hall C, Sigford B, Sayer N. Practice changes associated with the Department of Veterans Affairs’ Family Care Collaborative. J Gen Int Med. 2010;25(Suppl. 1):18–26. 10.1007/s11606-009-1125-3. 10.1007/s11606-009-1125-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hébert-Croteau N, Brisson J, Latreille J, Blanchette C, Deschenes L. Variations in the treatment of early-stage breast cancer in Quebec between 1988 and 1994. CMAJ. 1999;161:951–5. [PMC free article] [PubMed] [Google Scholar]
- 18.Janni W, Kiechle M, Sommer H, Rack B, Gauger K, Heinrigs M, et al. Study participation improves treatment strategies and individual patient care in participating centers. Anticancer Res. 2006;26:3661–7. 10.1016/S0960-9776(05)80107-9. 10.1016/S0960-9776(05)80107-9 [DOI] [PubMed] [Google Scholar]
- 19.Jha P, Deboer D, Sykora K, Naylor CD. Characteristics and mortality outcomes of thrombolysis trial participants and nonparticipants: a population-based comparison. J Am Coll Cardiol. 1996;27:1335–42. 10.1016/0735-1097(96)00018-6. 10.1016/0735-1097(96)00018-6 [DOI] [PubMed] [Google Scholar]
- 20.Jones B, Ratzer E, Clark J, Zeren F, Haun W. Does peer-reviewed publication change the habits of surgeons? Am J Surg. 2000;180:566–9. 10.1016/S0002-9610(00)00495-5. 10.1016/S0002-9610(00)00495-5 [DOI] [PubMed] [Google Scholar]
- 21.Karjalainen S, Palva I. Do treatment protocols improve end results? A study of survival of patients with multiple myeloma in Finland. BMJ. 1989;299:1069–72. 10.1136/bmj.299.6707.1069. 10.1136/bmj.299.6707.1069 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Kizer JR, Cannon CP, McCabe CH, Mueller HS, Schweiger MJ, Davis VG, et al. Trends in the use of pharmacotherapies for acute myocardial infarction among physicians who design and/or implement randomized trials vs physicians in routine clinical practice: the MILIS-TIMI experience. Am Heart J. 1999;137:79–92. 10.1016/s0002-8703(99)70462-x. 10.1016/s0002-8703(99)70462-x [DOI] [PubMed] [Google Scholar]
- 23.Knudsen HK, Abraham AJ, Johnson JA, Roman PM. Buprenorphine adoption in the national drug abuse treatment clinical trials network. J Subst Abuse Treat. 2009;37:307–12. 10.1016/j.jsat.2008.12.004. 10.1016/j.jsat.2008.12.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Laliberte L, Fennell ML, Papandonatos G. The relationship of membership in research networks to compliance with treatment guidelines for early-stage breast cancer. Med Care. 2005;43:471–9. 10.1097/01.mlr.0000160416.66188.f5. 10.1097/01.mlr.0000160416.66188.f5 [DOI] [PubMed] [Google Scholar]
- 25.Majumdar SR, Chang W-C, Armstrong PW. Do the investigative sites that take part in a positive clinical trial translate that evidence into practice? Am J Med. 2002;113:140–5. 10.1016/S0002-9343(02)01166-X. 10.1016/S0002-9343(02)01166-X [DOI] [PubMed] [Google Scholar]
- 26.Majumdar SR, Roe MT, Peterson ED, Chen AY, Gibler WB, Armstrong PW. Better outcomes for patients treated at hospitals that participate in clinical trials. Arch Intern Med. 2008;168:657–62. 10.1001/archinternmed.2007.124. 10.1001/archinternmed.2007.124 [DOI] [PubMed] [Google Scholar]
- 27.Meineche-Schmidt V, Hvenegaard A, Juhl HH. Participation in a clinical trial influences the future management of patients with gastro-oesophageal reflux disease in general practice. Aliment Pharmacol Ther. 2006;24:1117–25. 10.1111/j.1365-2036.2006.03046.x. 10.1111/j.1365-2036.2006.03046.x [DOI] [PubMed] [Google Scholar]
- 28.Morton AN, Bradshaw CS, Fairley CK. Changes in the diagnosis and management of bacterial vaginosis following clinical research. Sex Health. 2006;3:183–5. 10.1071/SH06024. 10.1071/SH06024 [DOI] [PubMed] [Google Scholar]
- 29.Pancorbo-Hidalgo PL, Garcia-Fernandez FP, Lopez-Medina IM, Lopez-Ortega J. Pressure ulcer care in Spain: nurses’ knowledge and clinical practice. J Adv Nurs. 2007;58:327–38. 10.1111/j.1365-2648.2007.04236.x. 10.1111/j.1365-2648.2007.04236.x [DOI] [PubMed] [Google Scholar]
- 30.Pons J, Sais C, Illa C, Méndez R, Suñen E, Casas M, et al. Is there an association between the quality of hospitals’ research and their quality of care? J Health Serv Res Policy. 2010;15:204–9. 10.1258/jhsrp.2010.009125. 10.1258/jhsrp.2010.009125 [DOI] [PubMed] [Google Scholar]
- 31.Puoane T, Sanders D, Ashworth A, Chopra M, Strasser S, McCoy D. Improving the hospital management of malnourished children by participatory research. Int J Qual Health Care. 2004;16:31–40. 10.1093/intqhc/mzh002. 10.1093/intqhc/mzh002 [DOI] [PubMed] [Google Scholar]
- 32.Rhyne R, Sussman AL, Fernald D, Weller N, Daniels E, Williams RL, et al. Reports of persistent change in the clinical encounter following research participation: a report from the primary care multiethnic network (PRIME Net). J Am Board Fam Med. 2011;24:496–502. 10.3122/jabfm.2011.05.100295. 10.3122/jabfm.2011.05.100295 [DOI] [PubMed] [Google Scholar]
- 33.Rich AL, Tata LJ, Free CM, Stanley RA, Peake MD, Baldwin DR, et al. How do patient and hospital features influence outcomes in small-cell lung cancer in England? Br J Cancer. 2011;105:746–52. 10.1038/bjc.2011.310. 10.1038/bjc.2011.310 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Rochon J, du Bois A. Clinical research in epithelial ovarian cancer and patients’ outcome. Ann Oncol. 2011;22(Suppl. 7):vii16–9. 10.1093/annonc/mdr421. 10.1093/annonc/mdr421 [DOI] [PubMed] [Google Scholar]
- 35.Salbach NM, Guilcher SJ, Jaglal SB, Davis DA. Determinants of research use in clinical decision making among physical therapists providing services post-stroke: a cross-sectional study. Implement Sci. 2010;5:77. 10.1186/1748-5908-5-77. 10.1186/1748-5908-5-77 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Siegel RM, Bien J, Lichtenstein P, Davis J, Khoury JC, Knight JE, et al. A safety-net antibiotic prescription for otitis media: the effects of a PBRN study on patients and practitioners. Clin Pediatr. 2006;45:518–24. 10.1177/0009922806290567. 10.1177/0009922806290567 [DOI] [PubMed] [Google Scholar]
- 37.Warnecke R, Johnson T, Kaluzny A, Ford L. The community clinical oncology program: its effect on clinical practice. Jt Comm J Qual Improv. 1995;21:336–9. [DOI] [PubMed] [Google Scholar]
- 38.Newington L, Wells M, Adonis A, Bolton L, Bolton Saghdaoui L, et al. A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine. BMC Health Serv Res. 2021;21:400. 10.1186/s12913-021-06354-y. 10.1186/s12913-021-06354-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Chalmers S, Hill J, Connell L, Ackerley S, Kulkarni A, Roddam H. The value of allied health professional research engagement on healthcare performance: a systematic review. BMC Health Serv Res. 2023;23:766. 10.1186/s12913-023-09555-9. 10.1186/s12913-023-09555-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Shahian DM, McCloskey D, Liu X, Schneider E, Cheng D, Mort EA. The association of hospital research publications and clinical quality. Health Serv Res. 2022;57(3):587–97. 10.1111/1475-6773.13947. 10.1111/1475-6773.13947 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Garner P, Hopewell S, Chandler J, MacLehose H, Akl EA, Bayene J, et al. When and how to update systematic reviews: consensus and checklist. BMJ. 2016;354: i3507. 10.1136/bmj.i3507. 10.1136/bmj.i3507 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Bramer WM, De Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc. 2018;106:531–41. 10.1136/bmj.i3507. 10.1136/bmj.i3507 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, et al. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10:e1001419. 10.1371/journal.pmed.1001419. 10.1371/journal.pmed.1001419 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Sys Rev. 2021;10:39. 10.1186/s13643-020-01542-z. 10.1186/s13643-020-01542-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Info Libr J. 2009;26:126–35. 10.1111/j.1471-1842.2008.00785.x. 10.1111/j.1471-1842.2008.00785.x [DOI] [PubMed] [Google Scholar]
- 46.Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—A web and mobile app for systematic reviews. Syst Rev. 2016;5:210. 10.1186/s13643-016-0384-4. 10.1186/s13643-016-0384-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Hong QN, Pluye P, Fàbregues S, Bartlett G, et al. The mixed methods appraisal tool (MMAT) version 2018. User guide. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf. Accessed 7 July 2024.
- 48.Dixon-Woods M, Cavers D, Agarwal S, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35. 10.1186/1471-2288-6-35. 10.1186/1471-2288-6-35 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Abraham AJ, Knudsen HK, Roman PM. The relationship between clinical trial network protocol involvement and quality of substance use disorder treatment. J Subst Abuse Treat. 2014;46:232–7. 10.1016/j.jsat.2013.08.021. 10.1016/j.jsat.2013.08.021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Alotaibi NM, Ibrahim GM, Wang J, Guha D, Mamdani M, Schweizer TA, et al. Neurosurgeon academic impact is associated with clinical outcomes after clipping of ruptured intracranial aneurysms. PLoS ONE. 2017;12: e0181521. 10.1371/journal.pone.0181521. 10.1371/journal.pone.0181521 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Bennett WO, Bird JH, Burrows SA, Counter PR, Reddy VM. Does academic output correlate with better mortality rates in NHS trusts in England? Public Health. 2012;126(Suppl 1):S40–3. 10.1016/j.puhe.2012.05.021. 10.1016/j.puhe.2012.05.021 [DOI] [PubMed] [Google Scholar]
- 52.Birkmeyer NJ, Goodney PP, Stukel TA, Hillner BE, Birkmeyer JD. Do cancer centers designated by the National Cancer Institute have better surgical outcomes? Cancer. 2005;103(3):435–41. 10.1002/cncr.20785. 10.1002/cncr.20785 [DOI] [PubMed] [Google Scholar]
- 53.Brown A, Griffiss M. Effect of integrated research programs on health care systems and costs. Mil Med. 1996;161:691–5. 10.1093/milmed/161.11.691. 10.1093/milmed/161.11.691 [DOI] [PubMed] [Google Scholar]
- 54.Carpenter WR, Meyer AM, Wu Y, Qaqish B, Sanoff HK, Goldberg RM, et al. Translating research into practice: the role of provider-based research networks in the diffusion of an evidence-based colon cancer treatment innovation. Med Care. 2012;50:737–48. 10.1097/MLR.0b013e31824ebe13. 10.1097/MLR.0b013e31824ebe13 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Corrigan MH, Glass HE. Physician participation in clinical studies and subsequent prescribing of new drugs. Pharm Ther. 2005;30(1):60–6. [Google Scholar]
- 56.de Arriba-Enriquez J, Sanz-Casado E, Vieta E, Rapado-Castro M, Arango C. Quality of care in psychiatry is related to research activity. Eur Psychiatry. 2021;64(1): e53. 10.1192/j.eurpsy.2021.16. 10.1192/j.eurpsy.2021.16 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.de Lange TS, Roos C, Bloemenkamp KW, Bolte AC, Duvekot JJ, Franssen MT, et al. Impact of a randomized trial on maintenance tocolysis on length of hospital admission of women with threatened preterm labor in the Netherlands. Eur J Obstet Gynecol Reprod Biol. 2015;186:8–11. 10.1016/j.ejogrb.2014.12.003. 10.1016/j.ejogrb.2014.12.003 [DOI] [PubMed] [Google Scholar]
- 58.Downing A, Morris EJ, Corrigan N, Sebag-Montefiore D, Finan PJ, Thomas JD, et al. High hospital research participation and improved colorectal cancer survival outcomes: a population-based study. Gut. 2017;66:89–96. 10.1136/gutjnl-2015-311308. 10.1136/gutjnl-2015-311308 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Eaton BR, Pugh SL, Bradley JD, Masters G, Kavadi VS, Narayan S, et al. Institutional enrollment and survival among NSCLC patients receiving chemoradiation: NRG oncology radiation therapy oncology group (RTOG) 0617. J Natl Cancer Inst. 2016;108(9): djw034. 10.1093/jnci/djw034. 10.1093/jnci/djw034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Fanaroff AC, Vora AN, Chen AY, Mathews R, Udell JA, Roe MT, et al. Hospital participation in clinical trials for patients with acute myocardial infarction: results from the National Cardiovascular Data Registry. Am Heart J. 2019;214:184–93. 10.1016/j.ahj.2019.05.011. 10.1016/j.ahj.2019.05.011 [DOI] [PubMed] [Google Scholar]
- 61.Farquhar DR, Masood MM, Lenze NR, Sheth S, Patel SN, Lumley C, et al. Academic affiliation and surgical volume predict survival in head and neck cancer patients receiving surgery. Laryngoscope. 2021;131:E479–88. 10.1002/lary.28744. 10.1002/lary.28744 [DOI] [PubMed] [Google Scholar]
- 62.Fernández-Domínguez JC, De Pedro-Gómez JE, Jiménez-López R, Romero-Franco N, Bays Moneo AB, Oliva-Pascual-Vaca Á, et al. Physiotherapists’ evidence-based practice profiles by HS-EBP questionnaire in Spain: a cross-sectional normative study. PLoS ONE. 2022;17(6): e0269460. 10.1371/journal.pone.0269460. 10.1371/journal.pone.0269460 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Fields D, Knudsen HK, Roman PM. Implementation of network for the improvement of addiction treatment (NIATx) processes in substance use disorder treatment centers. J Behav Health Serv Res. 2016;43:354–65. 10.1007/s11414-015-9466-7. 10.1007/s11414-015-9466-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Fong ZV, Chang DC, Hur C, et al. Variation in long-term oncologic outcomes by type of cancer center accreditation: an analysis of a SEER-Medicare population with pancreatic cancer. Am J Surg. 2020;220(1):29–34. 10.1016/j.amjsurg.2020.03.035. 10.1016/j.amjsurg.2020.03.035 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.García-Romero A, Escribano Á, Tribó JA. The impact of health research on length of stay in Spanish public hospitals. Res Policy. 2017;46:591–604. 10.1016/j.respol.2017.01.006. 10.1016/j.respol.2017.01.006 [DOI] [Google Scholar]
- 66.Gilbert GH, Gordan VV, Funkhouser EM, Rindal DB, Fellows JL, Qvist V, et al. Caries treatment in a dental practice-based research network: movement toward stated evidence-based treatment. Community Dent Oral Epidemiol. 2013;41:143–53. 10.1111/cdoe.12008. 10.1111/cdoe.12008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Jonker L, Fisher SJ. NHS Trusts’ clinical research activity and overall CQC performance—Is there a correlation? Public Health. 2015;129:1491–5. 10.1016/j.puhe.2015.07.026. 10.1016/j.puhe.2015.07.026 [DOI] [PubMed] [Google Scholar]
- 68.Jonker L, Fisher SJ. The correlation between National Health Service trusts’ clinical trial activity and both mortality rates and care quality commission ratings: a retrospective cross-sectional study. Public Health. 2018;157:1–6. 10.1016/j.puhe.2017.12.022. 10.1016/j.puhe.2017.12.022 [DOI] [PubMed] [Google Scholar]
- 69.Jonker L, Fisher SJ, Badgett RG. Relationship between staff thriving, through engagement and research activity, and hospital-related outcome measures: a retrospective cross-sectional study. J Healthc Qual Res. 2021;36(3):128–35. 10.1016/j.jhqr.2021.02.002 [DOI] [PubMed] [Google Scholar]
- 70.Kirby K, Brandling J, Robinson M, Thomas M, Voss S, Benger J. The experiences of EMS providers taking part in a large randomised trial of airway management during out of hospital cardiac arrest, and the impact on their views and practice. Results of a survey and telephone interviews. Resuscitation. 2020;149:1–9. 10.1016/j.resuscitation.2020.01.034. 10.1016/j.resuscitation.2020.01.034 [DOI] [PubMed] [Google Scholar]
- 71.Kirwan CC, Al Sarakbi W, Loncaster J, Chan HY, Thompson AM, Wishart GC. Tumour bed clip localisation for targeted breast radiotherapy: compliance is proportional to trial-related research activity: tumour bed clip localisation in breast radiotherapy. Eur J Surg Oncol. 2014;40:158–62. 10.1016/j.ejso.2013.11.016. 10.1016/j.ejso.2013.11.016 [DOI] [PubMed] [Google Scholar]
- 72.Levan JM, Brion LP, Wrage LA, Gantz MG, Wyckoff MH, Sánchez PJ, et al. Change in practice after the surfactant, positive pressure and oxygenation randomised trial. Arch Dis Child Fetal Neonatal Ed. 2014;99:F386–90. 10.1136/archdischild-2014-306057. 10.1136/archdischild-2014-306057 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.Lin DJ, McConkey CC, Nankivell P, Dunn J, Mehanna H. The impact of institutional clinical trial recruitment versus hospital volume on survival outcomes of patients with head and neck cancer: an analysis of the PET-NECK trial outcomes, UKCRN portfolio, and Hospital Episode Statistics (HES) in England. Oral Oncol. 2018;85:40–3. 10.1016/j.oraloncology.2018.08.006. 10.1016/j.oraloncology.2018.08.006 [DOI] [PubMed] [Google Scholar]
- 74.Litjens RJ, Oude Rengerink K, Danhof NA, Kruitwagen RF, Mol BW. Does recruitment for multicenter clinical trials improve dissemination and timely implementation of their results? A survey study from the Netherlands. Clin Trials. 2013;10:915–23. 10.1177/1740774513504150. 10.1177/1740774513504150 [DOI] [PubMed] [Google Scholar]
- 75.Ljunggren M, Weibull CE, Rosander E, et al. Hospital factors and metastatic surgery in colorectal cancer patients, a population-based cohort study. BMC Cancer. 2022;22:907. 10.1186/s12885-022-10005-8. 10.1186/s12885-022-10005-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Manes E, Tchetchik A, Tobol Y, Durst R, Chodick G. An empirical investigation of “physician congestion” in US University Hospitals. Int J Environ Res Public Health. 2019;16:761. 10.3390/ijerph16050761. 10.3390/ijerph16050761 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 77.Marmor S, Begun J, Abraham J, et al. The impact of center accreditation on hematopoietic cell transplantation (HCT). Bone Marrow Transplant. 2015;50:87–94. 10.1038/bmt.2014.219. 10.1038/bmt.2014.219 [DOI] [PubMed] [Google Scholar]
- 78.McBride R, Leroux B, Lindblad A, Williams OD, Lehmann M, Rindal DB, et al. Measuring the impact of practice-based research networks on member dentists in the Collaboration on Networked Dental and Oral Health Research, CONDOR. J Dent. 2013;41:393–403. 10.1016/j.jdent.2013.03.005. 10.1016/j.jdent.2013.03.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 79.McCarthy FH, Groeneveld PW, Kobrin D, Mcdermott KM, Wirtalla C, Desai ND. Effect of clinical trial experience on transcatheter aortic valve replacement outcomes. Circ Cardiovasc Interv. 2015;8: e002234. 10.1161/CIRCINTERVENTIONS.114.002234. 10.1161/CIRCINTERVENTIONS.114.002234 [DOI] [PubMed] [Google Scholar]
- 80.McDaniels-Davidson C, Feng CH, Martinez ME, Canchola AJ, Gomez SL, Nodora JN, et al. Improved survival in cervical cancer patients receiving care at National Cancer Institute—Designated cancer centers. Cancer. 2022;128:3479–86. 10.1002/cncr.34404. 10.1002/cncr.34404 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Merkow RP, Chung JW, Paruch JL, Bentrem DJ. Center accreditation and performance on publicly reported quality measures. Ann Surg. 2014;259(6):1091–7. 10.1097/SLA.0000000000000542. 10.1097/SLA.0000000000000542 [DOI] [PubMed] [Google Scholar]
- 82.Meyer AM, Reeder-Hayes KE, Liu H, Wheeler SB, Penn D, Weiner BJ, et al. Differential receipt of sentinel lymph node biopsy within practice-based research networks. Med Care. 2013;51:812–8. 10.1097/MLR.0b013e31829c8ca4. 10.1097/MLR.0b013e31829c8ca4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 83.Mold JW, Aspy CB, Smith PD, Zink T, Knox L, Lipman PD, et al. Leveraging practice-based research networks to accelerate implementation and diffusion of chronic kidney disease guidelines in primary care practices: a prospective cohort study. Implement Sci. 2014;9:169. 10.1186/s13012-014-0169-x. 10.1186/s13012-014-0169-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Murimwa GZ, Karalis JD, Meier J, et al. Hospital designations and their impact on guideline-concordant care and survival in pancreatic cancer. Do they matter? Ann Surg Oncol. 2023;30:4377–87. 10.1245/s10434-023-13308-7. 10.1245/s10434-023-13308-7 [DOI] [PubMed] [Google Scholar]
- 85.Ozdemir BA, Karthikesalingam A, Sinha S, Poloniecki JD, Hinchliffe RJ, Thompson MM, et al. Research activity and the association with mortality. PLoS ONE. 2015;10: e0118253. 10.1371/journal.pone.0118253. 10.1371/journal.pone.0118253 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Paulson EC, Mitra N, Sonnad S, Armstrong K, Wirtalla C, Kelz RR, et al. National cancer institute designation predicts improved outcomes in colorectal cancer surgery. Ann Surg. 2008;248:675–86. 10.1097/SLA.0b013e318187a757. 10.1097/SLA.0b013e318187a757 [DOI] [PubMed] [Google Scholar]
- 87.Penn DC, Chang Y, Meyer AM, Defilippo Mack C, Sanoff HK, Stitzenberg KB, et al. Provider-based research networks may improve early access to innovative colon cancer treatment for African Americans treated in the community. Cancer. 2015;121:93–101. 10.1002/cncr.29028. 10.1002/cncr.29028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 88.Prendergast EA, Perkins S, Engel ME, Cupido B, Francis V, Joachim A, et al. Participation in research improves overall patient management: insights from the global rheumatic heart disease registry (REMEDY). Cardiovasc J Afr. 2018;29:98–105. 10.5830/CVJA-2017-054. 10.5830/CVJA-2017-054 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 89.Rai A, Nastoupil LJ, Williams JN, Lipscomb J, Ward KC, Howard DH, et al. Patterns of use and survival outcomes of positron emission tomography for initial staging in elderly follicular lymphoma patients. Leuk Lymphoma. 2017;58:1570–80. 10.1080/10428194.2016.1253836. 10.1080/10428194.2016.1253836 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 90.Rieckmann TR, Abraham AJ, Bride BE. Implementation of motivational interviewing in substance use disorder treatment: research network participation and organizational compatibility. J Addict Med. 2016;10:402–7. 10.1097/ADM.0000000000000251. 10.1097/ADM.0000000000000251 [DOI] [PubMed] [Google Scholar]
- 91.Rieckmann TR, Abraham AJ, Kovas AE, Mcfarland BH, Roman PM. Impact of research network participation on the adoption of buprenorphine for substance abuse treatment. Addict Behav. 2014;39:889–96. 10.1016/j.addbeh.2014.01.016. 10.1016/j.addbeh.2014.01.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Rindal DB, Flottemesch TJ, Durand EU, Godlevsky OV, Schmidt AM, Gilbert GH. Practice change toward better adherence to evidence-based treatment of early dental decay in the National Dental PBRN. Implement Sci. 2014;9:177. 10.1186/s13012-014-0177-x. 10.1186/s13012-014-0177-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 93.Robinson TG, Wang X, Durham AC, Ford GA, Liao J, Littlewood S, et al. The National Institute for Health Research Hyperacute Stroke Research Centres and the ENCHANTED trial: the impact of enhanced research infrastructure on trial metrics and patient outcomes. Health Res Policy Syst. 2019;17:19. 10.1186/s12961-019-0417-2. 10.1186/s12961-019-0417-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 94.Rochon J, Du Bois A, Lange T. Mediation analysis of the relationship between institutional research activity and patient survival. BMC Med Res Methodol. 2014;14:9. 10.1186/1471-2288-14-9. 10.1186/1471-2288-14-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 95.Salge TO, Vera A. Hospital innovativeness and organizational performance: evidence from English public acute care. Health Care Manag Rev. 2009;34:54–67. 10.1097/01.HMR.0000342978.84307.80. 10.1097/01.HMR.0000342978.84307.80 [DOI] [PubMed] [Google Scholar]
- 96.Seaburg LA, Wang AT, West CP, Reed DA, Halvorsen AJ, Engstler G, et al. Associations between resident physicians’ publications and clinical performance during residency training. BMC Med Educ. 2016;16:22. 10.1186/s12909-016-0543-2. 10.1186/s12909-016-0543-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97.Shen M, Liang X, Li L, Wu Y, Yang Y, Zingg R. The association of attending physicians’ publications and patients’ readmission rates: evidence from tertiary hospitals in china using a retrospective data analysis. Int J Environ Res Public Health. 2022;19(15):9760. 10.3390/ijerph19159760. 10.3390/ijerph19159760 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 98.Siracuse JJ, Goodney PP, Menard MT, Rosenfield K, Van Over M, Hamza T, et al. Participation in a chronic limb threatening ischemia randomized trial is inversely correlated with regional amputation rate in limb threatening ischemia patients. Ann Surg. 2021;274(4):621–6. 10.1097/SLA.0000000000005058. 10.1097/SLA.0000000000005058 [DOI] [PubMed] [Google Scholar]
- 99.Tan HJ, Meyer AM, Kuo TM, Smith AB, Wheeler SB, Carpenter WR, et al. Provider-based research networks and diffusion of surgical technologies among patients with early-stage kidney cancer. Cancer. 2015;121:836–43. 10.1002/cncr.29144. 10.1002/cncr.29144 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 100.Tchetchik A, Grinstein A, Manes E, Shapira D, Durst R. From research to practice: which research strategy contributes more to clinical excellence? Comparing high-volume versus high-quality biomedical research. PLoS ONE. 2015;10: e0129259. 10.1371/journal.pone.0129259. 10.1371/journal.pone.0129259 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.Trusson D, Rowley E, Bramley L. A mixed-methods study of challenges and benefits of clinical academic careers for nurses, midwives and allied health professionals. BMJ Open. 2019;9: e030595. 10.1136/bmjopen-2019-030595. 10.1136/bmjopen-2019-030595 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.Tsang Y, Ciurlionis L, Kirby AM, Locke I, Venables K, Yarnold JR, et al. Clinical impact of IMPORT HIGH trial (CRUK/06/003) on breast radiotherapy practices in the United Kingdom. Br J Radiol. 2015;88:20150453. 10.1259/bjr.20150453. 10.1259/bjr.20150453 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 103.Tsang JLY, Binnie A, Duan EH, Johnstone J, Heels-Ansdell D, Reeve B. Academic and community ICUs participating in a critical care randomized trial: a comparison of patient characteristics and trial metrics. Crit Care Explor. 2022;4(11): e0794. 10.1097/CCE.0000000000000794. 10.1097/CCE.0000000000000794 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Van Der Tuuk K, Koopmans CM, Groen H, Mol BW, Van Pampus MG. Impact of the HYPITAT trial on doctors’ behaviour and prevalence of eclampsia in the Netherlands. BJOG. 2011;118:1658–60. 10.1111/j.1471-0528.2011.03138.x. 10.1111/j.1471-0528.2011.03138.x [DOI] [PubMed] [Google Scholar]
- 105.Venables K, Tsang Y, Ciurlionis L, Coles CE, Yarnold JR. Does participation in clinical trials influence the implementation of new techniques? A look at changing techniques in breast radiotherapy in the UK. Clin Oncol (R Coll Radiol). 2012;24:e100–5. 10.1016/j.clon.2012.06.010. 10.1016/j.clon.2012.06.010 [DOI] [PubMed] [Google Scholar]
- 106.Wolfson JA, Sun CL, Wyatt LP, Hurria A, Bhatia S. Impact of care at comprehensive cancer centers on outcome: results from a population-based study. Cancer. 2015;121(21):3885–93. 10.1002/cncr.29576. 10.1002/cncr.29576 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 107.Wuthrick EJ, Zhang Q, Machtay M, Rosenthal DI, Nguyen-Tan PF, Fortin A, et al. Institutional clinical trial accrual volume and survival of patients with head and neck cancer. J Clin Oncol. 2015;33(2):156–64. 10.1200/JCO.2014.56.5218. 10.1200/JCO.2014.56.5218 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 108.Yawn BP, Pace W, Dietrich A, Bertram S, Kurland M, Graham D, et al. Practice benefit from participating in a practice-based research network study of postpartum depression: a national research network (NRN) report. J Am Board Fam Med. 2010;23:455–64. 10.3122/jabfm.2010.04.090246. 10.3122/jabfm.2010.04.090246 [DOI] [PubMed] [Google Scholar]
- 109.Young C, Gunasekera H, Kong K, Purcell A, Muthayya S, Vincent F, et al. A case study of enhanced clinical care enabled by Aboriginal health research: the Hearing, EAr health and Language Services (HEALS) project. Aust N Z J Public Health. 2016;40:523–8. 10.1111/1753-6405.12586. 10.1111/1753-6405.12586 [DOI] [PubMed] [Google Scholar]
- 110.Rogers E. Diffusion of innovations. 5th ed. New York: Free Press; 2003. [Google Scholar]
- 111.Cohen WM, Levinthal DA. Absorptive capacity: a new perspective on learning and innovation. Adm Sci Q. 1990;35(1):128–52. 10.2307/2393553 [DOI] [Google Scholar]
- 112.Zahra SA, George G. Absorptive capacity: a review, reconceptualization, and extension. Acad Manag Rev. 2002;27:185–203. 10.5465/AMR.2002.6587995. 10.5465/AMR.2002.6587995 [DOI] [Google Scholar]
- 113.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. 10.1186/1748-5908-4-50. 10.1186/1748-5908-4-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 114.Roman PM, Abraham AJ, Rothrauff TC, Knudsen HK. A longitudinal study of organizational formation, innovation adoption, and dissemination activities within the national drug abuse treatment clinical trials network. J Subst Abuse Treat. 2010;38(Suppl 1):S44-52. 10.1016/j.jsat.2009.12.008. 10.1016/j.jsat.2009.12.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 115.Minasian LM, Carpenter WR, Weiner BJ, Anderson DE, McCaskill-Stevens W, Nelson S, et al. Translating research into evidence-based practice: the National Cancer Institute Community Clinical Oncology Program. Cancer. 2010;116:4440–9. 10.1002/cncr.25248. 10.1002/cncr.25248 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 116.Fennell M, Warneke R. The diffusion of medical innovations: an applied network analysis: environment, development, and public policy and social services. Boston: Springer; 1988. 10.1007/978-1-4684-5436-9. [Google Scholar]
- 117.Peckham S, Eida T, Hashem F, Kendall S. Research engagement and research capacity building: a priority for healthcare organisations in the UK. J Health Organ Manag. 2023;37(3):343–59. 10.1108/JHOM-12-2021-0436. 10.1108/JHOM-12-2021-0436 [DOI] [PubMed] [Google Scholar]
- 118.Edelman A, Brown A, Pain T, Larkins S, Harvey G. Evaluating research investment and impact at a regional Australian hospital and health service: a programme theory and conceptual framework. Health Res Policy Sys. 2020;18:30. 10.1186/s12961-020-0542-y. 10.1186/s12961-020-0542-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 119.Brown A, Edelman A, Pain T, Larkins S, Harvey G. “We’re not providing the best care if we are not on the cutting edge of research”: a research impact evaluation at a regional Australian hospital and health service. Int J Health Policy Manag. 2022;11(12):3000–11. 10.34172/IJHPM.2022.6529. 10.34172/IJHPM.2022.6529 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 120.San Francisco declaration on research assessment. https://sfdora.org/. Accessed 18 Apr 2024.
- 121.Ailawadhi S, Advani P, Yang D, Ghosh R, Swaika A, et al. Impact of access to NCI- and NCCN-designated cancer centers on outcomes for multiple myeloma patients: a SEER registry analysis. Cancer. 2016;122:618–25. 10.1002/cncr.29771. 10.1002/cncr.29771 [DOI] [PubMed] [Google Scholar]
- 122.Tsilimigras DI, Hyer JM, Diaz A, Moris D, Abbas A, Dillhoff M, et al. Impact of cancer center accreditation on outcomes of patients undergoing resection for hepatocellular carcinoma: a SEER-Medicare analysis. Am J Surg. 2021;222(3):570–6. 10.1016/j.amjsurg.2021.01.015. 10.1016/j.amjsurg.2021.01.015 [DOI] [PubMed] [Google Scholar]
- 123.Grant J, Buxton MJ. Economic returns to medical research funding. BMJ Open. 2018;8: e022131. 10.1136/bmjopen-2018-022131. 10.1136/bmjopen-2018-022131 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 124.Schlick CJ, Yang AD. Is there value in cancer center accreditation? Am J Surg. 2023;220(1):27–8. 10.1016/j.amjsurg.2020.05.001. 10.1016/j.amjsurg.2020.05.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 125.Hussein M, Pavlova M, Ghalwash M, Groot W. The impact of hospital accreditation on the quality of healthcare: a systematic literature review. BMC Health Serv Res. 2021;21:1057. 10.1186/s12913-021-07097-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 126.Yoong SL, Bolsewicz K, Reilly K, et al. Describing the evidence-base for research engagement by health care providers and health care organisations: a scoping review. BMC Health Serv Res. 2023;23:75. 10.1186/s12913-022-08887-2. 10.1186/s12913-022-08887-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 127.Soper B, Yaqub O, Hinrichs S, Marjanovich S, Drabble S, Hanney S, et al. CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation. J Health Serv Res Policy. 2013;18:53–64. 10.1177/1355819613499903. 10.1177/1355819613499903 [DOI] [PubMed] [Google Scholar]
- 128.Kislov R, Wilson PM, Knowles S, Boaden R. Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations. Implement Sci. 2018;13:111. 10.1186/s13012-018-0805-y. 10.1186/s13012-018-0805-y [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
No datasets were generated or analysed during the current study.