Evaluation of funding gastroenterology research in Canada illustrates the beneficial role of partnerships

Abstract

BACKGROUND:

Funders of health research in Canada seek to determine how their funding programs impact research capacity and knowledge creation.

OBJECTIVE:

To evaluate the impact of a focused grants and award program that was cofunded by the Canadian Institutes of Health Research Institute of Nutrition, Metabolism and Diabetes, and the Canadian Association of Gastroenterology; and to measure the impact of the Program on the career paths of funded researchers and assess the outcomes of research supported through the Program.

METHODS:

A survey of the recipients of grants and awards from 2000 to 2008 was conducted in 2012. The CIHR Funding Decisions database was searched to determine subsequent funding; a bibliometric citation analysis of publications arising from the Program was performed.

RESULTS:

Of 160 grant and award recipients, 147 (92%) completed the survey. With >$17.4 million in research funding, support was provided for 131 fellowship awards, seven career transition awards, and 22 operating grants. More than three-quarters of grant and award recipients continue to work or train in a research-related position. Combined research outputs included 545 research articles, 130 review articles, 33 book chapters and 11 patents. Comparative analyses indicate that publications supported by the funding program had a greater impact than other Canadian and international comparators.

CONCLUSIONS:

Continuity in support of a long-term health research funding partnership strengthened the career development of gastroenterology researchers in Canada, and enhanced the creation and dissemination of new knowledge in the discipline.

The Canadian Institutes of Health Research (CIHR), the Canadian Association of Gastroenterology (CAG), a national health charity (Crohn’s and Colitis Foundation of Canada) and industry partners have partnered to provide funding support for trainees, new researchers, and established investigators through an annual research support program, the CAG/CIHR/Partners Program (‘the Program’). The objective of the Program was to build capacity in gastroenterology research in Canada.

Originally launched in 1992, the CAG/CIHR/Partners Program has evolved from a single focus on supporting fellowship awards to include New Investigator operating grants, Career Transition awards, and operating grants, thereby providing support for gastroenterology research across the career trajectory. Operating grants and New Investigator operating grants provide up to two years of operating funds to support gastroenterology-related research proposals. Career Transition awards were a unique program providing a one-year Fellowship award followed by a two year New Investigator Salary award. Fellowship awards provide support for highly qualified post-PhD or post-health professional degree candidates to train in health research within Canada or abroad. Since 2000, the majority of grants and greatest portion of Program funds have supported Fellowship awards.

An evaluation of the first 10 years of the CAG/CIHR/Partners Program, conducted in 2002, concluded that it was meeting the objective of building capacity in gastroenterology research in Canada (1). The aim of the present updated evaluation, conducted in 2012 using data from 2000 to 2008, was to critically assess the contributions to research and training in gastroenterology in Canada generated through the CAG/CIHR/Partners Program. The specific objectives of the present evaluation were to measure the impact of funding on recipient career paths, and assess whether funding increased gastroenterology research output and impact in Canada.

METHODS

All recipients of Fellowship awards, Operating grants and Career Transition awards who received support from 2000 to 2008 were surveyed to assess the impact of the Program on their careers and to evaluate the impact of the Program on building capacity in gastroenterology research in Canada. The CAG prepared the survey in consultation with the CIHR Institute of Nutrition, Metabolism, and Diabetes (CIHR-INMD), and the CIHR Evaluation Unit. Using contact information for the grant and award recipients originating from the CAG membership database, the CAG circulated the survey to 160 recipients. Respondents predominantly completed and submitted their surveys electronically, although some written responses were also received. One hundred forty-seven (92%) of 160 recipients responded to the survey.

CIHR staff conducted a search of the CIHR Funding Decisions database to determine subsequent success in obtaining CIHR funding for grants and awards as a nominated principal investigator. To determine the impact of the supported research, the CAG and CIHR commissioned the Observatoire des sciences et des technologies (OST, Montreal, Quebec) to conduct a bibliometric analysis of publications identified by survey respondents as directly resulting from CAG/CIHR/Partners funding. Of 693 publications submitted by survey respondents, 523 were included in this analysis. OST excluded 170 from the analysis because of incomplete data, duplication, lack of new scientific knowledge (eg, book reviews and editorial material) or noncoverage within the OST database.

OST conducted analyses to determine both the average of relative citations (ARC) and the average of relative impact factor (ARIF) (2–4). The ARC measures the frequency with which the research findings are cited by other researchers, while the ARIF indicates the prestige of the journal in which the research is published. OST conducted additional analysis on the five scientific subfields in which articles were most frequently published: gastroenterology, immunology, microbiology, pharmacology and physiology.

In addition to the ARC and ARIF, OST analyzed the rate of international collaboration among grant and award recipients. International collaboration is an indicator of the relative intensity of scientific collaboration between countries. An article was considered to be an international collaboration when it contained international addresses as well as Canadian addresses.

RESULTS

From 2000 to 2008, the CAG/CIHR/Partners Program was financially supported annually by CIHR through the Research and Development Program and CIHR-INMD, 13 industry partners and the Crohn’s and Colitis Foundation of Canada (Box 1). These partnerships resulted in an investment of >$17.4 million to support 160 awards and grants, including 22 operating grants (including New Investigator grants), seven Career Transition awards, and 131 Fellowship awards between 2000 and 2008. Annual funding for the Program varied between $1.25 million to $2.6 million, with more than $13.8 million invested in Fellowship awards over the nine-year period.

BOX 1.

Canadian Association of Gastroenterology/Canadian Institutes for Research Partners, 2000–2008 (in alphabetical order)

AbbVie

Aptalis

AstraZeneca

Bristol Myers Squibb

Crohn’s and Colitis Foundation of Canada

Ferring

GlaxoSmithKline

Janssen

Nestlé

Novartis

Olympus

PENTAX Medical

Solvay

Takeda

Participation by the 14 partners varied, with seven partners contributing to the Program for ≥8 years, and five partners contributing to the Program for one year. Collectively, the partners provided $8.7 million (50%) of Program funding between 2000 and 2008. Over the same period, CIHR invested $8.4 million (48%), through 1:1 matching funds. The remaining funds were provided by the CAG, universities or research institutes. In addition to the financial support of partners, substantial in-kind administrative contributions were made by both CIHR and the CAG, including recruitment and management of partners and program administration by the CAG, and peer review and grant administration by CIHR.

Although not specifically targeted by the Program, clinician scientists received strong support through the CAG/CIHR/Partners Program from 2000 to 2008. Data regarding professional designation was available in the CAG database for 158 of the 160 survey recipients. These data indicated that 56% of recipients held a PhD and the remaining 44% held an MD degree.

To assess whether the Program was successful in building research capacity, as defined by a continued career in a research environment, past recipients identified their current work position and whether they continued to perform research in their present role. Including current trainees, 76% of 147 survey respondents currently hold a research position (Figure 1A). Of funding recipients no longer working in a research environment, slightly more than 15% hold positions as clinical teachers in an academic setting or as clinicians in community practice.

Figure 1).

A Of Canadian Association of Gastroenterology/Canadian Institiutes of Health Research/Partners Program funding recipients that received funding between 2000 and 2008, 76% continue to work in research positions (including trainee positions). B Seventy-two per cent of Fellowship Award recipients continued to work in a research position (including trainee positions)

Because Fellowship award recipients represent the group that is most vulnerable to leaving the research environment at the end of their award, additional data were generated to assess whether receipt of a Fellowship award resulted in a continued career in a research environment. Of the 119 survey respondents who received Fellowship award support, 74 (62%) were currently in a research position and 12 (10%) were completing a subsequent fellowship, resulting in a research retention rate of 72% (Figure 1B). Of Fellowship award recipients not currently in a research position, 18% hold positions as clinical teachers or as clinicians in community practice.

To determine whether the Program succeeded in increasing gastroenterology research capacity in Canada, current work location of funding recipients were assessed. Of the 150 recipients whose current location was known through either their response to the survey or the CAG database, 71% are situated in Canada (Figure 2A). Of the recipients who responded to the survey, 54% are currently in Canada and hold a research position. Of the 122 Fellowship award recipients whose current location is known, 66% are based in Canada (Figure 2B). Another indication of the continuation of the recipients’ research careers in Canada is their success in subsequently obtaining CIHR grants and awards funding. The 160 recipients surveyed subsequently held a total of 120 CIHR grants and awards.

Figure 2).

A Of Canadian Association of Gastroenterology/Canadian Institiutes of Health Research/Partners Program funding recipients that received funding between 2000 and 2008, 71% currently work in Canada. B Sixty-six per cent of Fellowship Award recipients currently work in Canada

To address the second objective of the present evaluation, the research output of the funding recipients was assessed. Survey respondents indicated that 545 research articles, 130 review articles, 35 book chapters and 11 patents resulted directly from Program funding (Figure 3).

Figure 3).

Canadian Association of Gastroenterology/Canadian Institiutes of Health Research/Partners Program funding from 2000 to 2008 resulted in the production of 710 publications and 11 patents

To determine the impact of the research supported by the Program, a bibliometric analysis on the publications that survey respondents identified as resulting directly from Program funding was performed. ARC analysis showed that Program-supported articles received substantially more citations than the world average for articles published in the same five subfields of research. The overall ARC for Program-supported articles was 1.73, compared with 1.00 for the rest of the world and 1.18 for Canada (Figure 4A), indicating that Program-supported articles had greater scientific impact than other publications by peers both in Canada and around the world.

Figure 4).

A Bibliometric analysis indicating that publications by Canadian Association of Gastroenterology (CAG)/Canadian Institiutes of Health Research (CIHR)/Partners Program recipients were cited more frequently than the worldwide and Canadian averages for similar articles in the five scientific subfields (gastroenterology, immunology, physiology, pharmacology and microbiology). B Analysis of average of relative impact factors of program-supported articles indicates that publications by CAG/CIHR/Partners Program recipients were more impactful compared with Canadian averages for similar articles in the five scientific subfields

ARIF scores indicate that Program-supported researchers published in journals with higher impact compared with other Canadian researchers and their peers in the rest of the world (Figure 4B).

Analysis of author addresses on Program-supported publications revealed that approximately 42% of the publications were the product of international collaborations. This is consistent with the international collaboration rate of publications by Canadian researchers within the five subfields of research, and it indicates potential for increased dissemination and translation of research results.

Survey respondents were also asked to identify disease areas studied in the research that was funded by the CAG/CIHR/Partners Program. The most common disease area studied was inflammatory bowel disease (Crohn disease and ulcerative colitis) (28% [n=73]). Other research areas studied included physiology of intestine (17% [n=42]), motility disorders (8% [n=20]), cancer (7% [n=19]), peptic ulcer disease (5% [n=14]), nutrition (5% [n=13]), liver (5% [n=12]), biliary tract (2% [n=5]), gastrointestinal reflux disease (2% [n=5]), obesity (2% [n=5]), pancreas (2% [n=4]) and anorectal disorders (one researcher). Seventeen per cent of recipients (n=43) indicated ‘other’ as the category for their research, which included research areas such as inflammation, infection, Clostridium difficile and endoscopy.

DISCUSSION

The results of the present evaluation demonstrate that CAG/CIHR/Partners Program funding has a significant positive impact on gastroenterology research in Canada and on recipient careers.

CAG/CIHR/Partners Program funding recipients produced an impressive number of publications and patents. Bibliometric analysis indicates that publications by Program-funded researchers were cited significantly more often than publications by peers around the world and in Canada, and that Program-funded publications were significantly more likely to appear in prestigious journals than publications by peers around the world and in Canada. Together, these data indicate that Program-funded researchers consistently produce excellent research. It is important to note that Canada is home to some of the most impactful gastroenterology research in the world. According to The State of Science and Technology in Canada, 2012 report, the ARC rank for Canadian citations in the subfields of gastroenterology and hepatology was second in the world. While it is tempting to conclude that the high Canadian ARC ranking may be due, in part, to funding support from the CAG/CIHR/Partners Program, our methodology does not permit attribution.

Previous evaluations on three partnered capacity-building programs, the Kidney Research Scientist Core Education and National Training (KRESCENT) Program, the Focus on Stroke Program and the Children’s Digestive Health and Nutrition Foundation (CDHNF) Young Investigator Development Awards, suggest that these programs positively impact researcher retention. The KRESCENT program, which is a partnership of CIHR-INMD, the Kidney Foundation of Canada, and the Canadian Society of Nephrology, supports Fellowships and New Investigator awards. In 2010, an evaluation of the KRESCENT program reported that 88% of KRESCENT Fellowship recipients subsequently secured an assistant professor position (5). Focus on Stroke, which was led by the Heart and Stroke Foundation of Canada in partnership with CIHR and industry, supported doctoral research awards, postdoctoral Fellowship awards and New Investigator awards. A 2012 study of Focus on Stroke revealed that 69% of funding recipients continued to work at a university or college as a researcher, academic, scientist or trainee (personal communication, C Evans, Heart and Stroke Foundation of Canada). The CDHNF Young Investigator Development Awards, which are led by CDHNF, in partnership with the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) and industry, fund Young Investigator and Career Transition awards. A 2010 evaluation showed that up to 94% of CDHNF grant recipients continued to work in a research position (6). The CAG/CIHR/Partners Program evaluation similarly indicates strong capacity building in gastroenterology research, with 72% of the 119 fellowship recipients continuing to work in a research position.

A potential limitation of the present study was the possibility of error in the self-reporting of research outputs. Publications were reviewed to reduce the risk that they could be attributed to other sources of funding concurrently held by the respondents. However, there was no verification of self-reported patents or book chapters. It should also be noted that it was necessary to use self-reported publications to perform program-focused bibliometric analyses, which represents a departure from traditional bibliometric citation analyses. Finally, because the survey did not ask Fellowship award recipients to report on subsequent peer-reviewed funding, such as national health charities, other federal granting agencies and international organizations, such data were not available for the present analysis.

An important aspect of the success of this program has been the continued partnership between CIHR, the CAG, voluntary health organizations and partners in industry. The contributions made by all partners helped build a solid and productive platform of support for a talented pool of new investigators. This allowed for continued support of the gastroenterology research community across Canada, which has resulted in increased research capacity and the generation of new knowledge with potential to impact greatly on the gastrointestinal health of Canadians.