Abstract
Background:
Health services and policy research is the innovation engine of a health care system. In 2000, the Canadian Institutes of Health Research (CIHR) was formed to foster the growth of all sciences that could improve health care. We evaluated trends in health services and policy research funding, in addition to determinants of funding success.
Methods:
All applications submitted to CIHR strategic and open operating grant competitions between 2001 and 2011 were included in our analysis. Age, sex, size of research team, critical mass, season, year and research discipline were retrieved from application information. A cohort of 4725 applicants successfully funded between 2001 and 2005 were followed for 5 years to evaluate predictors of continuous funding. Multivariate generalized estimating equation logistic regression was used to estimate predictors of funding success and sustained funding.
Results:
Between 2001 and 2011, 80 163 applications were submitted to open and strategic grant competitions. Over time, grant applications increased from 327 to 1137 per year, and annual funding increased from $12.6 to $48.0 million. Grant applications from young male researchers were more likely to be funded than those from female researchers (odds ratio [OR] 1.40, 95% confidence interval [CI] 1.01-1.95), as were applications from larger research teams and institutions with a large critical mass. Only 24.0% of scientists whose first funded grant was in health services and policy research had sustained 5-year funding, compared with 52.8% of biomedical scientists (OR 0.34, 95% CI 0.24-0.49).
Interpretation:
The CIHR has successfully increased the amount of health services and policy research in Canada. To enhance conditions for success, researchers should be encouraged to work in teams, request longer duration grants, resubmit unsuccessful applications and affiliate themselves with institutions with a greater critical mass.
Health services and policy research is the innovation engine of an effective health care system.1 Canada officially acknowledged the need for such research in 1969, when the federal government initiated the National Health Research Grant Program.2 Although the Medical Research Council and the Public Health Research and Development Program had already been established to support basic biomedical and communicable diseases research, it was recognized that a variety of factors influenced well-being that needed to be addressed through research on health system design and delivery supported by the new grant program. The farsightedness of this policy direction is echoed 4 decades later as countries grapple with the increasing prevalence of chronic disease,3 the need for interventions for lifestyle determinants of poor health4 and effective strategies to support chronic disease management.5
In 2000, the Medical Research Council and the National Health Research Grant Program were merged to form the Canadian Institutes of Health Research (CIHR) - an ambitious experiment to foster growth of all sciences that were key to improved health for Canadians, more effective health services and products, and a stronger Canadian health care system.6-8 The CIHR aimed to foster a new generation of interdisciplinary collaborative research through the creation of health research institutes, and by funding the spectrum of research disciplines: biomedical research, clinical research and research into health systems, health services, the health of populations, societal and cultural dimensions of health, and environmental influences on health. Health services and policy research was 1 of the 13 funding institutes at CIHR.6-11 Each institute manages a strategic allocation of funding that is used to address emerging priorities and gaps in science that are relevant to the health of Canadians from their respective areas of science, such as aging, genetics, Aboriginal health, and infection and immunity. However, most research funding is allocated to an open competition that aims to fund the best science and researchers by assessment of excellence by peers. Although the outcomes of this ambitious initiative are still unfolding, it is possible to assess if this new organization has influenced the funding of health services and policy research in Canada.
Our aim was to estimate funding trends in health services and policy research at CIHR compared with the biomedical, clinical and population health communities,12 the determinants of funding success in the open competition and the factors associated with sustained funding in the first cohort of successful applicants.
Methods
Design and data source
To assess funding trends and factors associated with funding success, we assembled a cohort of all applications submitted to CIHR strategic and open operating grant competitions between 2001 and 2011. Application data for operating grants were retrieved from the CIHR databases and deidentified. Each application included the sex and age of the applicant (in 5-year categories to protect confidentiality), the applicant's role in an operating grant (principal investigator, co-principal investigator, co-investigator), the applicant's institution, the self-reported pillar of the application (biomedical, clinical, health services and policy, population and public health), whether the application had been submitted previously to CIHR, whether it was a new application or a renewal, the assigned committee, amount and duration of funding requested, funding outcome, and amount and duration awarded.
To assess factors that were associated with sustained funding, the cohort of principal investigators who were awarded at least 1 grant between 2001 and 2005 was assembled and followed for 5 years after receiving their first grant. All records of CIHR applications and funding decisions were used to create the cohort and measure investigator-related predictors during the follow-up period. A unique encrypted identifier enabled applications from the same individual to be linked through time. CIHR legal and ethical review approved access to data for this study.
Funding success
In the open competition, applications are assigned to 1 of about 50 standing committees. Applications are reviewed and rated by a panel of peers who score the application on a scale from 1 to 4.9. A consensus score is initially reached after the application is presented and discussed. All panel members who are not in conflict of interest rate the application within plus or minus 0.5 of the consensus score. The mean score for the application is used to rank applications by score. The highest ranked applications within each committee are then funded. Applications with scores below a specific threshold - 3.0 between 2001 and 2003 and 3.5 from 2004 onward - are classified as being of insufficient quality to be fundable, even if funding is available. Funding success was defined as applications that were funded in the spring and fall open operating grant competitions from 2001 to 2011.
Potential predictors of CIHR funding success
Based on previous research, application characteristics that were measured included the year and season of the application submission (spring or fall), the pillar, the age and sex of the nominated principal investigator, the size of the research team (number of investigators listed on a grant application), whether the application was a resubmission of a previously unsuccessful application and whether it was a new grant or a renewal.13-23 Critical mass of research capacity within an institution is thought to be an important determinant of research success.13 To measure critical mass, we measured the number of researchers who submitted applications from the same institution as the principal investigator for the same competition. Critical mass was categorized as fewer than 50 colleagues, 50-100, 101-250, 251-350 and more than 350 colleagues. For subgroup analysis of the health services and policy research community, we measured critical mass of these applications from the same institution for the same competition, classified as fewer than 10 colleagues, 11-20, 21-35, 36-50 and more than 50 health services and policy research colleagues.
Sustained funding
Applicants were classified as having sustained funding if they had 5 years of continuous, uninterrupted funding from CIHR. To assess sustained funding, all successful applications to strategic and open competitions as either the principal applicant or as the co-principal applicant were included. The duration of funding for each grant received was used to assess whether the applicant had continuous funding in the 5 years following the first successful grant. A principal investigator who had a gap in CIHR funding of more than 1 month was classified as having unsustained funding.
Potential predictors of sustained CIHR funding
Two groups of potential predictors of sustained funding were assessed, based on theoretical models and previous research.13-23 First, we measured the characteristics of the first successful application, including the age and sex of the principal investigator, the pillar, the size of the research team, the critical mass of investigators at the applicant's home institution, whether the grant was new or a renewal, whether it was obtained in an open or a strategic competition, and grant duration. Second, during the 5-year follow-up period, we measured whether the principal investigator applied to only open competitions, only strategic competitions or both; whether the principal investigator had a salary award from CIHR; whether the principal investigator switched pillars, universities or research institutes in subsequent applications; and whether the principal investigator resubmitted any unfunded applications during follow-up.
Statistical analysis
Descriptive statistics were used to characterize funding trends between 2001 and 2011, including the proportions funded and classified unfundable, the total amount of funding for successful open and strategic grants per year, and average grant duration requested and awarded. To estimate predictors of funding success, we used multivariate logistic regression within a generalized estimating equation framework to account for clustering by principal applicant. Application was the unit of analysis, funding success was the outcome, application-related predictors were included as potential predictors, and an exchangeable correlation structure was used to account for clustering. To estimate predictors of sustained funding, multivariate logistic regression was used, and principal investigator was the unit of analysis. Sustained funding was the binary outcome, and potential predictors were included as covariates. Because a previous report of research funding in Canada suggested that more senior female scientists were less likely to be funded than their male counterparts,14 we used the Wald χ2 statistic to test whether success for female scientists was modified by age by fitting an age-sex interaction term in the model, as well as modelling the interaction separately as 4 dummy variables for sex-age combinations of older than and younger than 45 years of age to facilitate interpretation and reporting. Variance inflation factors were calculated for each variable in the model to assess multicollinearity. All analyses were conducted in SAS 9.3.
Results
Funding trends
Between 2001 and 2011, 80 163 grant applications were submitted to CIHR. Over time, there was a threefold increase in the number of applications; from 4411 to 12 723 per year overall, and from 327 to 1137 per year for health services and policy research (Figure 1). The overall funding success rate varied by calendar year, between 38.7% and 15.8%, but showed a steady decrease from 2008 onward, commensurate with a substantial and steady increase in the number of applications. The overall funding for grants increased from $399.2 million in 2001/02 to $759.7 million in 2011/12 (Figure 2). Although funding of health services and policy research grants represented only 3.2% of overall funding in 2001/02, this pillar experienced similar increases in funding, from $12.6 to $48.0 million, accounting for 6.2% of all funding in 2011/12. Applications from different pillars differed in the duration of funding requested and awarded. On average, biomedical applications requested the longest duration of funding (4.3 yr), whereas health services and policy research applications requested the shortest (2.8 yr) (Figure 3).
Figure 1.
Figure 1: Number of open and strategic grant applications by year, pillar and funding status (2001-2011).
Figure 2.
Figure 2: Total funding of open and strategic grant applications by fiscal year and pillar (2001-2011). CIHR = Canadian Institutes of Health Research.
Figure 3.
Figure 3: Requested and awarded duration of funding for open and strategic operating grant applications (2001-2011).
Predictors of funding success
The age and sex of the applicant modified the odds of funding success. Compared with female applicants younger than 45 years of age, male applicants of the same age were significantly more likely to receive funding (OR 1.12, 95% CI 1.02-1.24); particularly health services and policy applicants (OR 1.40, 95% CI 1.01-1.95; Table 1).
Table 1: The association between application characteristics and funding success for 33 155 new applications submitted to the open operating grant competition between 2001 and 2011.
| Application characteristic | All applications (n = 33 155) | Health services and policy applications (n = 2498) | ||||
|---|---|---|---|---|---|---|
| No. of applications | % funded | Odds ratio (95% CI) |
No. of applications | % funded | Odds ratio (95% CI) |
|
| Sex, age of PI, yr | ||||||
| Female, < 45 (reference) | 4 379 | 21.4 | 1.00 | 487 | 19.1 | 1.00 |
| Female, > 45 | 4 379 | 19.9 | 0.91 (0.8-1.01) | 603 | 19.2 | 0.99 (0.72-1.36) |
| Male, < 45 | 8 757 | 23.2 | 1.12 (1.02-1.24) | 489 | 24.5 | 1.40 (1.01-1.95) |
| Male, > 45 | 12 199 | 21.3 | 1.00 (0.91-1.10) | 645 | 21.4 | 1.10 (0.79-1.52) |
| Missing age or sex | 3 441 | 18.3 | 0.80 (0.71-0.91) | 274 | 19.0 | 1.02 (0.70-1.49) |
| Institutional critical mass, applicants (HSR) | ||||||
| < 50 (< 10) (reference) | 2 500 | 16.0 | 1.00 | 352 | 11.9 | 1.00 |
| 50-100 (10-20) | 1 330 | 19.5 | 1.15 (0.95-1.40) | 110 | 20.9 | 1.73 (0.96-3.13) |
| 101-250 (20-35) | 3 662 | 19.2 | 1.27 (1.09-1.47) | 516 | 21.7 | 1.74 (1.18-2.57) |
| 251-350 (35-50) | 9 458 | 21.2 | 1.45 (1.27-1.65) | 335 | 22.4 | 1.83 (1.21-2.76) |
| > 350 (> 50) | 16 205 | 22.9 | 1.67 (1.47-1.89) | 1 185 | 23.7 | 1.81 (1.28-2.57) |
| No. of co-investigators | ||||||
| 1 (reference) | 14 247 | 23.4 | 1.00 | 179 | 13.4 | 1.00 |
| 2 | 5 274 | 16.8 | 0.71 (0.65-0.77) | 186 | 15.1 | 1.19 (0.68-2.11) |
| 3 | 3 892 | 19.4 | 0.86 (0.78-0.95) | 298 | 14.8 | 1.15 (0.68-1.97) |
| 4 | 2 838 | 19.8 | 0.93 (0.83-1.04) | 351 | 17.1 | 1.28 (0.76-2.15) |
| ≥ 5 | 6 904 | 22.2 | 1.11 (1.01-1.22) | 1 484 | 24.5 | 2.08 (1.33-3.26) |
| Pillar | ||||||
| Biomedical (reference) | 21 321 | 21.6 | 1.00 | NA | ||
| Clinical | 5 394 | 18.7 | 0. 79 (0.72-0.87) | |||
| Health services/policy | 2 498 | 20.8 | 0.91 (0.80-1.03) | |||
| Pop and public health | 3 588 | 20.3 | 0.95 (0.85-1.06) | |||
| Missing | 354 | 63.0 | 8.12 (6.22-10.7) | |||
| Resubmission | ||||||
| No (reference) | 17 708 | 17.2 | 1.00 | 1 554 | 16.9 | 1.00 |
| Yes | 15 447 | 26.1 | 1.92 (1.82-2.03) | 944 | 27.1 | 1.83 (1.51-2.23) |
| Season | ||||||
| Fall (reference) | 16 429 | 20.5 | 1.00 | 1 262 | 19.7 | 1.00 |
| Spring | 16 726 | 22.1 | 1.11 (1.06-1.17) | 1 236 | 21.8 | 1.14 (0.93-1.39) |
| Year | ||||||
| 2001 (reference) | 2 416 | 27.6 | 1.00 | 225 | 20.9 | 1.00 |
| 2002 | 2 623 | 25.6 | 0.95 (0.84-1.07) | 221 | 21.7 | 0.94 (0.60-1.46) |
| 2003 | 2 581 | 24.9 | 0.97 (0.85-1.09) | 216 | 25.5 | 1.00 (0.64-1.57) |
| 2004 | 2 597 | 24.3 | 0.94 (0.83-1.07) | 198 | 26.3 | 1.16 (0.73-1.86) |
| 2005 | 3 134 | 22.3 | 0.84 (0.74-0.94) | 226 | 23.0 | 0.95 (0.61-1.47) |
| 2006 | 3 294 | 18.2 | 0.63 (0.56-0.72) | 213 | 16.0 | 0.58 (0.34-0.99) |
| 2007 | 3 309 | 24.5 | 0.87 (0.77-0.98) | 252 | 25.8 | 1.06 (0.70-1.62) |
| 2008 | 3 315 | 20.9 | 0.71 (0.63-0.80) | 235 | 20.4 | 0.73 (0.46-1.17) |
| 2009 | 3 622 | 17.5 | 0.57 (0.51-0.65) | 227 | 15.4 | 0.55 (0.34-0.92) |
| 2010 | 4 151 | 16.4 | 0.51 (0.45-0.58) | 330 | 17.6 | 0.68 (0.43-1.08) |
| 2011 | 4 654 | 18.3 | 0.49 (0.42-0.57) | 320 | 19.4 | 0.55 (0.31-0.95) |
Note: CI = confidence interval, HSR = health services research, NA = not applicable, OR = odds ratio, PI = principal investigator.
Larger health services and policy research teams had increased odds of funding for health services applications, but larger teams had the opposite effect - a significant reduction in the odds of funding - when applications from all pillars were assessed; the only exception being for very large teams of 5 or more investigators. Only 7.2% of health services and policy research applications were submitted by a single investigator, compared with 60.6% of biomedical applications, 10.2% of clinical and 9.7% of population health applications.
The critical mass of active investigators at an applicant's home institution also increased the odds of funding. Applicants whose home institution had more than 350 active investigators were more likely to receive funding (OR 1.67, 95% CI 1.47-1.89), with a linear increase in the odds of funding with increasing number of investigators. The same trend was evident for health services and policy applicants, although once above 10 active investigators, the odds of funding was similar (OR [10-20 investigators] 1.73, OR [ > 50 investigators] 1.81).
There were no significant differences in success rate by pillar, with the exception of clinical research applications, which were less likely to be funded than biomedical research applications (OR 0.79, 95% CI 0.72-0.87). The resubmission of a previously unsuccessful application significantly increased the odds of success, as did submission in the spring competition. Compared with 2001, there was a significant reduction in the odds of funding after 2004, with an odds of 0.84 in 2005 (OR: 0.84; 95% CI: 0.74-0.94), diminishing to an odds of 0.49 in 2011 (OR: 0.49; 95% CI: 0.42-0.57). We could not include new versus renewal status in the models, because almost all renewal applications were from the biomedical community (88% in the first 10 yr), and the models did not converge.
Predictors of sustained funding
Between 2001 and 2005, 4725 principal investigators had at least 1 successful grant application (Table 2). Within this cohort, 334 (7.1%) classified their first successful grant as health services and policy research, and 444 (9.4%) applied for a health services research grant in the next 5 years, all of which were included in the subgroup analysis. Older male investigators were more likely to have sustained funding compared with female investigators less than 45 years of age (OR 1.45, 95% CI 1.13-1.84). An equivalent increase in the odds of sustained funding in the health services subgroup was observed, but it was not statistically significant (OR 1.36, 95% CI 0.62-3.00). Of interest, older female health services and policy applicants were significantly less likely to have sustained funding than younger female applicants (OR 0.42, 95% CI 0.18-1.01). Clinical, health services, and population health investigators were all significantly less likely to have sustained funding than biomedical investigators. Only 24.0% of health services applicants had sustained funding, compared with 52.8% of biomedical applicants - a 66% reduction on the odds of sustained funding (OR 0.34, 95% CI 0.24-0.49). Investigators were also less likely to be funded if their home institutions had a lower critical mass, particularly for the subgroup of health services investigators, with a fourfold difference in sustained funding in institutions with more than 50 health services and policy applicants compared with institutions with fewer than 10 applicants (OR 3.88, 95% CI 1.21-12.5). Overall, a longer duration of funding awarded in the first successful grant increased the odds by 1.07 for each additional month requested (OR 1.07, 95% CI 1.06-1.08); odds increased by 1.08 in health services and policy research (OR 1.08, 95% CI 1.06-1.10).
Table 2: The association between application characteristics and odds of sustained funding among the 4725 principal investigators who received funding for a grant application between 2001 and 2005.
| Characteristic | All applications n = 4725 |
Health services and policy applications n = 444 |
||||||
|---|---|---|---|---|---|---|---|---|
| No. of applications | % sustained | OR (95% CI) | No. of applications | % sustained | OR (95% CI) | |||
| Sex, age of PI, yr | ||||||||
| Female, < 45 (reference) | 633 | 32.9 | 1.00 | 80 | 31.3 | 1.00 | ||
| Female, > 45 | 578 | 36.2 | 1.21 (0.90-1.63) | 104 | 19.2 | 0.42 (0.18-1.01) | ||
| Male, < 45 | 1346 | 43.7 | 1.18 (0.93-1.50) | 88 | 22.7 | 0.65 (0.28-1.54) | ||
| Male, > 45 | 1611 | 48.0 | 1.45 (1.13-1.84) | 129 | 35.7 | 1.36 (0.62-3.00) | ||
| Missing age or sex | 557 | 31.8 | 1.00 (0.74-1.35) | 43 | 25.6 | 0.52 (0.18-1.48) | ||
| Characteristics of the first funded grant | ||||||||
| Institutional critical mass, applicants (HSR) | ||||||||
| < 50 (< 10) (reference) | 512 | 21.1 | 1.00 | 48 | 10.4 | 1.00 | ||
| 50-100 (10-20) | 178 | 21.9 | 1.32 (0.79-2.20) | 64 | 23.4 | 2.28 (0.62-8.35) | ||
| 101-250 (20-35) | 1000 | 43.1 | 1.85 (1.37-2.51) | 91 | 26.4 | 2.79 (0.83-9.35) | ||
| 251-350 (35-50) | 902 | 45.1 | 2.04 (1.50-2.79) | 105 | 35.2 | 3.83 (1.19-12.4) | ||
| > 350 (> 50) | 2133 | 45.5 | 2.03 (1.52-2.70) | 136 | 30.1 | 3.88 (1.21-12.5) | ||
| No. of co-investigators | ||||||||
| 1 (reference) | 2544 | 46.7 | 1.00. | 34 | 29.4 | 1.00 | ||
| 2 | 558 | 37.8 | 0.76 (0.61-0.96) | 28 | 25.0 | 1.36 (0.28-6.73) | ||
| 3 | 432 | 29.9 | 0.67 (0.50-0.88) | 51 | 11.8 | 0.59 (0.12-2.87) | ||
| 4 | 309 | 33.0 | 0.84 (0.61-1.16) | 59 | 23.7 | 1.08 (0.26-4.51) | ||
| ≥ 5 | 882 | 36.8 | 1.06 (0.82-1.36) | 272 | 31.3 | 1.22 (0.33-4.47) | ||
| Pillar | ||||||||
| Biomedical (reference) | 2698 | 52.8 | 1.00 | NA | ||||
| Clinical | 484 | 36.4 | 0.47 (0.36-0.62) | |||||
| Health services/policy | 334 | 24.0 | 0.34 (0.24-0.49) | |||||
| Pop and public health | 523 | 26.6 | 0.52 (0.39-0.71) | |||||
| Missing | 686 | 20.0 | 0.77 (0.58-1.02) | |||||
| Funding source | ||||||||
| Open competition (reference) | 2934 | 51.5 | 1.00 | 210 | 32.4 | 1.00 | ||
| Strategic competition | 1791 | 24.8 | 1.11 (0.92-1.34) | 234 | 23.1 | 1.36 (0.76-2.44) | ||
| Grant type | ||||||||
| New grant (reference) | 3615 | 33.7 | 1.00 | 430 | 26.5 | 1.00 | ||
| Renewal | 1110 | 66.6 | 3.01 (2.48-3.66) | 14 | 57.1 | 2.32 (0.39-13.9) | ||
| Duration of grant (per month) | 4725 | 41.8 (19.8) | 1.07 (1.06-1.08) | 444 | 37.7 (19.7) | 1.08 (1.06-1.10) | ||
| Characteristics during follow-up | ||||||||
| Funds applied for: follow-up | ||||||||
| Both open and strategic (reference) | 2644 | 48.9 | 1.00 | 289 | 36.3 | 1.00 | ||
| Open only | 893 | 42.9 | 0.75 (0.62-0.90) | 38 | 26.3 | 0.70 (0.34-2.04) | ||
| Strategic only | 506 | 27.1 | 0.56 (0.41-0.75) | 71 | 7.0 | 0.39 (0.13-1.19) | ||
| Had a CIHR salary award | ||||||||
| No (reference) | 3949 | 38.0 | 1.00 | 381 | 25.7 | 1.00 | ||
| Yes | 776 | 58.9 | 2.21 (1.81-2.69) | 63 | 38.1 | 1.30 (0.64-2.66) | ||
| Resubmission during follow-up | ||||||||
| No (reference) | 2269 | 40.3 | 1.00 | 207 | 14.5 | 1.00 | ||
| Yes | 2456 | 42.4 | 1.10 (0.94-1.29) | 237 | 38.8 | 3.79 (1.99-7.22) | ||
| Switched pillars | ||||||||
| No (reference) | 3583 | 41.4 | 1.00 | 177 | 16.4 | 1.00 | ||
| Yes | 1142 | 41.4 | 1.63 (1.34-1.98) | 267 | 34.8 | 2.20 (1.18-4.11) | ||
| Switched universities | ||||||||
| No (reference) | 3735 | 40.9 | 1.00 | 281 | 25.6 | 1.00 | ||
| Yes | 990 | 43.1 | 1.43 (1.18-1.74) | 163 | 30.7 | 1.11 (0.63-1.98) | ||
Note: CI = confidence interval, HSR = health services research, NA = not applicable, OR = odds ratio, PI = principal investigator.
*Model adjusted for amount requested.
During follow-up, applicants who applied to both strategic and open competitions were more likely to receive sustained funding than those who applied only to the open (OR 0.75, 95% CI 0.62-0.90) or strategic (OR 0.56, 95% CI 0.41-0.75) competitions. Although similar trends existed for health services investigators, they were not statistically significant. There was a twofold increase in sustained funding among investigators with a CIHR salary award (OR 2.21, 95% CI 1.81-2.69), but not for health services and policy researchers. Resubmitting an unsuccessful application during follow-up significantly increased the odds of sustained funding for health services researchers (OR 3.79, 95% CI 1.99-7.22), but not for other investigators. Having research programs that encompassed more than 1 pillar increased the odds of sustained funding, particularly for health services investigators (OR 2.20, 95% CI 1.18-4.11), whereas switching universities increased the odds of sustained funding for all applicants (OR 1.43, 95% CI 1.18-1.74).
Interpretation
Between 2001 and 2011, there was a substantial increase in the number of applications submitted to CIHR, including applications for health services and policy research. The threefold increase in the number of applications may be related to substantial investment in health research infrastructure by the Canada Foundation for Innovation ($1.5 billion)24 and in world-class talent through the Canada Research Chairs program ($4.2 billion).25,26 In addition, CIHR and Canadian Foundation for Healthcare Improvement jointly funded regional training centres for almost a decade to boost capacity in health services research. Combined, these programs may have boosted the number and quality of applications to CIHR. Although the overall budget for CIHR in the first decade was $8.7 billion, the number of applications outstripped the capacity to fund excellent projects as funding rates in the open competition dropped from 27.6% in 2001 to 18.3% in 2011, while the proportion of fundable research increased. In Canada, the ratio of federal investment in infrastructure and people, relative to operational dollars to support these researchers, was 1.5. To guide science policy, future research should assess whether a higher ratio of operating funds might be needed to maximize the return on investment in health research.
Similar factors predicted funding success of health services and policy researchers and all CIHR applicants. Young male scientists were more likely to receive operating grant funding, as were applicants who were housed in institutions with a greater critical mass and with a willingness to resubmit unsuccessful grant applications. Other investigators have noted similar differences in research funding between male and female scientists.15,16 The most comprehensive analysis of this phenomenon was conducted with data from the Swedish Medical Research Council.15 The investigators found that female scientists had to have 2.5 times as many impactful publications as male scientists to receive an equivalent score by peer reviewers for scientific competence - a phenomenon they attributed to reviewer bias. The European Molecular Biology Organization reported the same phenomenon in an analysis of success rates for male and female postdoctoral fellows,16 as did a subsequent team of Swedish researchers using more recent data.17
Of interest, both the original and subsequent Swedish studies identified a "friendship bonus" or cronyism bias.15,17 Having a colleague who was in conflict with the applicant on the review panel increased the likelihood of funding. Even though the colleague did not rate the application, they appear to have influenced the committee. Institutions with greater critical mass may have more reviewers on the panel who are in conflict, increasing the odds that applicants from these institutions will be funded. However, a larger critical mass is known to confer other benefits, including research infrastructure, greater opportunities for collaboration,18,19 and professional support services that would help scientists in the institution develop and submit more polished proposals.20 The team size was one differentiating predictor of success with health services research that was not true of the general research community. This may reflect the requirement for multiple disciplines and collaborators to be involved in health services and policy research to produce high-quality science.
Health services researchers had a comparatively poor performance in obtaining sustained funding. Because the mean duration of funding requested by health services researchers was the shortest, and because most successful grants needed to be resubmitted to be funded, health services and policy researchers may be in a high-revolution grant treadmill. To maintain continuous funding, the average health services and policy researcher who requests a 2.5-year duration of funding would have to submit or resubmit many more applications. It is possible that these scientists draw from a more diversified pool of funders and thus can sustain their research teams through other sources of funding. However, the culture of health services and policy research, stemming from the early days of National Health and Development Research Program, was project-by-project-based funding. There was no support for multiyear programs of research that may be needed to address important gaps and relevant priorities. This may be why Canadian health services researchers, at least in 1 report, had the lowest rates of scientific productivity, as greater effort has to be made to sustain their research funding, robbing from time needed to publish.27
The amount of funded health research in Canada, including health services and policy research, has been successfully increased by CIHR. However, this sector of science still accounts for the smallest proportion of funding. If there is to be true innovation in health system reform to meet the challenges of the increasing burden of chronic disease and multi-morbidity, then greater investment in this sector will be needed. Knowledge-intensive industries invest at least 5% in research and development to sustain active innovation.28 Based on this assumption, with an annual expenditure of $215 billion in health care in Canada in 2014,29 $10.75 billion should be invested in health services and policy research to meet the goals of health system adaptation and transformation. Because health care costs amount to about 50% of provincial budgets, and 11% of the gross domestic product,29 there is an urgent need to act to address the research and development funding needed for system reform. The new Strategy for Patient-Oriented Research30 is intended to boost provincial and academic capacity to provide timely, responsive and relevant research for system transformation. The new Canadian Alliance for Health Services and Policy Research31 provides another avenue for collaboration on a common vision and strategy for funding research and the next generation of health services and policy researchers that can engage in a learning health system.
Limitations
Information was available only for CIHR applications, not all potential sources of support, and there were no data on potential losses to follow-up through retirement or out-migration. Although this limitation will not influence factors associated with CIHR funding success, it may bias the estimated prevalence of sustained funding and the factors associated with it.
Data on nationality, race and training were not available and have been shown to influence funding success in other studies.32,33 Moreover, scientific productivity was not measured, nor peer review reliability and validity. These are important areas for future research to understand sex differences in funding success rates and establish appropriate policy to enable optimal peer review.
Conclusion
The CIHR has successfully increased the amount of health services and policy research in Canada; however, health services research continues to represent a small proportion of research funding. To enhance conditions for success, health services and policy scientists should be encouraged to work in teams, request longer duration grants, resubmit unsuccessful applications and affiliate themselves with institutions with a greater critical mass. Future research should investigate potential sources of bias in peer review, as well as the impact of health services research on evidence-based decision-making and outcomes in health care to inform future policy.
Supplemental information
For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/4/2/E213/suppl/DC1
Supplementary Material
Footnotes
Funding: This study was supported by the Canadian Institutes of Health Research.
References
- 1.Scarbrough H, D'Andreta D, Evans S, et al. Southhampton (UK): NIHR Journals Library. Networked innovation in the health sector: comparative qualitative study of the role of Collaborations for Leadership in Applied Health Research and Care in translating research into practice. 2014. [PubMed] [Google Scholar]
- 2.Bégin M, Eggertson L, MacDonald N. A country of perpetual pilot projects. CMAJ. 2009;180:E88–9. doi: 10.1503/cmaj.090808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Facing the facts: the impact of chronic disease in Canada. Geneva: World Health Organization. 2005. [accessed 2016 Jan. 8]. Available www.who.int/chp/chronic_disease_report/media/CANADA.pdf.
- 4.Public Health Agency of Canada. Toward a healthy future: second report on the health of Canadians and strategies for population health: investing in the health of Canadians. Ottawa: Minister of Public Works and Government Services Canada. 2013. [accessed 2016 Jan. 8]. Available www.phac-aspc.gc.ca/ph-sp/determinants/determinants-eng.php.
- 5.Weingarten SR, Henning JM, Badamgarav E, et al. Interventions used in disease management programmes for patients with chronic illness-Which ones work? Meta-analysis of published reports. BMJ. 2002;325:925. doi: 10.1136/bmj.325.7370.925. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hoey J. Social sciences meet basic sciences in new era of health-services research. CMAJ. 2001;164:248. [Google Scholar]
- 7.Gray C. Clinicians will benefit from new research initiative, CIHR promises. CMAJ. 2000;163:586. [Google Scholar]
- 8.Kondro W. CIHR floats "research-into-action" trial balloon. CMAJ. 2003;168:209. [PMC free article] [PubMed] [Google Scholar]
- 9.Silversides A. Social scientists redirected to CIHR for grants. CMAJ. 2010;182:E27–8. doi: 10.1503/cmaj.109-3110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kondro W. Correcting the CIHR course. CMAJ. 2009;181:E223–4. doi: 10.1503/cmaj.109-3064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Institute of Health Services and Policy ResearchOttawa: Canadian Institutes of Health Research. 2014. [accessed 2015 Mar. 25]. Available www.cihr-irsc.gc.ca/e/13733.html.
- 12.National consultation summary report. Ottawa: Canadian Institutes of Health Research. 2009. [accessed 2016 Jan. 8]. Available www.cihr-irsc.gc.ca/e/39976.html.
- 13.Colledge L. Munich: Elsevier; 2014. Response to the HEFCE review of research. [accessed 2015 Mar. 25]. Available www.elsevier.com/__data/assets/pdf_file/0019/53416/Elsevier-response-HEFCE-review-role-of-metrics.pdf.
- 14.Strengthening Canada's research capacity: the gender dimension. Ottawa: Council of Canadian Academies. 2012. [accessed 2016 Jan. 8]. Available www.scienceadvice.ca/uploads/eng/assessments%20and%20publications%20and%20news%20releases/women_university_research/wur_fullreporten.pdf.pdf.
- 15.Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature. 1997;387:341–3. doi: 10.1038/387341a0. [DOI] [PubMed] [Google Scholar]
- 16.Gannon F, Quirk S, Guest S. Searching for discrimination: Are women treated fairly in the EMBO postdoctoral fellowship scheme? EMBO Rep. 2001;2:655–7. doi: 10.1093/embo-reports/kve170. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Sandstrom U, Hallsten M. Persistent nepotism in peer-review. Scientometrics. 2008;74:175–89. [Google Scholar]
- 18.Marsh HW, Jayasinghe UW, Bond N. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol. 2008;63:160–8. doi: 10.1037/0003-066X.63.3.160. [DOI] [PubMed] [Google Scholar]
- 19.Jump P. REF 2014 results: table of excellence. London (UK): Times Higher Education. 2015. [accessed 2016 Jan. 8]. Available www.timeshighereducation.com/news/ref-results-reopen-funding-debate/2017616.article.
- 20.Else H. REF results reopen funding debate. London (UK): Times Higher Education. 2015. [accessed 2016 Jan. 8]. Available www.timeshighereducation.com/news/ref-results-reopen-funding-debate/2017616.article.
- 21.Past performance as predictor of successful grant applications. The Hague: Rathenau Instituut. 2007. [accessed 2016 Jan. 8]. Available www.leydesdorff.net/magw/magw.pdf.
- 22.Ross RG, Greco-Sanders L, Laudenslager M, et al. An institutional postdoctoral research training program: predictors of publication rate and federal funding success of its graduates. Acad Psychiatry. 2009;33:234–40. doi: 10.1176/appi.ap.33.3.234. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bland CJ, Center BA, Finstad DA, et al. A theoretical, practical, predictive model of faculty and department research productivity. Acad Med. 2005;80:225–37. doi: 10.1097/00001888-200503000-00006. [DOI] [PubMed] [Google Scholar]
- 24.Understanding the role of CFI-funded infrastructure in intersectoral linkages. Ottawa: Canada Foundation for Innovation. 2012. [accessed 2016 Jan. 8]. Available www.innovation.ca/sites/default/files/pdf/Intersectoral_linkages_and_CFI.pdf.
- 25.Government of Canada supports top researchers, encourages brain gain at Canadian post-secondary institutions. Ottawa: Canada Research Chairs. 2014. [accessed 2016 Jan. 8]. Available www.chairs-chaires.gc.ca/media-medias/releases-communiques/2014/march-mars-eng.aspx.
- 26.Picard-Aitken M, Foster T, Labrosse I, et al. Tenth-year evaluation of the Canada Research Chairs Program: final evaluation report. Montréal: Science-Metrix. 2010. [accessed 2016 Jan. 8]. Available www.chairs-chaires.gc.ca/about_us-a_notre_sujet/publications/ten_year_evaluation_e.pdf.
- 27.Peckham M. Research and development for the National Health Service. Lancet. 1991;338:367–71. doi: 10.1016/0140-6736(91)90494-a. [DOI] [PubMed] [Google Scholar]
- 28.National Science Foundation, National Center for Science and Engineering Statistics. Industry, technology, and the global marketplace. In: Science and Engineering Indicators. Arlington (VA): National Science Board. 2010. [accessed 2016 Jan. 8]. pp. 4–61. Available www.nsf.gov/statistics/seind10/c6/c6h.htm.
- 29.Spending. Ottawa: Canadian Institute for Health Information. 2015. [accessed 2016 Jan. 8]. Available https://www.cihi.ca/en/spending-and-health-workforce/spending.
- 30.Canada's strategy for patient-oriented research - improving health outcomes through evidence-informed care. Ottawa: Canadian Institutes of Health Research. 2011. [accessed 2016 Jan. 8]. Available www.cihr-irsc.gc.ca/e/44000.html.
- 31.Pan-Canadian vision and strategy for health services and policy research 2014-2019. Montréal: CIHR Institute of Health Services and Policy Research. 2015. [accessed 2016 Jan. 8]. Available https://www.cahspr.ca/web/uploads/conference/2015-05-25_Pan_Canadian_Vision_and_Strategy.pdf.
- 32.Ginther DK, Schaffer WT, Schnell J, et al. Race, ethnicity, and NIH research awards. Science. 2011;333:1015–9. doi: 10.1126/science.1196783. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Shavers VL, Fagan P, Lawrence D, et al. Barriers to racial/ethnic minority application and competition for NIH research funding. J Natl Med Assoc. 2005;97:1063–77. [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.