National Institute of Mental Health Recruitment Monitoring Policy and Clinical Trial Impact

Abstract

Background/aims:

The National Institutes of Health (NIH) implemented a recruitment milestone and progress reporting policy in fiscal year 2019. While too recent to evaluate, the National Institute of Mental Health (NIMH) previously implemented a similar policy in fiscal year 2006 which may forecast likely effects of the NIH-wide policy.

Methods:

An observational, single-group, pre/post evaluation of the association between the NIMH policy and the Relative Citation Ratio was conducted for non-fellowship, competing clinical trial grants funded from fiscal years 2004–2007.

Results:

124 clinical trial grants were identified. After adjusting for covariates, the clinical trial grants subject to the NIMH recruitment monitoring policy were associated with a statistically significant mean-per-grant citation ratio (citations relative to the field norm) 1.98 times that of the clinical trial grants that were not subject to the policy (p = 0.005; 95% CI: [1.23, 3.20]). The clinical trial grants subject to the policy were also associated with a non-statistically significant 1.58 times maximum-per-grant citation ratio compared to the clinical trial grants not covered by the policy (p = 0.24; 95% CI: [0.73, 3.44]).

Conclusions:

The NIMH recruitment monitoring and reporting policy was associated with a statistically significant increase in the mean-per-grant Relative Citation Ratio. NIMH-specific results suggest that the NIH-wide policy might also be positively associated with improved Relative Citation Ratio.

1. Introduction

In 2016, the National Institutes of Health (NIH) developed and implemented a comprehensive suite of policy initiatives to “ensure rigor, transparency, and effectiveness of the US federally-funded clinical trial enterprise” [1,2]. Beginning in fiscal year (FY) 2019, one of these policies requires that grant applicants submit key information about a proposed clinical trial, including planned recruitment milestones, in the funding application and submit this application to a clinical trial-specific funding opportunity announcement [3]. Once an application has been funded, NIH requires that funded researchers submit Research Performance Progress Reports to NIH on an annual basis to track the conduct of clinical trials [4].

The Research Performance Progress Report allows the NIH Program Officer to monitor the progress of the research study toward completion of the aims. The Program Officer is the NIH staff member responsible for the “programmatic, scientific, and/or technical aspects of a grant” [5]. This includes monitoring study enrollment progress toward the participant recruitment target. Progress Reports for clinical trials are required to report recruitment progress toward recruitment target milestones on at least an annual basis. Because a clinical trial is designed with a sample size that provides sufficient power to detect an effect of the intervention [6], failure to reach recruitment targets can render many trials underpowered to answer the research question.

To avoid the consequences of under-recruitment, the Program Officer can take remedial actions to “course correct” grants that are falling behind on targets. Example actions include increased frequency of reporting (e.g., moving from annually to quarterly or monthly), restructuring the release of funds (i.e., restricting or delaying future funds until improvement is seen), or providing the opportunity for administrative supplements (additional funds for unforeseen costs) to ensure that researchers can reach recruitment targets.

Because the NIH-wide policy was recently implemented, it is too soon to evaluate its impact on grant performance and research outcomes. However, while this NIH-wide policy became effective in FY2019, the National Institute of Mental Health (NIMH) was an early adopter of a comparable policy. The NIMH Policy for the Recruitment of Participants in Clinical Research mandated tracking and progress reporting of participant recruitment in studies enrolling at least 150 participants as of FY2006 [7]. The NIMH and NIH policies both require recruitment milestones and regular reporting of progress toward those milestones. The remediation options available to Program Officers are consistent under both policies. Therefore, the NIMH policy may be used as a proxy to forecast the potential impact of the NIH-wide policy.

The current study examines the relation between the implementation of the NIMH recruitment monitoring policy and the Relative Citation Ratio (RCR) research outcome metric [8]. Given the premise that regular recruitment progress monitoring and remediation should improve the likelihood of successful recruitment completion and the notion that successful recruitment should lead to sufficient statistical power to answer the research question, it was hypothesized that the recruitment monitoring policy will be associated with better grant performance outcomes as measured by an increase in the RCR.

2. Methods

The design of this study is an observational single-group pre/post evaluation of the association between the NIMH Policy for the Recruitment of Participants in Clinical Research and the mean-per-grant and maximum-per-grant Relative Citation Ratios. This policy became effective beginning with FY2006 grants [7]; therefore, the study compares outcomes among clinical trials funded in FY2004 and FY2005 with outcomes among clinical trial grants funded in FY2006 and FY2007. The primary outcome is a measure of research performance known as the RCR [8]. It is hypothesized that the recruitment reporting policy will be associated with improved mean and maximum grant RCRs.

The study sample was comprised of grants that met the following eligibility criteria: [1] new competing applications and competing renewals (i.e., applications seeking new years of funding for new aims) for non-fellowship research grants; [2] awarded by NIMH from FY2004-2007; [3] self-identified as clinical trials; [4] with a planned sample size of at least 150 participants (a criterion for applicability of the NIMH policy of interest); and [5] had RCR data available in the iCite database. The RCR is “a field-normalized metric that shows the scientific influence of one or more articles relative to the average NIH-funded paper” [8]. Key demographics and grant performance outcomes were collected for each grant in the sample. These data were analyzed for a relationship between the implementation of the recruitment monitoring policy and the grant performance outcome measure.

Clinical trial grants were identified using NIH’s Information for Management Planning Analysis and Coordination (IMPAC II) database using the Query View Report search interface. Grant characteristics were identified using the IMPAC II database and Research Portfolio Online Reporting Tools (RePORT) [9]. The RCR was identified from the NIH Office of Portfolio Analysis’s iCite tool [10].

The independent variable (the NIMH recruitment policy) became effective at the beginning of FY2006. Grants applications were subject to the policy if initially awarded in FY2006 or later. The dependent variable (RCR) was obtained for the sample of grants described. The RCR metric is a ratio of the number of citations/year for a publication or group of publications compared to the average number of citations/year for all publications in that field of research [10]. A ratio of “1” indicates that the publication(s) was cited with the same frequency as other publications in the field. A value of greater than 1 indicates that the publication was more frequently cited than others in the field (ergo it is more “influential” than other publications by the field). Because most grants will result in multiple publications, two versions of the RCR were used. The first outcome, mean RCR, is the average RCR from all publications resulting from the grant. The second outcome, maximum RCR, is the highest RCR value from any single publication resulting from the grant. The mean RCR demonstrates total grant productivity across all publications, but it may dilute the impact of the primary publication with less-impactful secondary publications or protocol publications. The maximum RCR will show the impact of the single most prominent publication from a grant. That is likely to be the publication associated with the grant’s primary outcome.

The analyses controlled for the following variables as potential confounders: total grant cost (unadjusted), trial target sample size, grantee institution size, funding mechanism, and trial design. Total grant cost and trial target sample size were continuous numeric variables. Grantee institution size was defined as the size of the research portfolio at a given research organization. Each organization was labeled categorically as either <$200,000,000 USD; $200,000,000–$400,000,000; or ≥ $400,000,000 USD in total FY2006 NIH funding identified in NIH RePORT. Funding mechanism was a categorical variable that represented the different types of grant mechanisms (e.g., R01, U01, or U10). Trial design, also a categorical variable, represented study design types such as randomized controlled trial, cluster randomized controlled trial, and pre/post trial.

The clinical trial grants were identified through a search of the internal NIH IMPAC II grant database. The database was searched for NIMH grants competitively funded from FY2004-2007 with the word “trial” in the text of the grant application title, abstract, or aims. This information was exported to a Microsoft Excel spreadsheet. Fellowship awards were excluded from the sample. Each grant was then manually reviewed to determine if the applicant identified the study as a clinical trial. If not, the record was excluded. Next, each grant was reviewed to determine the planned sample size. Proposals with fewer than 150 participants were exempt from the NIMH policy and therefore were excluded. Each grant then was reviewed to determine the clinical trial design, and planned sample size, and other characteristics.

The mean and maximum RCR values were identified from the iCite database [10]. If no publication information was available or linked to the grant, then the record was excluded from the sample. The mean RCR and maximum RCR values were analyzed and determined to be non-normally distributed by histogram, Shapiro-Wilk test, and Q-Q plot. The mean and maximum RCR values were log-transformed to establish a normal distribution. The log-transformed distribution of mean RCR and maximum RCR was confirmed as normal with the Shapiro-Wilk test, kernel density plot of studentized residuals, and Q-Q plot. No severe outliers were identified.

A multiple linear regression was conducted to evaluate the association between the recruitment reporting policy and the mean RCR for each grant. A second multiple linear regression was conducted to examine the association of the recruitment reporting policy with the maximum RCR for each grant. The RCRs were log-transformed to establish a normal distribution. The relationship between the policy and RCR was adjusted for fiscal year, total grant award cost, grantee institution size, trial target sample size, funding mechanism, and clinical trial design. These analyses were conducted using Stata/IC Version 14.2 [11].

3. Results

The study sample selection process is outlined in Fig. 1. Of the 4388 NIMH-funded competitive research grants funded from FY2004-2007, 487 mentioned the word “trial” in the title, abstract, or aims. Thirteen of these grants were fellowships and 474 were non-fellowship grants. Two-hundred sixty-seven grants self-identified the study as a clinical trial in the grant application, and 134 of those grants had a planned sample size of at least 150 participants. The remaining 124 grants comprised the sample for this analysis.

Fig. 1.

Sample Selection.

The clinical trial sample demographics are displayed by fiscal year in Table 1. Between 26 and 40 clinical trials with planned sample sizes of at least 150 participants were funded each year. The most frequent clinical trial design was an individually-randomized controlled trial design (110 out of 124). Twelve of the remaining trials were cluster-randomized controlled trials and two were pre/post trials (prospective intervention studies without a control group) [12]. Similarly, a research grant mechanism (e.g., R01) was the most frequent mechanism with 95 of 124 trials in the sample. Cooperative agreements, or U-mechanisms, were the next most common with 26 of 124 trials. Across the four fiscal years, the mid-size and larger grantee institutions received 70 of the 124 trials. The planned sample sizes ranged from 150 participants to 5920 participants with a mean of 485.

Table 1.

Sample demographics by fiscal year.

Fiscal year	2004	2005	2006	2007
n	28	26	40	30
Clinical trial design
Cluster RCT	6	1	3	2
Pre/Post	1	1	0	0
RCT	21	24	37	28
Funding mechanism
Career development awards (K)	1	2	0	0
Research grants (R)	22	20	25	28
Cooperative agreements (U)	5	4	15	2
Grantee size
x < $200,000,000 USD	8	11	19	16
$200,000,000 ≤ x < $400,000,000 USD	7	9	7	7
x ≥ $400,000,000 USD	13	6	14	7
Maximum relative citation ratio
Mean	8.53	8.55	7.06	6.84
Range	1.40–47.46	0.62–41.06	0.32–34.81	0.54–45.48
Mean relative citation ratio
Mean	1.96	2.14	2.87	1.74
Range	0.79–5.30	0.46–5.05	0.32–5.86	0.45–3.77
Average total award cost	$ 2,583,064	$ 2,258,076	$ 1,902,647	$ 2,822,873
Average planned sample size	871	403	298	446

The maximum RCR ranged between 0.32 and 47.76 with a mean of 7.78. For all publications associated with a grant, the mean RCR was 2.24 with a range between 0.32 and 5.86. Two multiple linear regressions were performed, and the coefficients and confidence intervals were exponentiated and displayed in Table 2.

Table 2.

Multiple linear regression analyses for mean grant and maximum grant relative citation ratios.

	Mean grant relative citation ratioa				Maximum grant relative citation ratiob
	β	P value	[95% confidence interval]		β	P value	[95% confidence interval]
Recruitment policy1	1.984	0.005**	1.232	3.196	1.581	0.246	0.726	3.440
Grantee size2
x < $200,000,000USD	REF	REF	REF	REF	REF	REF	REF	REF
$200,000,000USD ≤ x < $400,000,000	1.000	0.998	0.767	1.304	1.051	0.821	0.682	1.618
x ≥ $400,000,000USD	0.884	0.340	0.686	1.140	0.909	0.650	0.601	1.376
Total award cost	1.000	0.361	1.000	1.000	1.000	0.403	1.000	1.000
Planned sample size	1.000	0.360	1.000	1.000	1.000	0.896	1.000	1.000
Fiscal year	0.738	0.006**	0.595	0.917	0.753	0.114	0.529	1.072
Funding mechanism3
R01	REF	REF	REF	REF	REF	REF	REF	REF
K (Career development)	1.625	0.203	0.767	3.442	2.665	0.115	0.784	9.063
R15	0.843	0.698	0.353	2.014	0.443	0.258	0.107	1.831
R21	0.471	0.206	0.146	1.522	0.257	0.162	0.038	1.740
R34	1.667	0.053	0.993	2.799	0.867	0.739	0.373	2.019
R43	0.158	0.002**	0.049	0.512	0.061	0.005**	0.009	0.414
U01	1.208	0.290	0.849	1.720	0.890	0.689	0.500	1.582
U10	0.564	0.104	0.282	1.127	0.777	0.659	0.251	2.402
Clinical trial design
RCT3	REF	REF	REF	REF	REF	REF	REF	REF
Cluster RCT4	0.831	0.441	0.518	1.334	0.807	0.584	0.373	1.746
Pre/post trial	0.583	0.229	0.241	1.410	0.292	0.093	0.069	1.233

^aMean Relative Citation Ratio of all publications for each grant

^bHighest Relative Citation Ratio of any publication for each grant

¹NIMH Recruitment Milestone Reporting Policy;

²Grantee’s total NIH funding in FY2006;

³Individually-randomized control trial;

⁴Cluster-randomized control trial.

^**:0.001 ≤ P ≤ 0.01

The first multiple linear regression was performed to investigate whether the implementation of the recruitment monitoring policy could significantly predict the mean RCR of publications associated with a given clinical trial grant. The results of the regression indicated that the model explained 23.3% of the variance and that the model was a significant predictor of mean RCR. The implementation of the recruitment milestone reporting policy was associated with a 1.984 increase in mean per-grant RCR (p = 0.005; 95% CI: [1.232, 3.196]) when adjusting for grantee size, total grant award cost, planned sample size, fiscal year, funding mechanism, and clinical trial design. Fiscal year and the R43 funding mechanism were also positively associated with mean RCR. Grantee size, total award cost, planned sample size, and clinical trial design did not have statistically significant associations with mean RCR.

The second multiple linear regression was performed to investigate whether the implementation of the recruitment monitoring policy could significantly predict the maximum RCR of any publication associated with a given clinical trial grant. The results of the regression indicated that the model explained 16.7% of the variance but the model itself was not a significant predictor of maximum RCR at the α = 0.05 level (p = 0.1423). Under this model, the implementation of the recruitment milestone reporting policy was associated with a 1.581 increase in the maximum per-grant RCR (p = 0.246; 95% CI: [0.726, 3.440]) when adjusting for grantee size, total grant award cost, planned sample size, fiscal year, funding mechanism, and clinical trial design. The R43 funding mechanism was positively associated with an increased maximum RCR as shown in Table 2.

4. Discussion

This study investigated the association between a NIMH policy requiring the establishment of recruitment milestones and progress reporting with grant performance outcomes. The hypothesis was that the implementation of this recruitment policy would be associated with improved clinical trial grant performance as demonstrated by an increase in mean and maximum RCRs. This hypothesis was based upon three logical assumptions: [1] establishing recruitment milestones and monitoring progress toward said milestones would increase the likelihood of reaching those milestones, [2] studies that reach their recruitment targets are more likely to have statistical power to test their research hypothesis, and [3] research publications based on studies analyzed with adequate statistical power are more valued in peer review than underpowered research. Findings from this study suggest a positive association between the implementation of the recruitment monitoring and reporting policy and both the mean and maximum RCR; however, only the positive association with the mean RCR was statistically significant.

The finding that the recruitment monitoring policy was associated with an increase in mean RCR suggests that the scientific community found the publications resulting from the trials covered by the recruitment monitoring policy to be, on average, more valuable (and therefore worth referencing) than those not covered by the policy. Assuming that enhanced attention to recruitment increases the likelihood of sufficient recruitment, then the higher mean RCR may illustrate that these trials achieved statistical power to answer the research questions and publish results.

Unlike the mean RCR association, the positive association between the recruitment policy and the maximum RCR was not statistically significant. If the mean RCR represents the influence of all publications from the grant as a whole, then the maximum RCR can be understood as a measure of the single most influential publication from each clinical trial grant. The non-significant finding suggests that, while the policy may be associated with more influential research, the single-most influential publication may be correlated to something other than the policy. One possibility includes bias toward publication and citation of positive results. Previous research has shown that positive outcomes are more likely to be cited than negative outcomes; therefore, the maximum RCR per grant may be influenced by whether the study findings were positive [13–15]. Positive versus negative trial outcomes were not accounted or adjusted for in this analysis and should be investigated further.

The initial premise for examining this relationship was to forecast the impact of the FY2019 NIH-wide policy requiring establishment of recruitment milestones and periodic (at least annual) reporting of recruitment progress. The NIMH policy was viewed as a proxy for the subsequent NIH-wide policy as it requires the establishment of recruitment milestones and triannual reporting on progress toward those milestones. NIH policy requires at least annual milestones and recruitment progress reporting (with the option for NIH staff to choose more frequent time intervals). Both policies establish a premise of target milestones and periodic reporting of progress toward those targets. Accepting that the NIMH and NIH policies are similar, one can infer that the results of this study suggest a potential positive association between the implementation of the FY2019 NIH-wide policy and the mean RCR. Further, these findings discount the notion that the administrative burden of regular recruitment monitoring is harmful to research progress.

A number of limitations to this study warrant further exploration. Ten grants did not have publications properly linked to the grant number and were excluded from the sample. Individual publications and/or references to manuscripts were found for eight of those ten grants, but inclusion of the RCR outcomes for only the papers found by manual identification would have introduced bias. This study was designed to examine association and does not prove a causal pathway; potential confounders could include secular trends that independently led to the publication of higher-impact studies over time. The study did not examine the degree to which individual clinical trial grants complied with the policy nor did it assess the extent to which recruitment milestones were met. Future research should consider these outcomes overall and among key subgroups, such as under-represented minority study participants. In addition, this study examined only NIMH-funded clinical trials with a planned sample size of at least 150 participants. Moreover, the study did not account for historically high/low performing researchers/research teams. Further, this study did not evaluate the impact of publication journal prestige, trial outcomes (i.e., positive versus negative findings), program official involvement, or influence from non-traditional bibliometrics (e.g., Altmetric) on RCR.

5. Conclusions

The NIMH recruitment monitoring policy was associated with a statistically significant increase in the mean-per-grant RCR in this sample of NIMH-funded grants from FY 2004 to 2007. NIMH-specific results suggest that NIH-wide recruitment milestones and progress monitoring requirements might also be positively associated with improved RCR, and thus reflect more impactful research. Further studies are recommended to examine the impact of researcher compliance with the policy on this association, the association with smaller clinical trials, the impact of data and safety monitoring board oversight on recruitment and trial performance, and the association for trials in other scientific disciplines to increase the generalizability of these results.