The Impact of Grantsmanship Self-Efficacy on Early Stage Investigators of The National Research Mentoring Network Steps Toward Academic Research (NRMN STAR)

Abstract

The NRMN STAR program was created to address the persistent underrepresentation in grant submissions and receipt of National Institutes of Health (NIH) awards by racial/ethnic minority groups. In our current study, we assessed program impact on trainees’ self-efficacy related to grant writing. The program was conducted with two cohorts: one in June 2014 and one in June 2015. We used a 19-item grant writing self-efficacy scale drawn from the 88-item Clinical Research Assessment Inventory of three domains (conceptualizing, designing, and funding a study) to predict whether self-efficacy influences researchers’ grant submissions. Trainees were assessed prior to and following program completion with subsequent assessments at 6 and 12 months beyond participation. The majority of trainees were Black (62%), female (62%), and had obtained a PhD (90%). More than half (52%) were assistant professors and 57% had none or <1 year of research experience beyond postdoctoral training. However, 24% of trainees reported no postdoctoral research training. NRMN STAR trainees’ self-efficacy significantly improved on all three domains exhibiting a 2.0-point mean change score on two domains (conceptualizing and design) and 3.7 point mean change score on the domain, funding a study. Findings suggest that NRMN’s STAR provides impactful, confidence-building training for diverse, early stage investigators with little-to-no skills, experiences, or low self-efficacy in writing research grants.

Introduction

The National Institutes of Health’s long-standing effort to diversify the biomedical and behavioral science research workforce underscores NIH’s appreciation for the growing change in the United States demographic that will likely represent the next generation of scientists to advance the health of our country.2 Academic institutions remain critical to innovation and discovery driven by faculty conducting independent research sponsored by NIH and other extramurally funding sources. However, as the United States becomes increasingly racially diverse, faculty from underrepresented groups (eg, African Americans, Hispanics, Native Indian and Alaskan Pacific Islanders and persons with disabilities) participate in smaller percentages as independent investigators, particularly at research-intensive institutions.3,4

Based on the 2017 report by NIH’s chief officer for Scientific Workforce Diversity, Dr. Hanna Valantine, there has been a continued funding gap between first-time NIH R01 applications from African American vs White faculty researchers.5 Between FY 2011 and FY 2015, 11% of first-time African Americans vs 17% of White faculty researchers were awarded NIH R01 awards.5 Furthermore, relative to their White peers, African American investigators: submitted fewer initial R01 grant applications; received lower overall priority scores; and resubmitted unfunded grants less frequently.5 These factors constitute critical barriers to reducing the racial disparity in funded initial R01 grant applications. Because securing extramurally funded research is a pre-requisite milestone for promotion and tenure, such barriers are expected to adversely impact underrepresented minority (URM) populations’ persistence and advancement in the biomedical and behavioral science research workforce.

NIH’s most recent initiative, “Enhancing the Diversity of the NIH-Funded Workforce,” led to the creation of the National Research Mentoring Network to assemble various professional development coaching groups with the goal of implementing effective grant writing practices on a national scale to address low grant success among URM groups.1,6 NRMN’s Steps Toward Academic Research (NRMN STAR) was one of the coaching group models chosen to prepare postdoctoral fellows and early stage investigators for research grant writing and enhance professional development skills needed to succeed as an independent researcher. The target audience was URM trainees who had little-to-no grant writing experience. NRMN STAR originated from the University of North Texas Health Science Center’s 6 Texas Center for Health Disparities (TCHD) successful STAR Fellowship grant writing and professional development program. The goal of the TCHD STAR Fellowship Program was to increase the number and success of early stage investigators, particularly those from minority-serving institutions to become health disparity researchers. Between 2008-2017, TCHD STAR participants garnered grant awards in excess of $6 million, warranting national attention as a best practice in grant writing and professional development for postdoctoral fellows and junior faculty.7

Identifying factors that predict successful outcomes could benefit the development of appropriate training programs for early stage investigators, particularly among URM groups. Self-efficacy is believed to be a critical determinant in advancing through career transition points.8-10 In this regard, measures have been developed and tested to assess research self-efficacy and improvements over time.11,12 Perhaps one of the prominent measures is the Clinical Research Assessment Inventory (CRAI), an 88-item questionnaire and its derivative versions, which evaluated the reliability of factors predictive of success among clinical researchers. Harwood and colleagues created an abbreviated, 19-item grant writing self-efficacy assessment drawn from CRAI to confirm the reliability of three domains (conceptualizing a study, designing a study, and funding a study) to monitor change in and predict the likelihood that self-efficacy influences biomedical and behavioral science researchers’ grant submission across NRMN professional development coaching groups.13 While these investigators did not find statistically significant differences in mean pre-assessment vs 6-month post-assessment test scores between training models for two grant writing self-efficacy domains (conceptualizing and designing a study), the NRMN STAR trainees demonstrated a 4-point improvement compared with the other NRMN grant writing coaching programs as it relates to the domain of funding a study.13 Herein, we report the NRMN STAR grant writing self-efficacy of trainees prior to program implementation and 12-months following completion. We hypothesized that there will be an improvement in all domains that is sustained beyond the 6-month post-assessment for the NRMN STAR trainees.

Methods

NRMN Steps Toward Academic Research (NRMN STAR)

Each cohort of the NRMN STAR program was recruited nationally through the NRMN web portal (www.nrmnet.org). Trainees were accepted into the NRMN STAR program in June 2014 and 2015. Each program year included postdoctoral fellows and early stage investigators who needed more than 6-month grant writing training experience and were recruited primarily from historically under-resourced colleges and universities across the nation. Although the NRMN STAR trainees were selected based on their motivation and potential to develop a research project, the majority of individual applicants had limited grant writing skills and experience in writing research grants. The NRMN STAR 12-month curriculum combined on-site professional development and education with distance learning, including online digital meetings and a resource repository. This structure emphasizes that postdoctoral fellows and early career faculty often find it difficult to accept summer-long fellowships that require extended absences from home and family.

Over the course of the year, program cohort participants focused on developing a research application, with the goal of completing and submitting their applications for the next NIH grant cycle or to non-federal funding agencies. The trainees’ program begins with an overview of the goals and mission of NRMN’s national initiative as well as an overview of the principles and curricular specifics of the NRMN STAR program. Throughout the 12-month curriculum, trainees engage in real time activities facilitated by the program directors. In addition, mid-career and senior research faculty with extensive NIH grant success are paired with two trainees to provide intensive coaching interactions. Furthermore, trainees and coaches benefit from interactions with their entire cohort as a working group. The NRMN STAR curriculum is described in greater detail by Jones and colleagues (manuscript in preparation). To date, two 12-month programs of NRMN STAR have been completed by 21 postdoctoral fellows and early career faculty who have received an 18-month follow-up assessment survey.

Data Collection

NRMN STAR trainees assessed their grantsmanship self-efficacy over the course of two and a half years. Specifically, trainees are assessed: before the program began (pre-assessment); shortly after program completion (post-assessment); and through online surveys (at 6-, 12-, and 18-month post-assessments) (Figure 1). The objective for this series of standardized assessments was two-fold: first to monitor change over time in grantsmanship self-efficacy scores and, second, to gather self-reported information on the status and dates of grant submissions and awards. All data including race/ethnicity, sex, and educational level, academic rank, institution type, postdoctoral research training and research experience were collected by administering the questionnaires in REDCap.14 All trainees were strongly encouraged to complete each assessment survey. Self-efficacy scores from pre-assessment and the 12-month post-assessment survey were used to monitor change over time. Appropriate institutional review board entities from the University of Utah and partnering institutions deemed the status of this work to be exempt.

Figure 1. Schematic of NRMN STAR assessment timeline.

The 19-item self-efficacy survey was administered to NRMN STAR trainees prior to program start and at various time intervals: at 6 months, 12 months, and 18 months post-assessment. Filled circles indicate data collected for the purposes the current study.

Measures

Grant Self-Efficacy

Self-efficacy is the confidence in one’s ability to perform specific tasks.10 The outcome variable, grantsmanship self-efficacy, was based on 18 items derived from the original 88-item Clinical Research Appraisal Inventory (CRAI).15 An additional item asks, in the conceptualizing a study domain, to rate confidence in their skills in “convincing reviewers that the research is worth funding.” The full 19-item instrument assessed three domains of grantsmanship self-efficacy: conceptualizing a study (8 items), designing and analyzing a study (7 items), and funding a study (4 items).13 Trainees rated their level of confidence in performing grantsmanship tasks using the same 0 to 10-point scale used in the original CRAI, 0 represents “no confidence” and 10 indicates “complete confidence” in one’s ability to successfully perform the task. Scores for each domain were summed, averaged, and recorded for each trainee. Details of the grantsmanship self-efficacy score have been described elsewhere.13

Demographic Characteristics

Demographic characteristics included race/ethnicity, sex, and education level. NRMN STAR trainees reported their race/ethnicity as Asian, Black, Hispanic/Latino, Native Indian and Alaskan Pacific Islanders, and more than one race. Sex was based on the trainees’ report of being male or female. Educational level was based on report of advanced degrees each participant had obtained. The responses included: None, Bachelor, Masters, PhD, Postdoctoral training, MD, MD/PhD, DDS, DVM, PharmD, and Other.

Academic Characteristics

These characteristics consisted of academic rank, postdoctoral research training, and research experiences. The trainees reported their current employment level: postdoctoral associate/fellow, instructor, assistant professor, or other (eg, scientist in non-academic setting). Trainees were asked about their amount of postdoctoral research training and experience beyond a post-doctoral program. Response options for both items included: none for training and none/<1 year for experience; 1 year and 1-2 years, 2-3 years and 3-5 years; and >3 years and >5 years.

Analyses

Frequencies were used to summarize the distribution of the demographic and academic-related characteristics for the total sample and by cohort. Chi-square tests were used to determine the significance of proportional differences between each of the two cohorts for the demographic and academic-related characteristics.16 Paired t-tests were used to compare pre-assessment grant writing self-efficacy scores with the grant writing self-efficacy scores at the 12-month post-assessment for each confidence domain for the total sample and by cohort. All statistical tests were two-sided; P<.05 were considered statistically significant. Analyses were conducted using STATA version 14.

Table 1 presents the distribution of demographic characteristics for the two NRMN STAR cohorts. Of the 21 trainees, the overwhelming majority were Black (62%), female (62%), and had obtained a PhD (90%). There were no statistical differences observed between cohorts as it relates to these demographic characteristics.

Table 1. Distribution of demographic characteristics of trainees in the first two cohorts of the National Research Mentoring Network Steps Toward Academic Research (NRMN STAR) Program.

Characteristic	Total, N =21	Cohort 1, N=11	Cohort 2, N=10	P
Demographic
Race/ethnicity, %				.208
Asian	4.7	.0	10.0
Black	61.9	81.8	40.0
More than one race	4.7	.0	10.0
Hispanic/Latino	28.5	18.1	40.0
Sex, %				.466
Male	38.1	45.5	30.0
Female	61.9	54.5	70.0
Education level, %				.366
MD	4.7	9.0	.0
PhD	90.4	81.8	100.0
MD, PhD	4.7	9.0	.0

The distribution of academic characteristics for the two NRMN STAR cohorts is shown in Table 2. More than half the trainees were assistant professors (52%) and had none or less than one year of research experience beyond postdoctoral training (57%). However, 24% of the trainees reported no postdoctoral research training, and 29% were employed at minority serving institutions. No statistical differences were observed between cohorts with respect to any of the academic characteristics.

Table 2. Distribution of academic characteristics of trainees in the first two cohorts of the National Research Mentoring Network Steps Toward Academic Research (NRMN STAR) Program.

Characteristic	Total, N =21	Cohort 1, N=11	Cohort 2, N=10	P
Academic rank, %				.249
Postdoctoral trainee	42.8	54.5	30.0
Assistant professor	52.3	36.3	70.0
Other	4.7	9.0	0.0
Postdoctoral research training, %
None	23.8	18.1	30.0	.756
1 year	38.1	45.4	30.0
2 to 3 years	23.8	27.2	20.0
>3 years	14.2	9.0	20.0
Research experience, %				.349
None or <1 year	57.1	72.7	40.0
1 to 2 years	19.0	9.0	30.0
3 to 5 years	19.0	18.1	20.0
>5 years	4.7	0.0	10.0
Institution type, %				.890
Non-minority serving institution	71.4	72.7	70.0
Minority serving institution	28.5	27.2	30.0

The grantsmanship self-efficacy mean comparisons for pre-assessment and 12 months post-assessment for each domain by cohort and the total sample is shown in Table 3. For the domain that represents conceptualizing a study, Cohort 1 (n=9) exhibited a statistically significant improvement with an average change score of 2.7 points. However, the average change score of 1.3 points for Cohort 2 (n=8) was not statistically significant. As it relates to the domain designing a study, average change scores for Cohort 1 and Cohort 2, both showed statistically significant improvements of 2.6 and 1.3, respectively. Finally, for the funding a study domain, Cohort 1 and Cohort 2 improvement scores were statistically significant at 4.7 and 2.7 points, respectively. When examining the total sample, NRMN STAR trainees’ self-efficacy significantly improved on all three domains exhibiting a 2.0-point mean change score on two domains (conceptualizing and design a study) and 3.7 points on the funding a study domain.

Table 3. Mean comparison of the pre- and 12-month post-assessment grantsmanship self-efficacy scores^a by domains for the National Research Mentoring Network Steps Toward Academic Research (NRMN STAR) Cohorts, n=17.

NRMN STAR COHORT	Conceptualize a Study (8 items)			Design a Study (4 items)			Fund a Study (7 items)
	Pre	Post	Change	Pre	Post	Change	Pre	Post	Change
Cohort 1	6.0	8.7	2.7b	5.8	8.4	2.6 b	3.6	8.3	4.7 b
Cohort 2	6.9	8.2	1.3	6.3	7.6	1.3 b	5.5	8.2	2.7 b
Both Cohorts	6.4	8.4	2.0 b	6.0	8.0	2.0 b	4.5	8.2	3.7 b

a. Confidence self-efficacy in ability perform associated tasks; item scale=0 to 10, ‘no confidence’ to ‘complete confidence’

b. Statistically significant change score differences (P<.05) within domains.

Discussion

Grant writing is an arduous skill to learn; yet, it is very critical to the success of most academicians. In this program, we sought to examine the grant writing self-efficacy of trainees at pre-assessment and at the 12-month post-assessment among 21 NRMN STAR trainees. Consistent with the program’s hypothesis, the NRMN STAR trainees improved grant writing self-efficacy from pre-assessment to the 12-month post-assessment. These findings suggest that NRMN’s STAR model curriculum provides impactful, confidence-building training for diverse early stage investigators with little-to-no skills, experiences, or low self-efficacy in writing research grants.

Overall, NRMN STAR trainees demonstrated improvements in their grant writing self-efficacy from program inception to the 12-month post-assessment. These findings extend the findings of Harwood and colleagues that showed an improvement from the beginning of NRMN STAR to the 6-month post-assessment.13 Further, these findings lend support to the sustainability of NRMN STAR program training for under-prepared research investigators. As of this writing, several STAR trainees continue to work with their coaches to submit or resubmit grant applications after their formal program training ended.

There are two possible explanations for sustained and improved grant writing self-efficacy of NRMN trainees. The design features of NRMN STAR is one explanation. Specifically, STAR trainees participate in a 12-month program that includes grant writing skills and supplemental activities, such as ways to: optimize access and use of NIH resources (eg, NIH RePORTER, sample NIH grant applications); effectively communicate with NIH program officials; establish effective research collaborations and networks; and, access and use a variety of reference repositories and databases. NRMN STAR trainees also receive formal trainings on professional development topics such as: negotiating faculty workloads, navigating the promotion and tenure process, and balancing work-life priorities, to name a few. Topics were discussed in a group and with coaches to allow trainees to hear practical examples and real-life experiences.

A second possible explanation for sustained grant writing self-efficacy is that STAR directors and coaches recognize the potential barriers confronted by diverse, underprepared, early-career research investigators. For example, in this study, 38% reported having only one year of postdoctoral training or exposure to grant writing processes (75% had no experience), which is considered insufficient to obtain funding for research that will produce enough publications for future grants and career promotions.17-19 Therefore, NRMN STAR’s supplemental activities fills a significant training gap between being a graduate student to holding research career positions.

Limitations

We acknowledge that this program has some limitations. First, although the sample size was somewhat small, it is consistent with the number of slots typically available in other training programs, such as a National Research Service Award (NRSA) training grant. Nevertheless, the improved NRMN STAR cohorts’ self-efficacy scores in grantsmanship domains provide evidence for future evaluation of training programs with similar cohort sizes. Second, as the grant self-efficacy scale is a validated scale on this population, it provides support for validity of its findings.13 In addition, because NRMN STAR trainee recruitment was based on a national effort across a variety of research institutions including both non-minority serving institutions and minority serving institutions, it lends support to the external validity of the findings. Based on findings from this program, we expect that follow-up analysis on grant submission and funding will reinforce the importance of developing best practices in grant writing that boost trainee’s self-efficacy.

Conclusions

The NIH continues to identify the barriers confronted by underrepresented principal investigators to achieve grant funding success inclusive of prior education, academic rank, training, publications and other biographical information.4,20,21 In addition to these quantifiable attributes, the results from previous studies suggests the need for a more comprehensive understanding of the minority/non-minority funding gap. In this program, we provide evidence of the benefits of the NRMN STAR grant writing program over a 24-month period, demonstrating a positive and sustained association in short-term and long-term gains in grant writing self-efficacy. Based on our findings, we believe the delivery of the NRMN STAR grant writing program and its positive impact on trainees’ confidence in grant development, accompanied by professional development activities, will bode well for increasing the productivity of URM grant success as well as having a positive impact on other relevant attributes for academic research success. Future NRMN STAR outcomes will determine the association of grant writing self-efficacy with grant submissions, grant awards with the expectation of increasing the pool of NIH grant applicants, representative of diverse racial/ethnic groups.