Description and Productivity of Emergency Medicine Researchers Receiving K23 or K08 Mentored Research Career Development Awards

Abstract

Objectives

The primary individual research career development awards for emergency medicine (EM) investigators are the K08 and K23 awards. To the best of the authors’ knowledge, postaward productivity of EM K08 and K23 awardees has not been previously described. The objectives were to describe EM researchers who have received K08 or K23 awards and to evaluate their postaward federal funding and publications.

Methods

This was a cross-sectional study, conducted during January 2012, of clinician-scientists who previously completed EM residency or fellowship programs and have received K08 or K23 awards from National Institutes of Health (NIH) or Agency for Healthcare Research and Quality (AHRQ) between fiscal years 2000 through 2011. Awardees were identified through the NIH reporting tool (NIH RePORTER). Postaward funding was abstracted, including R01-equivalent funding for K awardees who completed their K training by the end of the 2011 fiscal year. Postaward publications (with journal impact factor) were tabulated for all K awardees using PubMed and Journal Citation Reports. An e-mail survey was also conducted during September and October 2012 to describe the awardee characteristics (demographics and research background) of all EM K awardees using REDCap electronic data captures tools. Simple descriptive statistics are reported.

Results

Sixty-three EM awardees were identified; 24 (38%) were K08 awardees, and 39 (62%) were K23 awardees. Of the 38 (60%) awardees who completed their K training, 16 (42%) obtained subsequent federal funding, with six (16%) obtaining R01 funding (median time from end of K award to R01 award was 4.5 years). Overall, EM awardees published a mean (±SD) of 4.0 (±1.3) manuscripts per year (after the start of their K awards); the mean (±SD) impact factor of the journals in which these manuscripts were published was 4.5 (±5.7). Forty-five (71%) of EM K awardees responded to the survey. Respondents had a median age of 36 years (interquartile range [IQR] = 35 to 38 years), and 33 were male (73%). The median time from completion of residency to start of the K award was 6 years (IQR = 4 to 7 years), with 27 (60%) completing a research fellowship prior to the K award.

Conclusions

Over 40% of EM K awardees who completed their career development training subsequently obtained federal funding as principal investigator (PI) or co-PI, while approximately one in six obtained R01 equivalent funding. Given the numerous barriers facing emergency care researchers, EM K awardees demonstrate good postaward productivity.

In 2004, the Institute of Medicine Committee on the Future of Emergency Care in the United States health system identified a need to strengthen clinical evidence in emergency care and improve patient outcomes. To accomplish this, one of the Committee’s recommendations was to increase specialized research training for emergency physicians.¹ Formal research training is crucial given that emergency care researchers face numerous barriers to productive and successful research careers, including the lack of experienced mentors and training opportunities, inadequate protected research time, insufficient research infrastructure, and poorly defined research-based career tracks.² The relative paucity of emergency care researchers who have undergone formal research training may contribute to the fact that, compared to other specialties, EM receives proportionally less federal research funding.³

The National Institutes of Health (NIH) and the Agency for Healthcare Research and Quality (AHRQ) provide career development awards, or “K” awards, to advance the independence of new investigators conducting biomedical research through mentorship, salary support, and protected time. K awards play a crucial role in developing and maintaining the research workforce. K awardees are more likely to apply for and receive NIH research project grants, and have a higher percentage of years with NIH support, than comparable unfunded K applicants.⁴ Since 1987, the NIH has granted over 19,000 career development awards at a cost of more than $8 billion.⁴

The primary individual K awards for emergency medicine (EM) investigators are the K23 and K08 awards.⁵ Both awards are mentored research career development awards that usually last 3 to 5 years.⁶ The K23 award focuses on patient-oriented research, while the K08 award is for biomedical or behavioral research that is not patient-oriented. The exception is AHRQ, which uses the K08 activity code for patient-oriented research.

Prior studies have demonstrated that mentored research career development awards increase the likelihood that investigators will participate in research careers and contribute to their subsequent research success.^2,7 However, postaward productivity of EM career development awardees has not been previously described. Our objectives, therefore, were to describe EM researchers who have received K08 or K23 awards and evaluate their postaward federal funding and publications.

METHODS

Study Design

The study consisted of two parts. For the first part of the study, we conducted a cross-sectional analysis during January 2012, to identify EM researchers who have received federal, mentored, patient-oriented research career development awards and to evaluate subsequent postaward federal funding and publications. For the second part of the study, we conducted a survey of the EM awardees (identified in the first part of study) during September and October 2012 to collect awardee characteristics, which included demographics, research backgrounds, and career paths. The study was approved by the institutional review boards at the study sites.

Study Population

The study population consisted of clinician scientists who have previously completed residency or fellowship programs in EM and have received K08 or K23 awards from NIH or AHRQ between fiscal years 2000 to 2011. We did not include other nonfederal training awards (e.g., Robert Wood Johnson awards, professional society training awards), other federal training mechanisms of support (e.g., K12, K99, K01), or nursing training awards.

Study Protocol

Identification of EM K Awardees

We searched the NIH Research Portfolio Online Reporting Tools Expenditures and Results (NIH RePORTER) system to identify EM K awardees. Our search strategy is presented in Data Supplement S1 (available as supporting information in the online version of this paper). The NIH RePORTER system is an electronic tool that allows users to search a repository of both intramural and extramural federally funded research from 1988 to the present.⁸ The system includes research funded by NIH, the Centers for Disease Control and Prevention, AHRQ, the Health Resources and Services Administration, the Substance Abuse and Mental Health Services Administration, and the United States Department for Veterans Affairs. Data abstracted on individuals identified in NIH RePORTER included name of awardee, administering institution, activity code, and fiscal year awarded.

Since NIH RePORTER does not allow searches by specialty, each individual identified in the NIH RePORTER search was further reviewed through academic websites to determine if he or she completed an EM residency or fellowship program. Awardees who met our inclusion criteria were included for further analysis (Figure 1).

Figure 1.

Study flow chart. *See Data Supplement S1 for criteria. †By December 31, 2011.

Measurement of Subsequent Federal Funding

Because recently awarded K awardees would likely not have had enough time to obtain subsequent federal funding, we restricted this analysis to awardees who completed their career development training by December 31, 2011. We searched NIH RePORTER using the names of eligible awardees to evaluate for subsequent federal funding by the awardee as a principal investigator (PI) or co-PI (from year of K award to January 2012). NIH RePORTER does not report on funding obtained as coinvestigators. We defined the time to R01 equivalent as the time from the start of the K award to the year the R01 equivalent was awarded. We reviewed project titles, project descriptions, and organizations to ascertain which funded projects were associated with the EM K awardee, particularly for awardees with common names or multiple funded projects. We abstracted the year of the project, the funding institution, and the activity code for each federal funding project.

Measurement of Subsequent Publications

We evaluated all eligible awardees for postaward publications (from start of K award to April 2012) in PubMed using the EM K awardee name as the primary search criteria. We reviewed publications for topic content and author organization to verify that publications were associated with the EM K awardee. We abstracted the title of the article, the date of publication, and the journal name from PubMed. For each journal title, we searched the Thomson Reuters Journal Citation Reports to obtain a 5-year impact factor score for the journal. The 5-year journal impact factor is the mean number of times articles from the journal published in the past 5 years have been cited in the Journal Citation Report Year. It is calculated by dividing the number of citations in the Journal Citation Report Year by the total number of articles published in the 5 previous years.⁹

Survey Procedures

During September and October 2012, we conducted an online survey of the eligible EM K awardees to describe characteristics of awardees (see Data Supplement S2, the eSurvey [available as supporting information in the online version of this paper], for survey data form). We conducted cognitive interviews with two EM research fellowship directors to evaluate question style, wording, and content and to modify survey questions as needed. We then pilot-tested the online survey with five EM clinical researchers currently participating in research fellowship training programs to evaluate comprehension of survey questions, duration of survey, and online processes. We e-mailed survey invitations to awardees a maximum of three times. We collected and managed study data using REDCap electronic data capture tools hosted at the Clinical and Translational Science Institute at Children’s National in Washington, DC.¹⁰ The Research Electronic Data Capture (REDCap) is a secure, Web-based application designed to support data capture for research studies. Data collected from the survey included age, sex, race or ethnicity (mutually exclusive categories), years from residency completion to K award, research training prior to receiving the K award, mentorship, funding prior to K award, academic title at time of award, number of K submissions, area of research, current dedicated research time, and current administrative positions.

Data Analysis

We formatted data and coded variables using Microsoft Excel 2010 (Microsoft Corp., Redmond, WA) and STATA 11.0 statistical software (StataCorp., College Station, TX). We reported the publication productivity of K awardees by fiscal year. To standardize publication productivity for each year, we divided the total number of publications by K awardees published since the start of their K awards, by the total number of cumulative years during that period. For example, in 2000, there were three EM K awardees who started their K awards that year. This accounted for a total of 36 cumulative years among awardees at the time of our publication search (2000 to April 2012 = 12 years since start of K award; multiplied by three awardees). These awardees collectively accounted for 118 publications for a mean of 3.3 publications per year (118 publications divided by 36 cumulative years). We used means as the primary measure of central tendency for the journal impact factor (normally distributed).

RESULTS

Characteristics of Study Subjects

Between fiscal years 2000 and 2011, a total of 141 K awardees were identified using the NIH RePORTER search criteria. Sixty-three (45%) awardees met inclusion criteria of prior EM residency or fellowship training (median of 4.5 awardees per year; interquartile range [IQR] = 3.5 to seven awardees; Figure 2). Twenty-four (38%) were K08 awardees, and 39 (62%) were K23 awardees. The most common funding institutes were the National Heart, Lung, and Blood Institute (NHLBI; n = 11), and AHRQ (n = 8; Figure 3). Of the 63 EM K awardees, 38 (60%) completed their career development training (by the end of the 2011 fiscal year) and 25 (40%) were currently in training.

Figure 2.

Number of EM K awardees by year, n = 63.

Figure 3.

Funding institute of EM K awardees, n = 63. AHRQ = Agency for Healthcare Research and Quality; NCI = National Cancer Institute; NCRR = National Center for Research Resources; NHLBI = National Heart, Lung, and Blood Institute; NIA = National Institute on Aging; NIAAA = National Institute of Alcohol Abuse and Alcoholism; NIAID = National Institute of Allergy and Infectious Disease; NIAMS = National Institute of Arthritis and Musculoskeletal and Skin Diseases; NIBIB = National Institute of Biomedical Imaging and Bioengineering; NICHD = National Institute of Child Health and Human Development; NIDA = National Institute on Drug Abuse; NIEHS = National Institute of Environmental Health Sciences; NIGMS = National Institute of General Medical Sciences; NIMH = National Institutes of Mental Health.

Forty-five of 63 (71%) eligible EM K awardees responded to the e-mail survey (Table 1); 28 (62%) were K23 awardees and 17 (38%) were K08 awardees. Respondents had a median age of 36 years (IQR = 35 to 38 years). There was a higher proportion of male EM K awardees (n = 33, 73%) than female awardees (n = 12, 27%). The most common race or ethnicity groups were white (n = 27, 60%) and Asian (n = 13, 29%). The median time from completion of residency to K award was 6 years (IQR = 4 to 7 years). Over half of the respondents (n = 25, 56%) reported success on initial K application (see Data Supplement S3, Tables 1 and 2 [available as supporting information in the online version of this paper], for additional characteristics of respondents).

Table 1.

Characteristics of EM K Awardees (n = 45) ^*

Characteristic	n (%)
Age (yr), median (IQR)	36 (35–38)
Male	33 (73.3)
Race/ethnicity
White	27 (60.0)
Black or African American	1 (2.2)
American Indian or Pacific Islander or Alaska Native	0 (0)
Asian	13 (28.9)
Hispanic or Latino	2 (4.4)
Other race/ethnicity	0 (0)
No response	2 (4.4)
Years from residency to K award, median (IQR)	6 (4–7)
Completed a research fellowship prior to K award	27 (60.0)
Any advanced degree other than MD/DO prior to K award	26 (57.8)
Primary mentor on K award is from EM	11 (24.4)
Primary mentor on K award is at same institution	43 (95.6)
Career development award prior to K award	18 (40.0)
Federal funding prior to K award	8 (17.8)
Institutional funding (departmental or intramural) prior to K award	36 (80.0)
Foundation or industry funding prior to K award	29 (64.4)
Any funding (federal, institutional, foundation, or industry) prior to K award	45 (100)
Academic title at time of K award
Assistant professor	38 (84.4)
Associate professor	2 (4.4)
Instructor	5 (11.1)
Number of applications for K award (including award)
One	25 (55.6)
Two	7 (15.6)
Three	12 (26.7)
Four	0 (0)
Five or more	1 (2.2)
Obtained an advanced degree during K award	16 (35.6)
Percent of time currently dedicated to research activities
50% or greater	34 (75.6)
75% or greater	27 (60.0)
Current administrative positions
Department/division chair	0 (0)
Department/division vice-chair	8 (17.8)
Research director	10 (22.2)
Research fellowship director	2 (4.4)
Other administrative position	11 (24.4)

IQR = interquartile range.

^*See Data Supplement S3, Table 1, for additional survey results.

Main Results

Of the 38 EM K awardees who completed their career development training, there were 16 (42%) who went on to obtain subsequent (after the start of the K award) federal funding as PI or co-PI, for a total of 31 separate funded projects (Figure 4). Six (16%) awardees transitioned to R01 equivalents (includes R01, DP2, R23, R29, and R37 activity codes) in a median of 4.5 years (IQR = 4 to 7 years; calculated from the start of the K award to the year the R01 equivalent awarded; see Table 2 for subsequent federal funding by year of K awardee).

Figure 4.

Activity codes for subsequent federally funded awards (31 awards) for EM K awardees (n = 38).*† *Includes only EM K awardees who completed their K awards by December 31, 2011, and federally funded awards obtained after the start of the K awards. †R01 or equivalent (includes R01, DP2, R23, R29, R37); other R (includes R03, R15, R21); cooperative agreements (includes all U awards); program awards (includes all P awards); and other (includes all other awards that did not fit into prespecified categories above).

Table 2.

Subsequent Federal Funding for EM K Awardees (n = 38)^*

	Awardees With Subsequent	Total Subsequent Funded	Transitioned to R01 Equivalent
Year	Funding, n (%)	Projects, n	Funding, n (%)
2000	3/3 (100)	7	0 (0)
2001	1/2 (50)	1	0 (0)
2002	1/4 (25)	2	1 (25)
2003	1/4 (25)	3	0 (0)
2004	3/4 (75)	7	2 (50)
2005	1/2 (50)	3	0 (0)
2006	5/11 (45)	7	3 (27)
2007	1/6 (17)	1	0 (0)
2008	0/2 (0)	0	0 (0)
Total	16/38 (42)	31	6 (16)

^*Includes only EM K awardees who completed their K awards by December 31, 2011, and federally funded awards obtained after the start of the K awards.

The 63 EM K awardees published a total of 1,378 publications (after the start of their K awards) identified in PubMed. The mean (±SD) number of publications per awardee year was 4.0 (±1.3), and the journals in which these publications appeared had a mean (±SD) 5-year impact factor score of 4.45 (±5.69; see Table 3 for subsequent publications by year of K awardee).

Table 3.

Subsequent Publications in EM K Awardees (n = 63)^*

			Publications per Year	Mean Impact Factor (5 Year)
Year	EM K Awardees, n	Total Publications, n	(Since Start of K Award)	per Publication†
2000	3	118	3.3	4.16
2001	2	58	2.6	7.81
2002	4	136	3.4	4.69
2003	4	89	2.5	4.21
2004	4	205	6.4	4.55
2005	2	32	2.3	3.61
2006	11	311	4.7	4.01
2007	6	104	3.5	3.16
2008	8	132	4.1	5.08
2009	7	105	5.0	3.98
2010	5	59	5.9	3.87
2011	7	29	4.1	5.85
Total	63	1,378	4.0	4.45

^*Includes articles from the start of the K award year until the time the PubMed search was completed (April 2012).

^†Impact factor calculated using Thomas Reuters Journal Citation Reports during April 2012.

DISCUSSION

One of the primary objectives of mentored research career development awards, or “K” awards, is to advance the independence of new investigators. Given the numerous barriers for emergency care researchers, career development awards may play a crucial role in the development of EM researchers and improving the quality of emergency care. The primary objectives of this study were to describe characteristics of previously funded EM K awardees and to evaluate postaward measures of success. We identified a relatively small pool of EM K awardees (approximately five awardees per year) who account for about 1% of the total number of K awardees nationwide (approximately 450 K awardees per year, fiscal years 2000 to 2011). This proportion of federal funding for EM is similar to prior research that reported less than 1% of NIH grants and funds are awarded to EM.³ It does appear, however, that the number of EM K awardees has increased over the past 6 years with five to 11 awardees per year from 2006 to 2012, compared to two to four awardees per year from 2000 to 2005 (Figure 2).

We evaluated postaward funding and publications, particularly R01 equivalent funding, as a quantitative measure of transitioning to independence. While there are numerous measures of postaward success, the NIH R01 is frequently used as a criterion standard due to its generalizability across biomedical fields and the standardized process by which award decisions are made.¹¹ We only analyzed EM K awardees who had completed their career development training, since awardees still in their training would be less likely to have had sufficient time to have transitioned to R01 equivalent funding. For EM researchers, receipt of an R01 typically requires more than a decade of work following the completion of training.¹¹

To our knowledge, this is the only study that evaluates subsequent scientific productivity for career development awardees for a specific specialty. Thus, it is difficult to interpret the results of our study in the context of comparing EM to other medical fields. We did identify one prior NIH supported study that described characteristics of K applicants, the proportion of applicants retained in the biomedical workforce and their subsequent scientific productivity.¹² The study authors used multivariate models to create matched groups of funded and unfunded applicants for each K award to measure program effect. These models included a priority score “bubble,” which represented a pool of funded and unfunded applicants with similar priority scores to help create matched cohorts. Although the overall methodology used in this study was different than in our study, authors reported that 57% of K23 awardees (fiscal years 2000 to 2005, n = 190) received subsequent federal funding, and 18% received R01 or equivalent funding. In addition, the journals in which these K23 awardees published in journals with a mean impact factor of 4.7. These data suggest that postaward federal funding and publications in EM K awardees is on par with those of a general K awardee cohort.

A few interesting characteristics of EM K awardees should be noted from the results of the survey. Among respondents, there was a higher proportion of male EM K08/K23 awardees (73%) compared to female (27%) awardees. We also found there were relatively few underrepresented minorities among EM K08/K23 awardees. While our reported sex and race or ethnicity proportions were overall similar to those reported previously,¹² it does suggest a need for further evaluation and monitoring. The results of the survey also suggest that many respondents had considerable investment in research prior to their K08 or K23 awards. The median time from completion of residency to K08 or K23 award was 6 years, with many respondents reporting prior research career development training, either through a research fellowship (60% of respondents) or through another career development award (40% of respondents). In addition, all respondents received research funding (federal, institutional, foundation, or industry) prior to their K08/K23 awards. Only 24% of respondents reported their primary K08/K23 mentors were from EM. While this proportion is not surprising given the relative youth of EM as a specialty, it does suggest an ongoing need for more EM-trained researchers and senior investigators who can provide mentorship. Finally, many respondents hold various administrative positions such as department or division vice-chair or research director. This infers a potential broader effect of K08 and K23 awards on departmental research leadership.

A number of recent developments have the potential to improve the quality of EM research. In July 2011, the NIHLBI funded six institutional (K12) awards to promote multidisciplinary clinical research training programs in EM to prepare clinician-scientists for independent research careers and academic leadership roles in EM.¹³ This was the first emergency care specific research career development award funded through the NIH. This award complements current society- and foundation-funded EM institutional and individual research career development awards (e.g., Emergency Medicine Foundation and the Society for Academic Emergency Medicine) to grow the overall pool of EM researchers and mentors. Furthermore, the NIH recently announced the formation of the Office of Emergency Care Research (OECR).¹⁴ Because NIH institutions are organized around particular diseases or organ systems, the lack of an emergency care–specific NIH institute has been cited as a barrier to funding for EM researchers.¹⁴ The OECR will be a focal point for basic, clinical, and translational emergency care research and training across the NIH and will serve as an integrated, multi-institutional, common “home” for EM researchers.¹⁴ Finally, the development of a credentialing process will provide standardization of academic EM research fellowships.¹⁵

LIMITATIONS

Our results should be interpreted in the context of some limitations. Our methodology to identify EM K awardees may have missed eligible awardees, particularly because there are no EM-specific search criteria in NIH RePORTER. We did request access to the Information for Management, Planning, Analysis, and Coordination (IMPAC II) grants database from the NIH, which would provide us with additional data points on all K applicants. However, this request was denied due to privacy concerns. Although nonfederal funding such as intramural or society grants provide crucial funding for emergency care researchers, we did not include these funding sources in our analysis of postaward funding, because obtaining this data accurately and comprehensively was logistically prohibitive. We also did not capture federally funded projects where the EM K awardee was supported as a coinvestigator, since we were unable to capture these data in the NIH RePORTER. We used the awardee name as the primary search criteria, which may have missed identifying postaward funding and publications. This would be particularly true for common or changed names. However, we did analyze topic content and manuscript titles to help ascertain funding and publications of awardees. In addition, none of the survey respondents reported name changes. Although 71% is a relatively good response for surveys, our survey sample size was small (n = 63), imparting a higher degree of uncertainty that the respondents represented the complete cohort. Finally, it should be noted that impact factor is designed to be a metric of the quality of a journal, not of individual papers or their authors, and there is significant debate regarding the merits of the impact factor as a performance metric.¹⁶

CONCLUSIONS

We identified an increasing number of EM K awardees across the 2000–2011 fiscal years. After their K awards, awardees published a mean of 4.0 manuscripts per year in journals with a mean impact factor of 4.45. Over 40% of EM K awardees who completed their career development training subsequently obtained federal funding as primary investigator or co-primary investigator, while approximately one in six obtained R01 equivalent funding. Given the numerous barriers facing emergency care researchers, EM K awardees demonstrate good postaward productivity.