Indicators of Early Research Productivity Among Primary Care Fellows

Abstract

OBJECTIVE

Little is known about the impact of fellowship training in primary care on subsequent research productivity. Our goal was to identify characteristics of research fellows and their training associated with subsequent publications and research funding.

DESIGN

Mail survey in 1998.

SETTING AND PARTICIPANTS

1988–1997 graduates of 25 National Research Service Award primary care research fellowships in the United States.

OUTCOME MEASURES

1) Publishing 1 or more papers per year since the beginning of fellowship, or 2) serving as principal investigator (PI) on a federal or non-federal grant.

RESULTS

One hundred forty-six of two hundred fifteen program graduates (68%) completed the survey. The median age was 38 years, and 51% were male. Thirty-two percent had published 1 or more papers per year, and 44% were PIs. Male gender (odds ratio [OR], 3.6; 95% confidence interval [95% CI], 1.4 to 9.2), self-reported allocation of 40% or more of fellowship time to research (OR, 4.4; 95% CI, 1.8 to 11.2), and having an influential mentor during fellowship (OR, 5.0; 95% CI, 1.5 to 17.2) were independently associated with publishing 1 or more papers per year. Fellows with funding as a PI were also more likely to have an influential mentor (OR, 3.0; 95% CI, 1.3 to 7.2).

CONCLUSION

Primary care fellows who had influential mentors were more productive in research early after fellowship. Awareness of the indicators of early research success can inform the policies of agencies that fund research training and the curricula of training programs themselves.

In recent years, academic leaders in the United States have identified a critical shortage of clinical researchers who are trained to conduct clinical trials, epidemiological and behavioral studies, and outcomes and health services research.¹ This shortage has been attributed to rising educational debt, increasing demands for clinical productivity in academic medical centers, and declining prospects for acquisition of federal funding by physicians.^2–6 Recent strategies to remedy this problem include integrated programs of clinical and research training,⁷ new doctoral degrees,³ and federal programs to enhance faculty development.^4,6 Some authorities have proposed a core set of skills for these physician-scientists that includes 2 or more years of research training, course work relevant to the scientific area of interest, financial support, and protected time for the first 2 to 3 years of the faculty position.^4,8 These proposals have been based largely on retrospective surveys of physicians already in academic medicine, rather than on systematic efforts to track the graduates of research fellowships and identify the attributes of fellows or their training that are associated with subsequent academic success.^8,9 A lack of studies of such educational outcomes is a major impediment to the improvement of training programs at all levels of medical education.^10,11 To identify indicators of early research productivity, we conducted a national survey of 1988–1997 graduates of a primary care research fellowship, the Institutional National Research Service Award (NRSA) program for Research in Primary Medical Care.

PROGRAM DESCRIPTION

In the Health Research Extension Act of 1985, Congress required that 0.5% of funds from the NRSA program administered by the National Institutes of Health (NIH) be allocated to primary care research training.¹ Responsibility for this program was delegated by NIH to the Health Resources and Services Administration (HRSA) in 1988, and 10 Institutional NRSA Programs within academic centers in the United States were funded.¹ In 1993–1994, 8 renewal awards were made to institutions in the original funding cycle and 15 new programs were funded. In 1998, 28 awards were made for a third 5-year cycle, 20 to existing programs and 8 to new institutions. All but 1 of these institutions have been located in schools of medicine; the other was in a dental school. Most programs have enrolled 2 or 3 individuals per year from 1 or more of the 3 primary care disciplines (family medicine, general pediatrics, or general internal medicine). Some have also enrolled individuals from other medical specialties (psychiatry, rehabilitation), other clinical fields (nursing, dentistry), or nonphysicians with research interests in primary care. The median duration of fellowship training is 2 years. Most programs encourage the completion of an advanced degree, such as a master's in public health (MPH). The design and funding of these programs has previously been described.¹²

METHODS

Questionnaire Development

We developed a mailed, self-administered questionnaire for HRSA/NRSA program graduates to assess the demographics and clinical discipline of trainees, their duration of training, degrees obtained, and time allocated to course work, research, and other activities during the fellowship. Most items were newly developed for this survey based on the hypotheses of study investigators and other program directors about variables that might influence research success, although some questions were adapted from prior studies.^13,14 Fellows were asked to characterize the educational model of their fellowship as either 1) an “apprenticeship” model in which the trainee worked on a project closely related to (and often funded by) a senior investigator; 2) a strategy of “early independence” in which the trainee developed her/his own research idea and identified collaborators and advisors to provide input; or 3) a combination of the 2 approaches.^12,15 We asked the participants if they had a mentor during training, if their mentor was “particularly influential” in their development, whether they continued to receive mentorship from that individual at the time of the survey, and whether they remained affiliated with the institution where they had trained. Participants were also asked to describe characteristics of their current professional position, including their employer, academic affiliation, faculty rank, current time allocation, and hours worked per week, their role on up to 3 current research projects, and their number of first-authored and co-authored publications since the beginning of fellowship. We assessed the longevity and experience of the training program itself by the number of NRSA funding cycles (either 1 or 2 cycles at the time of the survey) that the program had received. The survey instrument is available from the principal author by request. Respondents were also asked to provide their current curriculum vitae. The survey was approved by the University of Colorado Multiple Institution Review Board.

Participants

All individuals who received HRSA/NRSA support between July 1988 and June 1997 were eligible for the survey. Individuals who received support after June 30, 1997 were excluded because they lacked sufficient follow-up time for assessment of program outcomes. We identified program participants using HRSA records and rosters from the NRSA programs. HRSA data were also used to compare survey respondents and nonrespondents. After pilot testing, the survey was initiated in June 1998. The survey was administered by mail, with 2 follow-up mailings at intervals of 3 weeks, followed by attempted phone or e-mail contacts with individuals who had not replied.

Measures of Research Productivity

We defined 2 outcome markers for the research productivity of program graduates, similar to measures used in prior studies^1,8,9,15,16: self-reported publication of 1 or more papers per year as first author or co-author since the beginning of fellowship, and self-reported acquisition of funding as principal investigator (PI) from any federal or non-federal source. Although prior studies have typically assessed only funding received from the NIH,^1,16 we included funding from federal agencies such as the Agency for Healthcare Quality and Research, Veterans Administration, Centers for Disease Control and Prevention, Health Care Financing Administration, or HRSA, because primary care researchers often conduct research supported by these agencies.⁹ We also included nonfederal funding, since many primary care researchers receive career development or project grants from sources such as state or local government, private foundations, or private industry. In a subgroup analysis, we also identified predictors of federal funding as a PI. To address the potential bias in self-reported information about fellowship outcomes, we repeated all analyses for the subset of individuals who provided curriculum vitae. The results of this analysis were not substantively different from the results for all respondents, and are not reported here.

Data Analysis

To select potential indicators of research productivity, we performed 2-group comparisons using Mann-Whitney U tests or χ² tests. Because our intent was to define personal and training program characteristics that were associated with research productivity, we did not include variables describing the current professional position in the analysis. All variables that were statistically significant at P≤ .10 on 2-group comparisons for each outcome variable were forced into a logistic regression model. The predictive ability of the multivariate models was assessed using the c-index, a statistic that represents the probability that a randomly chosen individual with the outcome of interest is correctly classified by the model, compared to an individual without that outcome.¹⁷ We determined the person-years allocated to research during fellowship by multiplying the self-reported proportion of time spent in research by the duration of training. The respondents were stratified into 3 groups based on their person-years of research during fellowship, and the relationship with outcomes was assessed using the χ² test for trend. All analyses were performed using the Statistical Analysis System (SAS) Version 8.1 (SAS Institute, Cary, NC).

RESULTS

Of the 215 individuals who participated in the HRSA/NRSA program between July 1988 and June 1997, 146 (68%) completed the survey. Survey respondents and nonrespondents did not differ in age, race or ethnicity, or clinical discipline. The response rate was 63% for male program graduates and 75% for women (P = .06). One hundred eleven individuals (76% of survey respondents) provided their curriculum vitae. Individuals who included their curriculum vitae were more likely to be on full-time medical school faculties, and reported more publications and more time currently conducting research than fellowship participants who did not provide vitae (data not shown). Pearson correlations between the number of publications by self-report and on the curriculum vitae were r = .91 for first-authored papers, r = .89 for co-authored papers, and r = .92 for the total number of papers (all P values < .001).

The characteristics of study respondents, their activities during training, and their current positions are summarized in Tables 1 and 2. Most respondents were general internists (36%), general pediatricians (31%), or family physicians (23%). The respondents had completed their HRSA/NRSA training a median of 4 years prior to the survey. Sixty-five percent of these fellows completed an MPH or MSPH degree, and only 22% did not obtain an advanced degree during training. Over two thirds of NRSA participants had positions as full-time faculty members. Only 32% had published 1 or more papers per year since the beginning of fellowship, and 44% reported having funding as a PI. Twenty-six percent of former HRSA/NRSA fellows reported having federal funding as a PI.

Table 1.

Characteristics of HRSA/NRSA Primary Care Research Fellows and Their Training Programs, 1988–1997

Personal characteristics
Mean age, y ±SD	38 ± 5
Gender, % male	51.0
Race/ethnicity, % white	84.1
Clinical discipline, %
General internal medicine	36.3
Family medicine	22.6
General pediatrics	30.8
Other	10.3
Training experience
Time since completion of training, y ±SD	4 ± 2
>2 Y fellowship training, %	28.1
Advanced degree obtained*, %
MPH/MSPH	65.1
PhD	5.5
Other degree	13.9
No degree obtained	21.9
Mean percentage of fellowship time ±SD
Class work	29 ± 17
Clinical practice	18 ± 13
Clinical or research teaching	10 ± 9
Conducting research	38 ± 17
Other fellowship activities	5 ± 5
Research model used, %
“Apprenticeship” model only	16.7
“Early independence” model only	61.8
Combination of models	21.5
Wrote extramural grant during fellowship, %	35.6
Training environment and mentorship, %
Trained in program funded for both grant cycles	76.0
Remains affiliated with fellowship training site	54.8
Identified a research mentor	93.1
Had a “particularly influential” mentor	73.4
Continues to receive guidance from mentor	49.3

^*Some participants completed more than 1 advanced degree.

Table 2.

Characteristics of the Current Position and Research Productivity of HRSA/NRSA Primary Care Fellows, 1988–1997

On full-time faculty, %	67.6
Mean hours worked per week ±SD	51.3 ± 13.0
Mean percentage of time in current position ±SD
Clinical practice	36 ± 28
Teaching of clinical or research skills	18 ± 16
Conducting research	31 ± 28
Administration	14 ± 18
Other activities	0 ± 3
Currently involved in research, %	75.2
Number of first-authored papers ±SD by self-report*	2.9 ± 4.2 (range 0 to 25)
Number of co-authored papers ±SD by self-report*	2.7 ± 4.6 (range 0 to 30)
Number of papers per year since beginning of fellowship ±SD by self-report	0.8 ± 0.9 (range 0 to 3.8)
Published ≥1 papers per year, %	31.5
Funding sources for current research as PI†, %
Federal	33.6
Other government	6.3
Private foundation	28.7
Industry	6.3
Other sources	18.9
Missing	6.2
≥1 Grants as PI, %	44.1

^*Pearson correlations between self-report and publications as recorded from curriculum vitae for 111 survey respondents who provided curriculum vitae were r = .91 (P < .001) for first-authored papers and r = .89 (P < .001) for co-authored papers.

^†Out of 143 projects reported.

PI, principal investigator.

The bivariate relationships between personal and fellowship training variables and the 2 measures of research productivity are shown in Tables 3 and 4. Male gender, being a general internist, the proportion of time spent conducting research during fellowship, training in a program funded for both NRSA cycles, and having an influential mentor were the strongest indicators of publication productivity. The proportion of time spent conducting research during fellowship, writing a grant for extramural funding during fellowship, and having an influential mentor were most strongly associated with acquisition of a federal or non-federal grant as a PI.

Table 3.

Indicators of Publishing 1 or More Papers per Year by Participants in Primary Care Fellowships

Variable	≥1 Paper/Y, N = 45	<1 Paper/Y, N = 98	P Value
Mean age, y ±SD	39 ± 4	39 ± 4	.57
Gender, % male	75.6	40.2	.001
Race/ethnicity, % white	91.1	81.4	.14
Clinical discipline, %
General internal medicine	48.9	29.6	.03
Family medicine	8.9	28.6
General pediatrics	28.9	32.7
Other	13.3	9.2
Time since completion of training, y ±SD	4 ± 3	4 ± 2	.37
>2 Y fellowship training, %	24.4	30.6	.45
Advanced degree obtained, %	86.7	75.5	.13
Mean percentage of fellowship time ±SD
Class work	26 ± 15	31 ± 18	.07
Clinical practice	18 ± 10	18 ± 14	.71
Clinical or research teaching	9 ± 9	11 ± 8	.06
Conducting research	45 ± 15	33 ± 17	.0001
Research model used
Apprenticeship only, %	22.2	13.5	.40
Early independence model only, %	55.6	64.6
Both models, %	22.2	21.9
Wrote extramural grant during fellowship, %	44.4	31.6	.14
Trained in program funded for both grant cycles, %	84.4	60.2	.004
Remains affiliated with fellowship training site, %	53.3	54.1	.93
Had a research mentor, %	95.5	91.8	.43
Had a “particularly influential” mentor, %	90.2	66.3	.004
Continues to receive guidance from mentor, %	59.1	46.4	.16

Table 4.

Indicators of Having a Federal or Non-federal Research Grant as Principal Investigator Among Participants in Primary Care Fellowships

Variable	≥1 Grants as Principal Investigator, N = 64	No Grants as Principal Investigator, N = 81	P Value
Mean age, y ±SD	39 ± 4	38 ± 5	.11
Gender, % male	57.8	46.3	.17
Race/ethnicity, % white	84.1	84.0	.98
Clinical discipline, %
General internal medicine	42.2	30.9	.54
Family medicine	21.9	23.5
General pediatrics	26.6	34.6
Other	9.4	11.1
Time since completion of training, y ±SD	4 ± 2	4 ± 2	.41
>2 Y fellowship training, %	26.6	29.6	.68
Advanced degree obtained, %	82.8	74.1	.21
Mean percentage of fellowship time ±SD
Class work	26 ± 18	31 ± 17	.12
Clinical practice	17 ± 11	18 ± 14	.59
Clinical or research teaching	9 ± 9	11 ± 8	.13
Conducting research	42 ± 17	34 ± 16	.007
Research model used, %
Apprenticeship only	9.5	22.5	.11
Early independence model only	65.1	58.8
Both models	25.4	18.8
Wrote extramural grant during fellowship, %	43.8	28.4	.06
Trained in program funded for both grant cycles, %	75.0	63.0	.12
Remains affiliated with fellowship training site, %	54.7	54.3	.96
Had a research mentor, %	96.8	90.0	.11
Had a “particularly influential” mentor, %	85.5	63.6	.004
Continues to receive guidance from mentor, %	57.1	43.8	.11

The time spent conducting research during fellowship was associated with a progressive increase in subsequent research productivity (Fig. 1). This trend was significant for publishing 1 or more papers per year (P = .02), but not for funding as a PI (P = .18).

Figure 1.

Relationship between time spent conducting research during fellowship and early academic productivity. The 146 survey respondents were divided into 3 groups, based on the person-years spent conducting research during fellowship. As the time spent in research increased, the likelihood of publishing 1 or more papers per year increased significantly (P = .02 by the Mantel-Haenszel χ² test for trend), but the trend for funding as a principal investigator was not significant (P = .18).

Table 5 summarizes the multivariate models that defined independent indicators of publication and funding as a PI. Male gender, having spent 40% or more of fellowship time conducting research, and having an influential mentor were associated with publication of 1 or more paper per year after controlling for other variables. Influential mentorship was the only variable significantly associated with obtaining grant funding as a PI. In a separate multivariate model, training in a program funded in both NRSA funding cycles (multivariate odds ratio [OR], 3.0; 95% confidence interval [95% CI], 1.0 to 0.9), spending 40% or more of fellowship time conducting research (multivariate OR, 2.2; 95% CI, 0.9 to 5.0), and remaining affiliated with the fellowship training site (multivariate OR, 2.0; 95% CI, 0.9 to 4.5) were the variables most strongly associated with federal grant funding as a PI.

Table 5.

Multivariate Indicators of Academic Productivity Among Participants in Primary Care Fellowships

Variable	Published ≥1 Papers/y, Multivariate OR (95% CI)	Received ≥1 Grants as PI, Multivariate OR (95% CI)
Male gender	3.6 (1.4 to 9.2)	—
General internal medicine versus other disciplines	1.2 (0.5 to 3.0)	—
Trained in program funded in both grant cycles	1.9 (0.6 to 5.7)	—
≥40% Time spent in research during fellowship	4.4 (1.8 to 11.2)	1.4 (0.7 to 2.9)
Had a “particularly influential” research mentor	5.0 (1.5 to 17.2)	3.0 (1.3 to 7.2)
Wrote grant for extramural funding during fellowship	—	1.9 (0.9 to 3.8)
C-index for model	0.81	0.66

When no values are reported in a column, the corresponding variable was not a significant (P≤ .10) predictor of that fellowship outcome on 2-group comparisons. All variables that were significant at P≤ .10 were entered into a multivariate logistic regression model for that outcome. OR, odds ratio; CI, confidence interval.

DISCUSSION

In this survey of 1988–1997 graduates of the HRSA-funded NRSA program for research training in primary medical care, we found that a minority of these individuals had achieved either marker of research productivity defined for this study: 32% had published 1 or more papers per year, and 44% reported federal or nonfederal research funding as a PI. Having an influential mentor was the only characteristic significantly associated with both fellowship outcomes, while the amount of time spent conducting research during fellowship and male gender were independently associated with greater publication productivity, but not with funding as a PI.

Few prior studies have presented data that can be compared to these findings. Two Robert Wood Johnson Foundation–funded fellowship programs reported that 58% and 66% percent of program alumni were in full-time academic positions,^15,18 comparable to the rate for HRSA/NRSA program graduates reported here. A 1990 evaluation of physicians who participated in institutional and individual NRSA programs sponsored by the Division of Kidney, Urologic, and Hematologic Diseases in the National Institute of Diabetes and Digestive and Kidney Diseases found that 22% subsequently received NIH funding over the period up to 6 years after completion of their training,¹⁶ compared to 26% with self-reported federal funding and 21% with curriculum vitae–confirmed federal funding in the HRSA/NRSA program. PhDs who had received predoctoral or postdoctoral NRSA support from 1981–1982 published a median of 8.5 papers over the subsequent 13 years,¹ compared to the median of 3 papers published over 4 years by the graduates of the HRSA/NRSA program. Despite the many differences in study design and method of assessment, these comparisons suggest that the research productivity of HRSA/NRSA graduates has probably been comparable to that of other national, fellowship training programs.

Our study has several limitations. First, the sample size is small, and the response rate of 68% limits its generalizability. Second, all information was obtained by self-report. Self-reports of the number of publications were corroborated by information from the curriculum vitae, however, and the results of analyses limited to fellows who returned vitae were comparable to the findings for the entire study group. Third, many of the grants received by these junior investigators were small in scope and funding, such as individual career development awards or pilot project grants. Long-term research success is more likely to be determined by the receipt of larger research awards. Fourth, a longer period of follow-up would have permitted a more comprehensive assessment of the success of HRSA/NRSA fellowship graduates in publications or research. This is particularly problematic in primary care research, where prospective epidemiological, clinical, and health services research studies typically require several years to design and complete. Additionally, publication or research productivity may be delayed for many fellows because of competing professional demands for clinical care or teaching, or for personal reasons such as family obligations. Finally, our study relied on easily quantifiable measures of the process and outcomes of research, such as degrees attained, time allocation, the presence of a mentor, publications, and grants received, rather than a broader definition of academic productivity. We adopted this perspective because the legislative intent of the Institutional NRSA program is to promote research careers. Many individuals who enter Institutional NRSA training programs plan to become clinician-educators in academic settings or to use their skills outside academic medicine, however. Since we did not ask HRSA/NRSA participants about their intended career focus at the time they entered training, we could not assess their success in achieving their individual career goals.

Our study has several implications for current efforts to develop research capacity among physicians. The HRSA-funded NRSA primary care fellowship programs conformed closely to curricular recommendations for the training for clinical investigators, through a 2-year program that combines quantitative training, a master's degree, and hands-on research.^4,8 Despite this, only a minority of participants achieved early milestones for research productivity. Current proposals to revitalize the “endangered species” of clinician-investigators have advocated that a degree, typically a PhD, should be part of the training process,^3,7 and that research training should be prolonged beyond the traditional 1 to 2 years after completion of clinical training.⁷ Our findings demonstrate that the amount of time invested in research during fellowship was a stronger indicator of subsequent research productivity than the acquisition of a master's degree, typically an MPH. The MPH degree has become the standard credential for primary care fellowships, in part because it is feasible to complete the degree along with a research project during a 2-year training program. The curriculum of most MPH programs is primarily designed to train public health workers rather than researchers, however. Although the skills gained through an MPH may well have many benefits that we could not quantify, additional research training or experience may be necessary to further increase research productivity.

Clinical practice and teaching consumed over 28% of the time of HRSA/NRSA participants during the program (Table 1), although restriction of clinical work and teaching to less than 20% effort during fellowship has been recommended.¹⁹ Some NRSA fellows may have chosen to allocate more time to clinical and teaching activities because they intended to enter careers as clinician-educators. Others may have been encouraged to devote time to these activities by the primary care divisions or departments that sponsored their training programs. In a prior survey, directors of these HRSA/NRSA programs reported substantial financial pressures resulting from the low stipend provided for NRSA fellows, the lack of full reimbursement by NRSA training grants for tuition expenses, and the lack of support for faculty mentorship or program administration.¹² As a result, many programs used the clinical revenues generated by their NRSA fellows to help underwrite their programs.^12,20 To the extent that training programs felt compelled to allocate their fellows' time to clinical activities rather than research, the financial policies of the HRSA/NRSA program and other Institutional NRSA programs funded by the NIH may have constituted a “false economy” that impeded the success of the program in achieving its primary objective of increasing primary care research. Recognizing this problem, the recent National Research Council review of the NRSA program has recommended that stipend and tuition support be increased to levels that are “revenue-neutral” for the programs, and allow the clinical commitment to be more limited during fellowship.¹ The costs of administering 1 NRSA training program have been estimated²⁰; more generalizeable estimates of training costs would be useful to help set rational funding levels for these programs.

Our finding of differences in early publication productivity between male and female HRSA/NRSA program graduates complement prior surveys in which female faculty members reported fewer publications, less research funding as a PI, less-satisfactory mentorship, and lower career satisfaction than male faculty members.^21–23 Prior research has shown that family responsibilities,²³ less time allocated to research,^22,23 and lower institutional support^21–23 may all contribute to these differences. Because we did not assess the intended career focus, the anticipated career trajectory, or the life goals of fellows in the HRSA/NRSA program, we are unable to determine whether these important considerations differed between male and female fellows. In a separate analysis of data from this survey, we found that 41% of HRSA/NRSA graduates who had faculty positions emphasizing research were women, compared to 52% of those who were faculty clinician-educators, and 55% of those who were not on a full-time academic faculty.²⁴ These differences were not statistically significant (P = .34), however. Women in our study also reported working significantly fewer hours per week (47 ± 14 hours/week vs 56 ± 10 hours/week for male HRSA/NRSA graduates; P < .001). Further study would be necessary to determine whether these differences in the careers of women HRSA/NRSA graduates were due to their own choices or to the range of opportunities available to them after fellowship.

Although 93% of HRSA/NRSA program graduates identified a mentor during their training, merely having a mentor was not a predictor of subsequent research productivity, while having a “particularly influential” mentor was strongly predictive of both markers of research productivity. Studies that quantify the impact of mentorship in medicine are rare, despite the universal recognition of the importance of mentorship.^2,15,25,26 In other studies, faculty members with mentors reported greater career satisfaction²⁷ and research productivity^28,29 than those without. The attributes of effective mentors, and the components of successful mentor-protégé relationships require additional investigation.

In conclusion, our survey of graduates of the HRSA/NRSA fellowship program in primary care research has identified predictors of research productivity, such as the value of influential mentorship and the amount of time committed to hands-on research. These findings have substantial implications for the design of fellowship training programs. In an era when we expect clinical practice to be informed by evidence of effectiveness, we have not made comparable demands on the fellowship programs that train individuals who will generate that knowledge.^10,11,30 Studies of the long-term outcomes of fellowship training, prospective rather than cross-sectional in design, should be incorporated into the ongoing evaluation process of federal agencies and private foundations. Such studies are essential to make the case for continued support of these programs, and to identify training strategies that can assist institutions in meeting programmatic goals.