Academic Productivity of Directors of the Accreditation Council for Graduate Medical Education-Accredited Residency Programs in Surgery and Anesthesiology

Abstract

Background

Scholarly activity is expected of program directors of the Accreditation Council for Graduate Medical Education (ACGME)-accredited residency training programs. Anesthesiology residency programs are cited more often than surgical programs for deficiencies in academic productivity. We hypothesized that this may in part reflect differences in scholarly activity between program directors of anesthesiology and surgical trainings programs. To test the hypothesis, we examined the career track record of current program directors of ACGME-accredited anesthesiology and surgical residency programs at the same institutions using PubMed citations and funding from the National Institutes of Health (NIH) as metrics of scholarly activity.

Methods

Between November 1, 2011 and December 31, 2011, we obtained data from publicly available websites on program directors at 127 institutions that had ACGME-accredited programs in both anesthesiology and surgery. Information gathered on each individual included year of board certification, year first appointed program director, academic rank, history of NIH grant funding, and number of PubMed citations. We also calculated the h-index for a randomly selected subset of 25 institution-matched program directors.

Results

There were no differences between the groups in number of years since board certification (P = 0.42), academic rank (P = 0.38), or years as a program director (P = 0.22). However, program directors in anesthesiology had less prior or current NIH funding (P = 0.002), fewer total and education-related PubMed citations (both P < 0.001), and a lower h-index (P = 0.001) than surgery program directors. Multivariate analysis revealed that the publication rate for anesthesiology program directors was 43% (95% CI 0.31–0.58) that of the corresponding program directors of surgical residency programs, holding other variables constant.

Conclusions

Program directors of anesthesiology residency programs have considerably less scholarly activity in terms of peer-reviewed publications and federal research funding than directors of surgical residency programs. As such, this study provides further evidence for a systemic weakness in the scholarly fabric of academic anesthesiology.

Introduction

Training of resident physicians is a challenging, multifaceted enterprise that in the course of a few years transforms a trainee from an inexperienced neophyte to a knowledgeable, skillful, independent medical practitioner and professional. This requires that the trainee acquire clinical knowledge and skills, develop sound clinical judgment, and become proficient with critical evaluation of new medical information. The responsibility for ensuring that trainees receive the requisite clinical experience and educational opportunities to become independent practitioners and lifelong learners falls to the residency program director. This is a complicated job with multiple expectations and responsibilities. One of these, as codified by the Accreditation Council for Graduate Medical Education (ACGME), which sets standards for accrediting residency programs that are then operationalized as program requirements by the various Residency Review Committees (RRCs), is participation in scholarly activities. This expectation is intended to generate in program directors an aptitude for inquiry and emphasize for residents and faculty the critical importance of questioning traditional practices, generating, disseminating, and incorporating new evidence into clinical practice, and to underscore the commitment all physicians should have to lifelong learning.

Thus, we found it interesting that the March, 2012 Newsletter of the Anesthesiology RRC reported that 16% of anesthesiology programs reviewed over the previous 2 years were cited for failure to document scholarly activity whereas according to the Surgery RRC Newsletter from August 2011 such citations were issued to fewer than 5.9% of surgery programs. We hypothesized that this difference in raw percentage citation rate may in part reflect a discrepancy in scholarly activity between program directors of anesthesiology and surgery. To test this hypothesis, we examined scholarly productivity of current program directors of ACGME-accredited anesthesiology residency programs based on peer-reviewed publications cited in PubMed and success in obtaining funding for research from the National Institutes of Health (NIH). To control for institutional differences and benchmark anesthesiology program directors against another specialty, we performed the same analysis for program directors in surgery within the same institutions.

Methods

Data publically available on the web sites of the ACGME (http://www.acgme.org), the NIH Research Portfolio Online Reporting Tools (RePORT, http://projectreporter.nih.gov/reporter.cfm), PubMed (http://www.ncbi.nlm.nih.gov/pubmed/), and the various department or medical school websites were surveyed. Furthermore, we queried the publicly available databases of the American Board of Surgery (ABS; www.absurgery.org) and The American Board of Anesthesiology (ABA; www.theaba.org) in order to determine the year of American Board of Medical Specialties (ABMS) member board certification, which we used as a surrogate for years each program director has been practicing in their specialty.

On November 1, 2011, a list of current program directors of ACGME-accredited anesthesiology and surgery residency programs was obtained from the ACGME website. To control for institutional differences, we only included institutions that had ACGME-accredited programs in both anesthesiology and surgery. The name of the program director and year they were first appointed to the position were obtained from the ACGME website. Year of board certification, verification of middle initial, and academic rank of each program director was obtained from the website appropriate to each specialty (i.e. the ABS or ABA) and/or, when available, applicable departmental or medical school websites.

The name of each individual was then entered into PubMed and all citations to work published between 1962 and October 31, 2011 documented. Each search result was manually reviewed by one author for accuracy and plausibility (nature of the research; year it was performed; institution; etc.), and by a second when there was ambiguity or uncertainty. In the initial analysis we did not differentiate between the types of publications; letters to the editor were counted the same as original research, editorials, or reviews. We did differentiate between publications related to education, defined as those containing the words “medical student,” “resident,” “fellow,” or “education” in the article title or abstract, and those that were not. Lastly, the name of each individual was entered into NIH RePORT, which lists grants funded by NIH between 1988 and the present. RePORT lists only the principal investigators; therefore a person having a lesser role on a grant or listed as a co-investigator does not appear and would not be credited.

To further evaluate scholarship and publication quality, we randomly selected 25 institutions (20% of total) and performed a post hoc analysis on this subset of institution-matched surgery and anesthesiology program directors to compare authorship position (first, middle or last), type of publication (original research; editorials and reviews; case reports and letters to the editor), and h-index, a bibliometric statistic used widely in academic medicine to evaluate the relative quality of a person’s collective body of work.^1–3 The h-index is determined mainly by how often a publication is cited and is defined as the number (h) of an investigator’s publications that have been cited at least h times.⁴

Statistical Analysis

The baseline characteristics of our study cohort are presented as percentages for categorical variables and mean (± SD) or median with interquartile range (lack normality data) for continuous variables. Univariate associations of the paired samples were evaluated with nonparametric methods using Wilcoxon signed-rank sum test for continuous variables and McNemar's test (for dichotomous data) or Stuart-Maxwell test (generalized McNemar’s), as appropriate, for categorical variables. Multivariate regression models with generalized estimating equations (GEE modeling) were used to examine predictors of publications since our observations are matched by institution, and thus correlated. Modified Poisson regression with robust variance estimates⁵ was used to directly estimate rate ratios and Spearman correlations were calculated for the multiple endpoints. All statistical tests were two-sided, with a type I error of 0.05. A P-value of less than 0.05 was considered to be statistically significant. All statistical analyses were performed with SAS version 9.3 (SAS Institute, NC)

Results

There were 131 anesthesiology and 250 surgery ACGME-accredited residency programs as of November 1, 2011. Of these, 127 institutions had ACGME-accredited residency programs in both anesthesiology and surgery; only these programs were included in this analysis. One anesthesiology and one surgery program director could not be identified reliably in Pubmed because there was more than one person in that specialty publishing under the same name; these individuals and the corresponding program director in the same institution were excluded from the PubMed analysis.

Univariate analysis identified no baseline differences between the groups in terms of sex (P = 0.24), academic rank (P = 0.38), year of program director board certification (P = 0.42), number of years since board certification (P = 0.40), year of program director appointment (P = 0.22) or time serving as a program director (P = 0.15)(Table 1). However, program directors in anesthesiology had less prior or active NIH funding compared to program directors of surgical residencies (P = 0.002; Table 1) and fewer total PubMed citations (P < 0.001), education-related publications (P < 0.001), and recent publications (Nov 2006- Oct 2011; P < 0.001)(Table 2). Curiously, approximately 75% and 40% of anesthesiology and surgery program directors, respectively, had no education-related publications (defined as those containing “medical student,” “resident,” “fellow,” or “education” in the article title or abstract; Table 2). Multivariate analysis revealed that the publication rate for anesthesiology program directors was 43% (95% CI 31%-58%) that of the corresponding program directors of surgical residency programs holding other variables constant.

Table 1.

Baseline Characteristics

Characteristic	Anesthesiology (N = 127)	Surgery (N=127)	P Value
Sex, N (%)			0.24
Female	32 (25%)	22 (19%)
Male	95 (75%)	103 (82%)
NIH Funding, N (%)			0.002
None	124 (98%)	109 (86%)
Yes	3 (2%)	18 (14%)
Academic Rank, N (5)			0.38
Instructor	1 (1%)	1 (1%)
Assistant Professor	29 (26%)	20 (18%)
Associate Professor	38 (34%)	34 (31%)
Professor	43 (39%)	56 (56%)
Year Certified, median (Q1, Q3)	1994 (1990, 2001)	1994 (1988, 2000)	0.42
Current Cycle Length, median (Q1, Q3)	4.0 (3.0, 5.0)	4.0 (3.0, 5.0)	0.55
1st Appointment of Program Director, median (Q1, Q3)	2008 (2004, 2009)	2006 (2003, 2010)	0.22
Years as PD, median (Q1, Q3)	3.0 (2.0, 7.0)	5.0 (1.0–8.0)	0.15

NIH = National Institutes of Health

PD = Program Director

Table 2.

Publication comparison

Variable	Anesthesiology (N = 127)	Surgery (N=127)	P Value
Publications, median (Q1, Q3)	6.0 (2.0–14.0)	25.8 (11–55.0)	< 0.001
Education Publications, median (Q1, Q3)	0.0 (0.0, 1.0)	1.0 (0.0, 3.0)	< 0.001
< 5 Year Publications, median (Q1, Q3)	1.0 (0.0, 3.0)	6.0 (2.0, 12.0)	< 0.001

Among the randomly selected subset of program directors, post hoc analysis revealed that anesthesiology program directors had fewer first (P = 0.003), middle (P = 0.002), or last author publications (P = 0.009) and fewer original manuscripts (P < 0.001) and editorials and reviews (P < 0.001) compared to program directors in surgery but similar numbers of case reports and letters to the editor (P = 0.45) (Table 3). The h-index for surgical program directors was also significantly higher than that of anesthesiology program directors (P < 0.001, Table 3).

Table 3.

Order of authorships and type of manuscript

Order of Authorship Type of Manuscript h Index	Anesthesiology (N=25)		Surgery (N=25)		Median difference (Q1, Q3)		P Value
First, median (Q1, Q3)	2.0	(1.0, 5.0)	8.0	(1.0, 16.0)	5.0	(−1.0, 11.0)	0.006
Middle, median (Q1, Q3)	2.0	(0.0, 5.0)	8.0	(4.0, 26.0)	5.0	(−1.0, 23.0)	0.001
Last, median (Q1, Q3)	1.0	(0.0, 2.0)	5.0	(1.0, 9.0)	3.0	(0.0, 9.0)	0.005
Original manuscript, median (Q1, Q3)	2.0	(1.0, 6.0)	11.0	(6.0, 34.0)	10.0	(3.0, 28.0)	<0.0001
Review, median (Q1, Q3)	0.0	(0.0, 1.0)	4.0	(0.0, 9.0)	3.0	(0.0, 8.0)	0.0001
Case report or letter to the editor, median (Q1, Q3)	3.0	(1.0, 5.0)	4.0	(1.0, 10.0)	4.0	(−2.0, 6.0)	0.105
h index, median (Q1, Q3)	3.0	(2.0, 6.0)	11.0	(4.0, 19.0)	7.0	(0.0, 16.0)	0.001

Finally, total number of citations in PubMed, publication type, h-index, number of education-related publications, history of NIH funding, and publications at each academic rank are all highly correlated (P < 0.0001 for all correlations), with program directors of anesthesiology residency programs being lower than program directors of surgery.

Discussion

This is the first study to examine and compare the scholarship credentials of program directors of ACGME-accredited anesthesiology and surgery residency programs. Whether judged by total number of citations in PubMed, publication type, h-index, number of education-related publications, history of NIH funding, or PubMed-citable publications at each academic rank, program directors of anesthesiology residency programs underperform those of surgery. The differences are sizeable; program directors for anesthesiology contribute about 40% as many publications as their surgical peers within the same institutions. This is unlikely to be due to differences in the ACGME core program requirements between the fields because in both specialties academic achievement is stipulated as a key qualification of the program director. Nor can it be explained by the ever escalating burden of monitoring and complying with onerous administrative and regulatory requirements associated with directing an ACGME-accredited residency program, as these are the same for both specialties. In addition, the ACGME has specific requirements to ensure that program directors have protected time to fulfill the responsibilities and expectations of the position. Differences in the experience or seniority of program directors is also not responsible as years since board certification or tenure as a program director were the same in both groups. Therefore, although a causal relationship with the program director’s performance cannot be established, the fact that ACGME-accredited residency programs in anesthesiology are cited more frequently than surgical programs for inadequate scholarly activity seems justified.

The ACGME and RRC require program directors to have “significant academic achievements” in their specialty. The definition of academic achievements is broad enough to allow program directors to participate in bench or clinical trials, pursue and publish creative approaches to education, or assess and report educational outcomes. This is appropriate given the service of program directors as educational leaders within their respective institutions, their influence over department educational initiatives, responsibilities for oversight of the core program as well as all subspecialty ACGME-accredited fellowship programs, and major role in establishing and nurturing an environment of inquiry and scholarship. The latter is especially vital for trainees and the program as a whole, as a willingness to challenge assumptions and practices is crucial for lifelong learning. As such, the program director and faculty should continuously evaluate established local educational practices and newly introduced initiatives, apply and test evolving theories of postgraduate medical education, and refine them as needed to ensure training of the residents and fellows is appropriate. If done thoughtfully and systematically, the results of such assessments can and should be disseminated broadly so as to advance the specialty. The data presented here raise serious questions about the qualifications, interest, or commitment of many anesthesiology program directors to fulfill this particular responsibility.

We recognize that the nature of scholarly achievements expected of a program director may be broader than the sources of data used for this analysis. The NIH and PubMed databases used here include only NIH-funded projects and manuscripts published in peer-reviewed journals cataloged by PubMed. For academic anesthesiologists and surgeons, lectures, curriculum development, and educational leadership are important aspects of creative activities assessed by the RRCs. The databases used in this analysis do not capture such activities but they do include the most significant scholarly activity that undergoes peer evaluation. The RRCs of both specialties consider peer-reviewed publications to be important. The Surgery RRC takes the position that the program director “sets the tone for the scholarly environment of the residency program” and, “because it is expected that both faculty members and residents are involved in research and publications, the program director should have evidence of peer-reviewed publication…” and be recognized and respected as having demonstrated success in scholarship (see 2012 Frequently Asked Questions; http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines/SurgicalAccreditation/Surgery.aspx). Likewise, the Anesthesiology RRC position is that participation in lectures, journal clubs, or anesthesia committees … “cannot substitute for publication in scholarly journals and other academic pursuits” (see 2012 Frequently Asked Questions; http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines/Hospital-BasedAccreditation/Anesthesiology.aspx). Both specialty RRCs point out that scholarship on the part of the program director and faculty is essential to their ability to influence resident scholarly activity, expose trainees to research methodology and ways to assess the literature, and, most importantly, to advance the specialty. Based on the findings of this study, anesthesiology is clearly lagging behind its surgical peers in this important dimension. However, this does not imply that the program director alone is responsible for meeting a department’s ACGME-mandated benchmarks for scholarship.

In fact, the entire faculty share responsibility for creating and encouraging a scholarly environment, but here too anesthesiology is weak. Based on 2006 data, departments of anesthesiology receive < 1% of the research dollars awarded by the NIH.⁶ Furthermore, a recent study of 24 accredited residency programs in the United States showed that the median number of career publications of academic anesthesiologists was just 3, with assistant, associate, and full professors having 1, 13, and 46, respectively, although the range within each rank was large.¹ Similar but modestly higher numbers were reported for groups of pediatric anesthesiologists³ and academic anesthesiologists³ in Canada. None of these studies included a nonanesthesiologist control group, but similar analyses of radiation oncologists and faculty in neurosurgery and urology suggest anesthesiologists publish less often than do these other specialists.^7–9 Cardiothoracic anesthesia may be an exception; faculty members from 30 randomly selected cardiothoracic anesthesia fellowship programs contributed 28 ± 46 publications on average, with senior faculty having more than junior faculty and ACGME-accredited fellowship program faculty having more than those in nonaccredited programs.² Still, the 15 faculty directing non-ACGME cardiothoracic anesthesia fellowships had 20 ± 24 publications,² which is several fold more than the directors of ACGME-accredited core anesthesia residency programs included in our study.

Moreover, in terms of scholarship, chairs of anesthesiology follow the same pattern compared to their counterparts in surgery, with the former having fewer than 40% as many PubMed citations and being less than half as likely to have had any type of NIH funding during their career as the latter.¹⁰ Indeed, in 2006 when the study was done, almost 1 in 5 academic anesthesiology chairs had 10 or fewer career publications.¹⁰ This is pertinent to the present work for a few reasons. First, inasmuch as anesthesiology faculty are generally academically weaker than surgical faculty, the pool of academically qualified faculty from which to choose a program director is likely to be smaller in anesthesiology than it is in surgery. Second, with more scholarly credentials, chairpersons of surgery may understand and value scholarship more and be in a better position than some chairs of anesthesiology to emphasize it and mentor their program directors in this regard once they assume the post.

Indeed, the low scholarly activity of anesthesiology residency program directors may simply mirror the weak academic environment of anesthesiology as a whole. This path might begin with anesthesiology residency itself. Many surgical residency programs require that trainees spend 1 or more years in research whereas few anesthesiology programs do. With no background in scholarship, junior faculty tend not to be productive in traditional academic endeavors even when enrolled in a formal departmental faculty development program.¹¹ Whatever the reason, failure to establish a strong foundation in research among our trainees and junior faculty, or to recruit and retain those with such an interest, may relegate our specialty to perpetually underperform our more academically oriented colleagues in other specialties.

One implication of the discrepancy between the scholarship record of surgery and anesthesiology program directors is that it suggests the criteria and expectations for scholarship in program directors are not applied uniformly across these specialties by institutional offices and national organizations (e.g. Designated Institutional Official, local Graduate Medical Education Office) or the ACGME RRCs that are responsible for approving and sanctioning program director appointments. Of course, it could be that expectations for program director scholarship are sufficiently modest that anesthesiologists satisfy them and surgeons easily exceed them. Either way, greater transparency and guidance around expectations for program director scholarship at the ACGME and RRC levels is necessary. Along similar lines, the fact that program directors of anesthesiology had fewer publications in PubMed at each academic rank than surgery program directors (Figure 1) suggests that institutions have dissimilar criteria, review processes, or mechanisms for academic promotion of physicians in different specialties. This practice appears to favor academic anesthesiologists, although we cannot exclude the possibility that surgeons and anesthesiologists get promoted in different academic series / tracks.

Figure 1.

Number of publications by academic rank among program directors of anesthesiology or surgery residency programs. Data are expressed as mean ± SD with 5 and 95% confidence intervals and analyzed using a 2 way ANOVA demonstrating a significant difference (p<0.0001) in terms of publications at each academic rank between the two departments.

One of the more interesting findings is that although a program director is first and foremost an educator, few of their peer-reviewed publications (5–7%) reflect education-related work. In fact, approximately 75% and 40% of anesthesiology and surgery program directors, respectively, have no education-related publications. This irony is difficult to explain, although it may be related to lack of scientific training, unfamiliarity with use of psychometrics, a perception among program directors that research on educational programs is unnecessary or less professionally rewarding than other types of research, a tendency to publish in other forums such as newsletters or abstracts at educational meetings, or that peer-reviewed journals cited by PubMed consider work on educational innovation to have a lower priority for publication. Regardless, the impact of advances in education of anesthesiology and surgery residents will be limited unless they find their way into peer-reviewed journals. There is no shortage of worthy education-related topics to study. Most of the recent changes in postgraduate medical education are driven by educational theory rather than empirical evidence and, as such, there is both ample opportunity and great need to test the validity and efficacy of such practices by the rigorous standards required for publication in peer-reviewed journals.

Our methodology has important weaknesses. As noted, the metrics of scholarly activity were NIH funding and peer-reviewed publications. Clearly this misses other areas of legitimate academic achievement and scholarly work in which program directors might engage (e.g. contributions to books, syllabi, educational software or web-based educational materials). Furthermore, because we defined education-related work narrowly, we may not have credited all educationally relevant publications (e.g. simulation, training, mentoring) to this category. Similarly, NIH funding may not be a suitable metric for an educator; funding rates were low for program directors in both specialties, possibly because NIH funding for educationally related projects is limited. Furthermore, the NIH RePorter database lists only the principal investigator so participation as a co-investigator or expert consultant would not be captured. Similarly, we did not include other federal sources of grant funding (Department of Defense and Veteran’s Administration) or nonfederal sources of funding such as those offered by foundations or industry, though these are rarely available for innovative educational projects. It is unlikely, however, that any of these potentially confounding factors would disproportionately affect or bias the results in favor of one group relative to the other. In fact, a major strength of the study design is that the same criteria were applied to both groups; program directors were not only matched by institution but held similar academic rank and had similar years in practice and tenure as a program director.

In summary, our study provides further evidence for a systemic weakness in the scholarly fabric of academic anesthesiology. On every measure of scholarly activity we examined, anesthesiology program directors fall short of their colleagues in surgery. What constitutes the minimum required level of scholarly activity among physicians to be an effective trainer of others is unknown but this discrepancy between the specialties is worrisome because residency program directors are in a powerful position to mold the intellectual attitudes, aptitudes, and expectations of the next generation of physicians. If scholarly activity is essential for the advancement of medical science, improvement of postgraduate medical education, and enhancement of patient care within each subspecialty, then a program director must have credibility as a scholar. With medicine and medical education becoming ever more complicated and data more abundant, the need for scholarship in anesthesiology generally and anesthesia education specifically is greater than ever. The fact that a closely related specialty is better fulfilling this need makes it imperative that anesthesiology achieves the same high standard.