Productivity, prominence, and the effects of academic environment

Significance

Past studies have shown that faculty at prestigious universities tend to be more productive and prominent than faculty at less prestigious universities. This pattern is usually attributed to a competitive job market that selects inherently productive faculty into prestigious positions. Here, we test the extent to which, instead, faculty’s work environments drive their productivity. Using comprehensive data on an entire field of research, we use a matched-pair experimental design to isolate the effects of training at, versus working in, prestigious environments. We find that faculty’s work environments, not selection effects, drive their productivity and prominence, establishing that where a researcher works serves as a mechanism for cumulative advantage, locking in past success via job placement and thereby facilitating future success.

Abstract

Faculty at prestigious institutions produce more scientific papers, receive more citations and scholarly awards, and are typically trained at more-prestigious institutions than faculty with less prestigious appointments. This imbalance is often attributed to a meritocratic system that sorts individuals into more-prestigious positions according to their reputation, past achievements, and potential for future scholarly impact. Here, we investigate the determinants of scholarly productivity and measure their dependence on past training and current work environments. To distinguish the effects of these environments, we apply a matched-pairs experimental design to career and productivity trajectories of 2,453 early-career faculty at all 205 PhD-granting computer science departments in the United States and Canada, who together account for over 200,000 publications and 7.4 million citations. Our results show that the prestige of faculty’s current work environment, not their training environment, drives their future scientific productivity, while current and past locations drive prominence. Furthermore, the characteristics of a work environment are more predictive of faculty productivity and impact than mechanisms representing preferential selection or retention of more-productive scholars by more-prestigious departments. These results identify an environmental mechanism for cumulative advantage, in which an individual’s past successes are “locked in” via placement into a more prestigious environment, which directly facilitates future success. The scientific productivity of early-career faculty is thus driven by where they work, rather than where they trained for their doctorate, indicating a limited role for doctoral prestige in predicting scientific contributions.

The prestige of an academic institution is closely related to most measures of the quantity and quality of its faculty’s scholarly outputs. Faculty at more-prestigious institutions produce more of the scientific literature (1), receive more citations (2–4) and scientific awards (5–7) (Fig. 1), and train more of the faculty hired by other prestigious institutions (8).

Fig. 1.

Institutional prestige predicts early-career productivity and prominence of computer science faculty. Shown are median publication (left axis) and ${\log }_{10}$ citation (right axis) counts per faculty per institution (minimum three faculty per institution), accumulated through their first 10 years posthire, adjusted for growth in publication rates over time (SI Appendix, section A). Shaded regions denote 95% confidence intervals for least squares regression.

The origin of these imbalances is often attributed to the competitive nature of the academic job market, which serves to sort individuals into positions at different institutions according to their reputation and record of past achievements, including their publication and citation counts (9). Reputations and achievements may be influenced by meritocratic characteristics, such as an individual’s skill, effort, or potential, by nonmeritocratic characteristics like age or gender, or by external factors such as work environment, social connections, or even chance events (9–13).* Untangling the influence of these factors has proved difficult because of endogenous cumulative advantage (14), in which past achievement broadly correlates with future achievement. Identifying the social mechanisms that explain individual and institutional differences in scholarly output would clarify the degree to which academia operates according to meritocratic principles, inform efforts to address persistent social inequalities in academia, and ultimately expand scientific discovery.

Materials and Methods

The fact that more-productive individuals tend to have been trained at prestigious institutions and also currently work at other prestigious institutions presents a causal puzzle: Which is more important in explaining their greater productivity (number of publications) and prominence (number of citations), where they trained or where they work? To answer this question, we infer the causal effect of each environment on scholarly output, treating as a quasi-natural experiment the discontinuity in an individual’s circumstances that is caused by moving from their doctoral institution to their faculty institution.

We reconstruct these experiments from a unique and comprehensive dataset that documents the doctorate-to-faculty transitions of 2,453 tenure-track faculty at all 205 PhD-granting computer science departments in the United States and Canada, spanning 1970–2011, along with complete records of their scholarly output through 2017, encompassing more than 200,000 publications and 7.4 million citations. Here, productivity is defined as the number of papers published, and prominence as the number of new citations accrued across all existing papers, where both are measured on a yearly basis (see SI Appendix, section A). We examine faculty productivity and prominence in their first 5 years prehire and posthire, excluding the hiring year itself (i.e., $y$=0 in Fig. 2) to mitigate the effects of work in progress carried between institutions. Each quasi-natural experiment is parameterized by the prestige of the doctoral and the faculty institutions, in which we assign unique values to each institution using a network-based measure of an institution’s ability to place its graduates as faculty at other prestigious departments. This prestige measure correlates with authoritative rankings but has greater predictive power (8). Replacing prestige with the 2010 departmental rankings by U.S. News & World Report in our analysis produces similar results (see SI Appendix, section B and Fig. S1).

Fig. 2.

Early-career productivity is driven by work environment prestige. For pairs of computer science faculty matched by (A and B) work environment prestige or (C and D) training environment prestige, (A) publication and (B) citation counts are statistically independent of differences in doctoral prestige but are driven higher by (C and D) placing into a more prestigious work environment. Shaded regions denote 95% confidence intervals for the mean. Similar results are obtained using U.S. News & World Report department rankings in place of prestige (see SI Appendix, Fig. S1).

The annual faculty job market generates two kinds of quasi-natural experiments: It colocates at the same institution individuals who trained at more or less prestigious institutions than each other (Fig. 2, Top Left), and it separates individuals with similar training into faculty appointments at more or less prestigious institutions than each other (Fig. 2, Bottom Left). To isolate the effect of prestige differences on posthire productivity and prominence in each case, we combine exact and caliper matching techniques to mitigate the confounding effects of differences in the age, gender, subfield productivity norms, and postdoctoral training (see SI Appendix, section B). If where an individual trained determines their early-career scholarly output, individuals with more-prestigious training should be, on average, more productive and more prominent than colocated peers with less prestigious training. On the other hand, if where an individual works determines their early-career scholarly output, individuals with appointments at more-prestigious institutions should be more productive and more prominent than similarly trained peers with appointments at less prestigious institutions.

Results

For matched pairs of faculty with appointments at similarly prestigious institutions, the individual with the more prestigious training was not more productive in the first 5 years posthire ($N=359$ pairs; $p=0.59$, $t$ test) but received, on average, 301 more citations ($N=129$ pairs; $p<0.05$, $t$ test) during this period (Fig. 2 A and B). Among the pairs, the individual with more-prestigious training was more productive in 52.1% ($p=0.23$; one-tailed binomial test) of trials but more highly cited in 63.9% ($p<0.005$; one-tailed binomial test).

In contrast, for matched pairs of faculty with similarly prestigious training and with similar prehire productivity and prominence (publications, $N=194$ pairs; citations, $N=194$; see Fig. 2 C and D), the individual with the more prestigious appointment produced, on average, 5.1 more papers in the first 5 years posthire ($p<0.005$, $t$ test), with 57.4% of trials exhibiting an advantage of any magnitude ($p<0.05$, binomial test) and significant differences in years $y\in \left\{\mathrm{1,2,4,5}\right\}$ ($p<0.05$, $t$ test). Similarly, individuals with the more prestigious appointment received, on average, 344 more citations in this period ($p<0.001$, $t$ test), although the median difference was a more modest 112 additional citations. For context, faculty at the top 20% of institutions by prestige produced, on average, 17 more publications in their first 5 years and received 824 more citations than faculty at the bottom 20% of institutions, and they produced 9 more publications and received 543 more citations than faculty at the middle 20% of institutions.

Hence, conditioned on an individual holding a faculty position somewhere, we find no evidence that training at a prestigious institution confers any advantage to an individual’s subsequent productivity, while it does lead to marginally significant ($p=0.013$) increase in prominence relative to peers with similarly prestigious faculty appointments. Furthermore, we find strong evidence that the prestige of an individual’s faculty appointment, or correlates thereof, drives both their early-career productivity and prominence. That is, where an individual works—not where they were trained—explains the quantity of their scholarly output, and both environments contribute to their prominence. That both environments enhance prominence is, in a sense, unsurprising given the time-delayed, cumulative nature of citations and the heightened visibility of work originating from prestigious origins (15–17). Nevertheless, this insight also explains why elite institutions train the majority of highly productive scientists, and reap the majority of rewards that they produce: The prestige of an individual’s doctorate is known to drive the prestige of their initial faculty appointment (8, 9), and the prestige of that appointment then drives their early-career productivity (18–20). Additional analyses regarding the sensitivity of these matched-pair experiments and the universality of their results are included in SI Appendix, section B.

Identifying pairs of faculty that match on observable confounding variables, such as their age, gender, subfield norms, postdoctoral training, and prehire productivity and prominence, implies that the influence of these variables on posthire outcomes has been mitigated. However, matching cannot rule out the possibility that hiring committees are sensitive to unmatched variables unobserved here, unrelated to the matched variables, and which accurately distinguish individuals who will be more or less productive. For instance, adjusting for prehire variables, candidates with noticeably better prospects for future funding, more charismatic demeanors, or stronger letters of recommendation may be more productive in the future, and hence place into more-prestigious faculty appointments (21). However, if such attributes predict future productivity, it would be reasonable to expect them to also correlate with observed prehire productivity. Furthermore, studies of initial placements (20, 22) and midcareer relocations (23) in other fields provide evidence that changes in faculty productivity correlate with changes in work environment, suggesting that our results are not driven by peculiarities related to hiring, computer science, or even academia alone (24).

The precise manner by which institutional prestige controls posthire productivity remains unknown (25). Prestigious institutions could create environments that lead to higher faculty productivity through four different mechanisms, based on selection, expectation, retention, or facilitation. Institutions could (i) select inherently more-productive faculty via hiring, (ii) require that all faculty meet high expectations for productivity, (iii) selectively retain more-productive faculty at tenure or other formal evaluations, or (iv) facilitate productivity by providing a conducive working environment. We now investigate the degree to which each of these four mechanisms can explain the observed prestige–productivity effect. Because selective retention may introduce survivorship biases, we focus our next analyses of the selection, expectation, and retention mechanisms on early-career faculty, who were pretenure at the time our data were collected in 2011 ($N=555$; see SI Appendix, section A).

If selection of inherently productive faculty explains the effect, faculty with prestigious appointments should exhibit substantially greater prehire productivity than faculty with less prestigious appointments. Among early-career faculty, there is no significant correlation between prehire publication counts and the prestige of their doctorate ($p=0.067$, $t$ test), indicating that individuals who place into any faculty appointment are similarly productive during their training, regardless of where they trained (SI Appendix, section C). Also, prehire productivity correlates only modestly with the prestige of an individual’s posthire work environment, such that, for every 10-rank improvement in faculty appointment prestige, individuals produced only 0.28 additional papers, on average, over the 5 years prehire. Although faculty hiring does sort individuals somewhat by productivity, differences in prehire productivity are weak predictors of differences in placement, indicating that selection cannot explain the magnitude of the observed prestige–productivity effect.

If departmental expectations explain the effect, then, after joining an institution, an individual’s posthire productivity would move closer to the typical productivity of their departmental colleagues. However, only 39.1% of early-career faculty exhibited productivities in the 5 years posthire that were closer to their faculty department’s median individual productivity than to their productivity in the 5 years prehire ($p=0.99$, one-tailed $t$ test; SI Appendix, section D). This lack of evidence of adaptation indicates that expectations of productivity contribute minimally to the observed prestige–productivity effect.

On the other hand, if the retention of more-productive faculty explains the effect, then (i) relatively low posthire productivity by an individual should predict a failure to be retained, and (ii) more-prestigious institutions should retain fewer faculty than less prestigious institutions. However, retention rates are similar across all levels of prestige ($p=0.96$, ${\chi }^2$ test; Fig. 3): Fully 71.9% of early-career faculty in our sample remained faculty in 2017 at the institution of their initial faculty appointment, and productivity alone is a weak prediction of retention with an area under the receiver-operator curve (AUC) of only 0.62 (SI Appendix, section E). Thus, selective retention of more-productive individuals cannot explain the magnitude of the prestige–productivity effect.

Fig. 3.

Binned by departmental prestige, early-career faculty leave their initial appointments, either for other institutions or for a nonacademic job, at similar rates. Error bars indicate 95% confidence intervals around the means. Dotted magenta line indicates the total fraction of departed faculty from all 205 institutions (0.281).

Mechanisms based on selection, expectation, or retention each provide, at best, weak evidence that higher levels of productivity at prestigious environments simply reflect more-stringent requirements for faculty. These results leave the majority of prestige’s effect on productivity and prominence to be explained by the fourth mechanism, departmental facilitation and its variation with prestige. Under facilitation, the characteristics of a department, such as its location, resources, and organization, enable or constrain the productivity and prominence of individual faculty, through more-specific mechanisms. Using explanatory modeling, we establish a set of relationships between departmental characteristics and differences in scholarly output that represent testable hypotheses for further investigation but offer no specific claims of causality.

For each department, we summarize the typical scholarly output of its individual faculty as the median time-adjusted publication count (26), fractional contribution (publication count divided by number of authors), and citation count (raw and log-transformed), as well as the average fraction of papers that are within- vs. out-of-department collaborations (see SI Appendix, section A). We then estimate the dependence of these 10 variables on 17 or 19 covariates (Fig. 4) that quantify different characteristics of a department’s work environment. These covariates are drawn primarily from the Computing Research Association’s 2011 Taulbee survey of doctoral computing departments in the United States and Canada (27), which documents the sizes of graduate and undergraduate programs, number of administrative staff and nontenure track faculty, external funding, and research space, among other items (see SI Appendix, section A). We also include characteristics derived from our data on faculty hiring, e.g., department prestige, size, and both gender and junior-to-senior faculty ratios, and whether a paid parental leave policy exists.

Fig. 4.

Departmental attributes correlate with productivity measures, suggesting possible mechanisms. The heatmap summarizes the results of 20 regression analyses, describing the relationships between 19 departmental attributes (rows) and 10 measures of productivity across the department’s individual faculty (columns). Each cell depicts the strength of the corresponding standardized regression coefficient, if significant, under two models. Model 1 (top wedge) includes prestige and private covariates, which are omitted in model 2 (bottom wedge).

Because a department’s prestige and its status as private or public are not especially mutable characteristics, we estimate models that both include (model 1) and exclude (model 2) them. Contrasting the corresponding coefficients can shed light on mechanisms that correlate with prestige but may be more easily replicated by lower-ranked institutions. Overall, we find that departmental prestige correlates significantly and positively with 8 of 10 scholarly output variables (Fig. 4), including publication counts, fractional contributions, citations counts, and out-of-department collaborations. In contrast, private status correlates only with early-career contributions and citations. In both models, productivity variables like publication counts and fractional contributions correlate strongly with the number of doctoral students per faculty in a department, and, in Model 2, the number of research faculty (nontenure track). While a greater number of students and research faculty may correlate with greater individual productivity, we find that these additional papers are not necessarily more highly cited. In fact, few departmental characteristics correlate with longer-term citation counts, beyond its correlation with prestige alone (Fig. 4). These departmental data contain no measures of student quality, and characterizing its relevance to faculty productivity and prominence remains an important direction for future research.

Furthermore, department size consistently correlates with individual productivity and fractional contributions (both models). That is, as a department increases in size, individual faculty are, on average, individually more productive. This evident advantage of larger size is tempered by a negative correlation in scholarly output with the sizes of departmental administration, support staff, and the undergraduate program (in model 2). Together, these results suggest the existence of an optimal departmental size—a size at which the fixed costs of operating a department are spread relatively thinly, thereby freeing up a maximal amount of time for scholarship, and the coordination and teaching costs of a large program are relatively low. These results are consistent with past observations of a nonlinear effect of department size on productivity (28–30), although squared department size was insignificant by most measures.

Most departmental characteristics significantly correlate with few if any scholarly output variables. For instance, the composition of a department, in terms of its junior-to-senior faculty ratio or its gender ratio, is largely unrelated to individual productivity, prominence, and collaborations. This null result does not necessarily contradict past studies, which found that departments with more senior faculty tended to be more productive (30), as our analyses focus on early-career individuals. Gender’s lack of significance contrasts with the long history of productivity gaps separating male and female faculty (31–33), suggesting that the gender productivity gap may have narrowed among recent early-career faculty (34–36).

Discussion

The emerging field of the science of science aims to develop a causal understanding of the social drivers of scientific discovery, which will improve the evaluation of and investment in good science. A common assumption is that faculty’s scholarly productivity mainly reflects their scientific skill, which is often assumed to correlate with the prestige of their doctoral institution. Here, we show that this assumption is false: For early-career faculty, the characteristics of their working environment, and not the prestige of their doctoral training, drive their productivity, and the greater productivity of faculty in more-prestigious departments cannot be explained by the preferential selection or retention of more-productive scholars. Separately, faculty prominence is influenced by both training and work environments, allowing individuals to benefit from the prestige of either location.

Hence, where an individual works establishes an environmental mechanism for cumulative advantage, by which prestige in the past gets “locked in” via placement into more-prestigious departments, which directly facilitate greater success. This mechanism indicates a more limited role for doctoral prestige in predicting scientific contributions, and suggests that an individual’s productivity and prominence cannot be separated from their place in the academic system.

The matched-pair analysis used in this paper allows us to quantify the effects of training and placement on productivity and prominence, while accounting for many individual characteristics. However, the possibility remains that unmeasured—and therefore unmatched—variables could account for some of the patterns we observe. For instance, if future productivity is driven by characteristics like charisma or collaboration potential, then universities who base their decisions on these factors may be able to identify and recruit more-productive individuals in a way that our matching cannot detect. Where possible, quantifying additional meritocratic and nonmeritocratic characteristics of individual faculty would be valuable for future studies.

Measurement of additional environmental variables and resources would also be useful to department chairs, deans, and policy makers in their efforts to evaluate the benefits of different faculty hiring or evaluation policies. For instance, our current analyses suggest that it would be unwise to directly compare the productivity of faculty at one university to their productivity at another, due to the confounding effects of environmental differences. However, an obvious, but difficult to quantify, difference between such environments is the graduate students themselves. Understanding how the preparedness of students drives the success of their mentors, rather than vice versa (37, 38), and how student preparedness varies across the prestige hierarchy remains an open but important challenge in the science of science.

More broadly, our findings have direct implications for research on the science of science, which often assumes, implicitly if not explicitly, that meritocratic principles or mechanisms govern the production of knowledge. Theories and models that fail to account for the environmental mechanism identified here, and the more general causal effects of prestige on productivity and prominence, will thus be incomplete. The causal importance of working environment indicates that past findings in the science of science should likely be reevaluated in light of this effect, and future studies should more explicitly account for it.