Personal Reflections on Big Science, Small Science, or the Right Mix

Late June, 1969, at a suburban Philadelphia day camp. There was much to look forward to: color war, soccer, nature hikes, bug juice, barbecues, swimming, softball, and, of course, “no school!” But I paid little attention, as for me these paled compared to the upcoming big event, the much-anticipated launch of Apollo 11, the rocket that would fulfill President Kennedy’s call to land men on the moon and bring them back home. Over the next few weeks, my parents and I spent hours glued to the TV; and my parents quietly endured the smell of glue emanating from model rockets and lunar modules taking form in my room.

Some twenty years later Bernadine Healey, Director of the National Institutes of Health, called for a “moon shot” for women’s health. The resulting Women’s Health Initiative (WHI) would be a “big-science” effort that would invest substantial resources enabling scientists to address questions about hormone replacement therapy, supplementation with Vitamin D and calcium, and diet as interventions that might prevent serious health conditions faced by post-menopausal women. Like NASA’s moon shot, the NIH’s WHI succeeded¹; a national team of top-notch scientists enrolled well over 100,000 women into a number of randomized trials and observational studies. The surprising findings of the relative harms of hormone replacement therapy led to sweeping changes in clinical practice and likely played a role in the last decade’s decline in breast cancer incidence.

Now, twenty-five years later, American science and engineering face an uncertain future. While big-science projects have always generated controversy, shrinking budgets have invited only more criticism. The United States invests a lower proportion of its Gross Domestic Product (GDP) into research and development than a number of other economically developed nations.² Of perhaps greater concern, there has been a steady decline in public support of science; whereas in the 1960s the federal government provided 2/3 of research funds, today it only provides 1/3.² At the NIH there has been a steady decade-long decline of purchasing power. At the NHLBI, this has translated into lower pay lines, with the Institute awarding 36% fewer new R01 grants than it did ten years ago.

Some prominent thought leaders have argued that the NIH and other funders need to seriously re-think their business models.³ Instead of funding a smaller number of big-science projects, projects like the WHI, government research agencies, they argue, should rediscover small science, science that is based on the work of many scientists working in many settings, each getting relatively small sums of money to do their work.⁴ NHLBI Project officers often hear skeptics of big-science initiatives ask questions like, “How many R01’s we will have to sacrifice in order for your big project to happen?”

So what strategy should an agency like the NHLBI take? Should we aim to fund more “small projects,” meaning low-budget investigator-initiated R01s?⁵ Should we trim back on our big-science projects, such as relatively expensive clinical trials and epidemiology studies?⁶ Should we stop funding trials and epidemiological projects altogether, or insist on shifting to low-cost pragmatic alternatives that leverage rapidly evolving information technologies? And what about projects that involve highly innovative, but risky technologies or candidate therapies, where cheap options are few or absent? And if we are to fund a mix of big and small science, to follow the maxim of a “diversified portfolio,”⁷ what’s the right mix? And how do we determine, in advance, which projects constitute the right strategy?

Prominent thought leaders have struggled with these questions. Some, like Gregory Petsko, argue that “the right way to direct science is almost not to direct it all,” but rather allow priorities to set themselves through “the free exchange of ideas in the scientific literature, in meetings, and in review panels.”⁸ Others, like Niki Vermeulen and colleagues, counter that they are “less sanguine about [this] belief that the scientific community alone has the capacity to ascertain the practical value of particular lines of inquiry.”⁹ Stuart Firestein points out that science by its very nature is based on ignorance and that it is nary impossible to predict which technologies and hypotheses will succeed.¹⁰ We recently published a report showing that percentile rankings of NHLBI R01 grants were unable to predict subsequent academic productivity.¹¹

Others wonder whether big-science investments are worth the opportunity costs, the pathways foregone because of monies directed elsewhere. Bruce Alberts, the former editor of Science, worried that laboratory-based investigators are being crowded out of the decreasing funding pool in part because “the scale [of big science projects] creates a constituency that makes these projects difficult to stop, even where there are clear signs of diminishing returns.”⁴ Nobel laureates Joseph Goldstein and Michael Brown worry about an even deeper impact, namely a harmful change in fundamental scientific paradigms. They recently lamented that “individual curiosity-driven science has been replaced by large consortia dedicated to the proposition that gathering vast amounts of correlative data will somehow provide the answer to life’s fundamental questions.”¹² The science economist Paula Stephan summarized the conundrum when she wrote that “we just don’t know” whether it is “better to spend $3 billion on the Human Genome Project or to support 6,000 researchers each to the tune of $500,000?” Mega-projects, like epidemiological cohorts, that “provide inputs for more research down the road” but don’t by themselves provide answers “are especially difficult to evaluate.”¹³

In his AAAS Presidential Address, William Press noted that science is “heavy tailed,” meaning that a small number of efforts will account for most of the impact²; Nasim Taleb describes this phenomenom as “Extremistan,” meaning “a process where the total can conceivably be impacted by a single observation … also called ‘fat tailed.’”¹⁴ At the NHLBI we too observe such a pattern. The Pareto plot in the Figure shows 2-year citation data for 118,070 papers published between 1990 and 2010 and supported by at least one NHLBI R01 or R21 grant. Over 75% of the citations were generated by just 30% of the papers, whereas 40% of the papers generated fewer than 2% of the citations.

Figure.

Pareto plot of 2-year citations for 118,070 papers funded by NHLBI R01 or R21 grants and published between 1990 and 2010. The x axis divides the number of papers according number of 2-year citations. The bars show the sum of 2-year citations within each decile, while the line graph shows cumulative values going from the best- to the worst producing-deciles of papers. The top three deciles (that is, the 30% most frequently cited papers) generated 75% of the citations.

Despite the unquestionable successes of previous big-science investments, I worry that in a time of unprecedented fiscal constraints big-science presents an existential threat to the invaluable offerings of small science. As stewards of scarce public monies, we at NIH have an even more pressing responsibility than in years past to explicitly consider the opportunity costs of new or renewed large-scale projects that diminish our ability to support individual laboratories. How many paradigm-changing, yet wholly unpredictable, discoveries will be lost? We should also worry about the impact on our ability to nurture science trainees, who not only are aware of decreasing chances for funding, but also will have fewer opportunities to be exposed to the “individual, curiosity-driven science” that Goldstein and Brown credited with enabling a series Nobel-winning breakthroughs.¹²

So, what might be the characteristics of worthy big-science projects? As Stephan writes¹³, it may be impossible to come up with definitive answers, given the small sample size – by definition big science projects are few in number – and the uncertainties about how to measure outcome. But thinking back on my experiences, I might offer a few suggestions. There were some prima facie successes, like the Apollo moon shots, the WHI, the Hubble telescope, and the nation’s research attacks on infectious diseases like AIDS and polio.

So, here are six suggested criteria for prospective evaluation of large-scale projects in our current era of ever shrinking resources:

1. Will it capture the public imagination? The Apollo moon shots sure captured my 8-year-old boy sense of wonder. Abraham Lincoln once remarked that anything is possible but only with the backing of public sentiment. Admittedly some highly worthwhile projects, like discovering the molecular structure of DNA, may be difficult to communicate to a lay audience.

2. Is government leadership and financial support critical for success? Some projects won’t happen on their own because no one private party has the financial incentives to embark on them.

3. Is there a clear, measurable and achievable objective? In July 1969 we could say unequivocally that we sent men to the moon and back. Some objectives may be less clear, but the WHI trials were finished and did lead to marked changes in practice, and today AIDS, at least in the United States, is a chronic disease, not the fast-paced killer it once was. Some large-scale projects have enabled the scientific community to better understand where to shift investments; for example, Mendelian randomization studies¹⁵ that leverage discoveries stemming from the Human Genome Project may inform worthwhile and less worthwhile lines of inquiry.

4. Is it—in a timely manner—stretching new technological and organizational capabilities? The moon shot was possible because of recently developed rocketry technologies, the advent of high-power computers, and strong partnerships between the private and public sectors. In today’s parlance, we might ask whether a new project leverages contemporary scientific advances or resources.

5. Is it possible to proceed in measured stages? And to learn from inevitable stumbles and failures? The manned space program started with the small Mercury rockets, then the bigger Gemini missions that sent two men up at once and tested the ability to dock orbiting spacecraft, and only then the giant Apollo rockets. Even so there were bumps, even tragic disasters, along the way. Clayton Christensen, an expert in innovation, urges open-minded organizations to discover and exploit disruptive technologies by planning to fail often, quickly, and inexpensively.¹⁶

6. Irrespective of success or failure, will something be learned and will there be the courage to stop? Or at least recognize when it’s time to scale back so as to allow new efforts with emerging priorities to get their turn. This question is particularly important, but difficult, indeed gut-wrenching, during a time of shrinking resources.

A few weeks ago, NASA posted a video that re-created the events leading to the famous Apollo 8 photograph of earthrise. I enjoyed watching it, and even more so, sharing it and my boyhood recollections with my two college-age sons who are both planning careers in science or engineering. I also shared this essay with them, and, I must admit, that despite the amazing accomplishments of the moon program, I advised for the short-term that they seek training opportunities in the kind of nurturing “individual curiosity-driven” laboratories that Goldstein and Brown benefitted from during their formative years.¹² In the long-term, I hope that they will be able to thrive and contribute to a scientific and technological enterprise that sees its share of great achievements, whether they come from big science, small science, or whatever the right mix is.