The Law of Holes

Reuters recently published an article titled “Years after Brigham-Harvard scandal, U.S. pours millions into tainted stem-cell field,” which revealed that the National Institutes of Health spent at least $588 million on stem cell heart research since 2001.¹ The authors of the article, Marisa Taylor and Brad Heath, commented that 43% of the total National Institutes of Health expenditure for stem cell research was awarded, even after numerous concerns had been raised by the scientific community with respect to the validity of basic science that served as the basis for pursuing stem cell clinical trials in humans. In the Reuters article, noted basic scientist Jeff Molkentin commented: “Now that we know that adult stem cells do not regenerate the heart and that past work suggesting otherwise was false, why hasn’t this knowledge traversed its way through the medical and research systems, and why do such studies persist?”¹ As I read through the Reuters article I was reminded of the adage about the law of holes, which states that “if you find yourself in a hole, stop digging.”² The law of holes is an allegorical reminder that when faced with an untenable situation, it is best to disengage from that activity, rather than to continue digging in pursuit of an unsound or unobtainable idea.

The law of holes is rooted, at least in part, in a phenomenon that psychologists refer to as “sunk cost bias,” which describes the tendency to continue to invest in an endeavor if one has have already invested time, effort, or money into it, regardless of whether the cost of pursing the endeavor outweighs the benefit.³ Indeed a prominent stem investigator who was interviewed for the Reuters article remarked: “Why would we give up after so many years and investment (in stem cell research)?”¹ In economic terms, sunk costs refer to costs that have been already been incurred and therefore cannot be recovered. The potential fallacy of the sunk cost basis is that it values an endeavor based on how much has already been invested (ie, its sunk costs), rather than on what the present or future value of the endeavor might be.³ The saga of the supersonic Concorde airplane, developed conjointly by the British and French governments in the 1950s and1960s, serves as one of the more enduring reminders of the perils of the sunk cost fallacy and has been coined the Concorde fallacy.⁴ Long before the wheels went up on the first commercial flight of the Concorde in 1976, it had already become clear to the British and French governments that because of the unanticipated delays and cost overruns of the project that the per-unit cost of building a single Concorde plane was going too far exceed what the aircraft could recoup commercially. Although the supersonic Concorde was heralded widely as a significant achievement in aviation technology and design, it was never viable commercially and the fleet was ultimately retired by Air France and British Airways in 2003.

Not surprisingly, the psychology of the sunk cost bias also creeps into scientific decision making in cardiovascular translational science. As investigators we learn early on in our training that perseverance and problem solving are essential for success. This training also influences the thought processes of scientists as they try to advance their ideas from the laboratory into early phase clinical trials. Because translational clinical studies are smaller by design, they rely on softer surrogate endpoints, such as changes in biomarkers or changes in left ventricular function, which may or may not translate into clinically meaningful endpoints in subsequent phase III trials. Given that there are no formal statistical guidelines for how investigators should handle secondary endpoints in phase Ib and phase II clinical trials, it is tempting to perform additional non-prespecified post hoc analyses by combining or splitting treatment groups or by retrofitting new mechanisms of action to explain efficacy signals observed in exploratory endpoints derived from small sizes. These types of post hoc studies often serve as the scientific rationale for investigators to continue along the same line of research. As noted by Derek Lowe in his blog In the Pipeline: “Our human psychology is always ready to hold out the hope that something will occur to make it all worthwhile.”⁵ In a fascinating article in Science that is particularly germane to this discussion, Sweis et al⁶ conducted a series of experiments in mice, rats, and humans that demonstrated that the amount of time that is dedicated to a task in pursuit of a reward (ie, a sunk cost) reduces the likelihood of giving up that pursuit, even when there is little or no indication that the pursuit will be successful. They suggest that “multiple, parallel decision-making valuation algorithms implemented in dissociable neural circuits have persisted across species and over time through evolution.”⁶

Recognizing that as translational scientists we are prone to factor our own personal/professional sunk costs (eg, time invested, money spent, career reputation) into future decision making, how is it possible to avoid digging a deeper hole, if at some point, we discover that the scientific path that we have chosen is not necessarily supported by robust data? Conventional economic theory suggests that all decisions should be based on valuations of future expectations, and that spent resources, which cannot be recovered, should simply be ignored. This type of decision making is far simpler for large pharmaceutical companies than it is for academic laboratories or small start-up companies. Large pharmaceutical companies have learned to avoid the Concorde fallacy by incorporating economic modeling into all their decision-making processes as they develop new therapeutic targets. Large pharmaceutical companies also have deep pipelines of candidate drug targets and even deeper pockets that allow them to flip the kill switch on projects that are not economically viable. It is much harder for academic laboratories to pivot in a new scientific direction if their translational efforts falter, in part because of the myriad of issues related to funding of grants and contracts, as well as the important issue of supporting the careers of predoctoral and postdoctoral trainees, whose likelihood of getting a future job is linked to the successful outcome of the project they are working on. Similarly, small biotech companies often have limited pipelines and even more limited cash flows. They simply cannot afford to pivot in a new direction if their lead candidates do not pan out. Accordingly, the psychology of the sunk cost bias likely influences the decision making in many academic and small biotech laboratories and may explain, at least in part, why they keep digging, even when the data suggest that it is time to put down the shovel.

In their article, Marisa Taylor and Brad Heath report that the National Institutes of Health issued a statement regarding the $588 million expenditure on stem cell research since 2001, indicating that the funding was approved because it was “supported by a substantial body of evidence” performed in animal studies.¹ Whether this represents a wise investment of public tax dollars that will enhance the health of all individuals so that they can live longer and more fulfilling lives or whether this represents an example of the Concorde fallacy writ large is unknowable at present and will take time to tell.