What’s wrong with single hypotheses?

The classic description of the scientific method begins with devising a hypothesis. The problem with starting with a hypothesis, however, is that bias and self-delusion can arise due to an emotional attachment to the hypothesis, as honestly described by Chamberlin in 1897:

The moment one has offered an original explanation for a phenomenon which seems satisfactory, that moment affection for [one’s] intellectual child springs into existence, and as the explanation grows into a definite theory [one’s] parental affections cluster about [the] offspring and it grows more and more dear …. There springs up also unwittingly a pressing of the theory to make it fit the facts and a pressing of the facts to make them fit the theory…¹

The temptation to misinterpret results that contradict the desired hypothesis is probably irresistible. This mistake occurs repeatedly in the history of science. Some examples were collected by Langmuir,² who correctly called them "Pathological Science."

Peer-review of research can help avoid these mistakes, so long as the reviewers are not in the thrall of the same hypothesis. But if there is a shared enthrallment among the reviewers in a commonly believed hypothesis, then innovation becomes difficult because alternative hypotheses are not seriously considered, and sometimes not even permitted.

To speed scientific progress, Platt³ suggested the use of "Strong-Inference,” which built on Chamberlain’s “Multiple Hypotheses” method. A thumbnail sketch of Strong-Inference is:

1. Identify a scientifically-interesting observation.

2. Enumerate all alternative hypotheses that can account for the observation, based on present knowledge.

3. Reject hypotheses by experimental observations until a single hypothesis remains that has survived an experimental test by which it could have been rejected. The remaining hypothesis is the “currently held view” of the “cause” of the observation.

To the laity, the remaining hypothesis is "truth,” but the scientist knows that this currently-held view can change if new hypotheses arise from new knowledge. The presence of Strong-Inference in publications and grant proposals can identify those areas of science that are rapidly evolving, and if funding were to be shifted towards these areas, progress in science would be more rapid, and efficient.

Yet, despite its logical structure, Strong-Inference does not deal with the following question: What method can efficiently generate the observations needed to start new cycles? Strong-Inference is neither useful nor needed for creating the critical “seed observation” upon which alternative hypotheses can be based. Instead, we need Strong-Inference-PLUS, which consists of three phases:

1. the Exploratory Phase,

2. the Pilot Phase, and

3. the Hypotheses-Testing Phase.

The Exploratory Phase has different goals from – and uses different methods than the other two phases. During the Exploratory Phase the factors that influence success are unclear, so the research must proceed based upon hunches, serendipitous observations, and selections based upon inadequate information. “Pet” theories are certainly acceptable here, if they work. Such “non-scientific” procedures are necessary whenever the parameter space is too large for an exhaustive search. It is here that scientific creativity can play a crucial role. Here, years of experience may enhance – or hinder – discovery. And here is where new scientific instruments or methods may – or may not – demonstrate their worth.

When the Exploratory Phase provides reasonably reliable observations, the research enters the Pilot Phase, during which a small number of experiments – usually 6–8 – are replicated under identical experimental conditions. The Pilot Phase is the experiential test of the Exploratory Phase. A statistical power calculation of the Pilot data will predict the size of study necessary in order for the Hypotheses-Testing Phase to have a reasonable probability of success. (Unfortunately, there is considerable difficulty at NIH of funding Exploratory and Pilot Phases, so these Phases are usually funded out of other, ongoing projects.)

If the analysis leads to further funding, the project moves into the Hypotheses-Testing Phase. None of the experimental data from the Exploratory and Pilot Phases can be re-used in the Hypotheses-Testing Phase, in order to avoid statistical bias. The Hypotheses-Testing Phase will be most efficient when the multiple-hypotheses of Strong-Inference are utilized to guide the experiments.

Quality in Strong-Inference is judged on the efficiency of ruling out alternative hypotheses. Quality in the Exploratory Phase of Strong-Inference-PLUS should be judged on how interesting the Observations are, and especially whether the Observations cannot be “explained” by the “standard view.” The Observations that are not explainable by current scientific theories are the most valuable for they may propel the field forward in the next cycle of innovation, possibly to a paradigm shift.

Strong-Inference-PLUS is a paradigm shift for some scientists. For example, it is common knowledge that R21 "Exploratory Research" grants at NIH are evaluated not in terms of the Exploratory and Pilot Phases, but instead, are critiqued as if the proposal was for R01 "hypothesis-testing" research. Explicitly labeling the Phases of Strong-Inference-PLUS makes it easier for students to identify changes in the approach to experiment and to data as new knowledge is first encountered, then elaborated on, and finally codified.