How Do Scientific Views Change? Notes from an Extended Adversarial Collaboration

Abstract

There are few examples of an extended adversarial collaboration, in which investigators committed to different theoretical views collaborate to test opposing predictions. Whereas previous adversarial collaborations have produced single research articles, here we share our experience in programmatic, extended adversarial collaboration involving three laboratories in different countries with different theoretical views regarding working memory, the limited information retained in mind, serving ongoing thought and action. We have focused on short-term memory retention of items (letters) during a distracting task (arithmetic), and effects of aging on these tasks. Over several years, we have conducted and published joint research with preregistered predictions, methods, and analysis plans, with replication of each study across two laboratories concurrently. We argue that, although an adversarial collaboration will not usually induce senior researchers to abandon favored theoretical views and adopt opposing views, it will necessitate varieties of their views that are more similar to one another, in that they must account for a growing, common corpus of evidence. This approach promotes understanding of others’ views and presents to the field research findings accepted as valid by researchers with opposing interpretations. We illustrate this process with our own research experiences and make recommendations applicable to diverse scientific areas.

In the hypothetico-deductive method long considered by many philosophers and scientists to be a key to scientific progress (Popper, 1935/1959; 1963), a hypothesis or expectation is tested and, if the outcomes of experiments do not support it, the hypothesis is abandoned and other hypotheses are devised for future testing. Other philosophers, however (especially Lakatos, 1968–1969, 1978) have noted that scientific theories contain many interrelated hypotheses, which can lend a theory multiple ways to explain any one result. Most research in the field of experimental psychology still seems to follow the hypothetico-deductive method. Although we recognize its importance we think it is not enough to reconcile conflicting theoretical views, and we argue that the assessment of broad theories can occur more effectively if proponents of competing views work together in a sometimes tense but productive joint effort that has been termed an adversarial collaboration, whether or not each participant adheres to the hypothetico-deductive method. In this effort, different participating groups work together to collect data jointly, but openly expect (and often hope for) different results. We convey our thoughts about this prospect based on a three-way adversarial collaboration, the longest-lasting one we have encountered, which focuses on theories of working memory in young adults and in cognitive aging. Working memory is the small amount of information that can be temporarily maintained in a readily accessible state and used in tasks such as problem-solving and language comprehension. It is essentially the information held in mind at a particular time, required for the current tasks, and updated moment to moment.

To explain why adversarial collaborations can help, consider that each theoretical framework is based on shared formative experiences and assumptions among a group of scientists, and the shared assumptions within each group affect the interpretation of scientific results (Kuhn, 1962). Alternative interpretations can result in contrasting theories with entrenched positions, without necessarily advancing understanding. Kuhn (p. 148) remarked, “The proponents of competing paradigms are always at least slightly at cross-purposes. Neither side will grant all the non-empirical assumptions that the other needs in order to make its case. …they are bound partly to talk through each other. Though each may hope to convert the other to his way of seeing his science and its problems, neither may hope to prove his case.” We submit that the same is true for proponents of different theories of a body of findings. Consequently, we argue, an adversarial collaboration is beneficial whether one theory is more apt and another, less apt or whether there is substantial value in more than one theory; and it is beneficial whether someone is willing to abandon a theory, nobody is, or some new theory emerges that incorporates elements of each. We have found that it results in useful theory modifications.

This issue of competing views is compounded by the use of different methodologies by proponents of different theoretical frameworks, and sometimes use of the same terms to refer to slightly different concepts, or different terms to refer to very similar concepts (Broadbent, 1984, pp. 86–91; Cowan, 2017). Rarely do individuals or groups with contrasting views collaborate directly using common methods. When such adversarial collaborations arise, they tend to be short-term, culminating in single research articles (e.g., Mellers, Hertwig, & Kahneman, 2001). The results typically can still be accommodated by both of the opposing theoretical views that motivated the research, in which case an extended collaboration could have been helpful.

To compare two or more theoretical frameworks, the effort must be sustained. Most theoretical frameworks are not houses of cards that easily collapse by pulling out a single card (or disconfirming a single hypothesis). For a specific hypothesis, it is plausible that data from a crucial experiment can determine whether one’s view remains tenable or can be falsified using the hypothetico-deductive method. This method is a positivist concept, meaning it comes from sensory input interpreted through reason and logic (Popper, 1935/1959, 1977). However, a single crucial experiment is unlikely to change a theoretical view that incorporates a broad web of hypotheses (cf. Kuhn, 1962; Lakatos, 1968–1969; Newell, 1973).

Nor can different theoretical views be compared simply by determining if one view is more parsimonious (simpler) than the other. Of course, it is the case that in deciding among competing theoretical explanations of a phenomenon, one guideline is to use Ockham’s Razor (Sober, 2015. That is the notion that the preferred explanation is the one that requires the fewest explanatory principles. In practice, though, this notion often cannot be used to adjudicate which of two views is simpler (i.e., more parsimonious) because the range of relevant phenomena is in question and because, to some extent, parsimony is in the eye of the beholder. For example, pertaining to our own project that we will describe, is it most parsimonious to postulate two separate modules (self-contained systems) in the brain for visual and verbal working memory, respectively, even if these modules appear to operate according to some similar rules? Or, alternatively, is it more parsimonious to postulate a single, general mechanism that holds information regardless of its visual or verbal nature, such as the focus of attention, even though that mechanism has to be more sophisticated if it is to operate across materials like that? The notion of parsimony alone cannot resolve that sort of dilemma because researchers differ on which account of working memory they find most parsimonious.

To address the field’s need to find ways to resolve differences in theoretical viewpoints, we first provide an orientation to the relevant scientific principles, followed by a brief, recent history of adversarial collaborations and their limits. Then we describe an extended adversarial collaboration we have carried out among our three research groups. With our experiences in mind, we elaborate further the scientific principles and reach some recommendations for future collaborations.

Potential Outcomes of Adversarial Collaborations

One outcome of an adversarial collaboration is to change theories to become more accurate by presenting critically important data to be accounted for. Changes of a theory to accommodate new data can be either useful, if the changes are principled, or counterproductive, if the changes are makeshift and awkward for the theory. We believe, though, that adversarial collaboration is helpful in either case, in different ways.

To elaborate on what can happen when scientific theories need to change, Lakatos (1968–1969) distinguished between theoretically progressive and degenerative theory-testing paths. In the progressive path, the data lead to modified versions of theories that remain useful in accounting for a body of evidence, including the new evidence. In the degenerative path, the data lead to modified versions that are increasingly awkward and improbable, with new auxiliary assumptions added only to protect core assumptions of the theory from falsification. The desirable path, of course, is the progressive one. In practice, each theorist may think that their own path is the progressive one and that some alternative theorists are taking a degenerative path. This can occur, for example, if the theorists do not consider all of the same evidence to be valid, important, and sufficiently general or applicable across situations. In the long run, however, an extended adversarial collaboration may help to overcome this problem of how we perceive one another’s theories by increasing each investigator’s understanding of the opposing views and, ultimately, by presenting the collaborative group’s progress to the world for the judgment of other scientists.

Scientists observing the adversarial collaboration may encounter several alternative possible situations. Perhaps one of the theories clearly fits the evidence, whereas the others clearly do not; in practice, though, we believe that this outcome rarely occurs, at least in a complex field such as psychology. The reason is that alternative theories are often flexible enough that a theorist can propose plausible alternative versions of a theory to accommodate new evidence. Alternatively, multiple theories can fit the evidence and, hopefully, researchers can envision a way to resolve the theoretical ambiguity in follow-up research. The theories might evolve in a way that points toward some intermediate theoretical solution that includes some elements of more than one of the original theories. Perhaps some theories can evolve with the evidence on a progressive path, whereas other theories are on a degenerative path and should be abandoned. Even if theorists within the collaboration continue to disagree, the products of the collaboration do, we believe, help to indicate to the field what the true situation is, inasmuch as the opposing views are now applied to a common data set emerging from agreed-upon methods.

Having articulated the general aim of adversarial collaborations, we now illustrate the merits and pitfalls of such collaborations in practice by assessing actual cases that have already occurred, including our extended, three-way collaboration. The lessons learned can steer future collaborations. What has gone right and what has gone wrong in our collaboration, and what guidelines might we provide?

Prior Adversarial Collaborations and their Limits

It seems likely that if two investigators have different worldviews that lead to different predictions for a particular kind of experiment, they may be naturally motivated to design the experiment in different ways. Each investigator would expect to obtain confirming evidence; the important decision to design experiments to collect potentially disconfirming evidence is less pleasant and may often be avoided. Designing one’s experiment in a manner that makes it too favorable to one’s own theory can be unintentional and can occur because humans, including scientists, are affected by considerable confirmation bias in which they seek to verify rather than disprove their own ideas (e.g., Lilienfeld, 2010; Nickerson, 1998; Stanovich, West, & Toplak, 2013). Believers and disbelievers in a particular phenomenon thus may have a history of testing it in different ways, with subtle methodological differences that are more important than one or both camps realize.

One way to overcome this issue of entrenched approaches is for investigators who disagree strongly on theory to work together to agree upon a method, and carry out the experiment(s) jointly. In one early example, the editor of Psychological Science, John Kihlstrom, reacted to a commentary by Michel Treisman on work by Cowan, Wood, and Borne (1994) on evidence for the existence of short-term memory (a small amount information saved temporarily in a manner separate from the vast amount of information in long-term memory). The editor suggested that he would publish a new empirical study in which the researchers worked together to resolve their differences. After they conducted and analyzed their agreed-upon experiments, though, they still could not agree on how to interpret the results, and Kihlstrom suggested splitting the discussion section. Cowan decided, however, that the available space for discussion was too short to be split effectively. Instead, the authors compromised on what to say in the resulting collaborative article (Cowan, Wood, Nugent, & Treisman, 1997). The reward of a prestigious publication made compromise easier.

Mellers et al. (2001) carried out perhaps the first adversarial collaboration under that rubric, with research on seemingly illogical judgments. For example, the statement Linda is a bank teller and is active in the feminist movement is often judged by participants to be more probable than the statement Linda is a bank teller, though that is logically impossible because a subset (bank tellers who are active in the feminist movement) cannot be more frequent then a more general set containing it (bank tellers regardless of other traits). The researchers set ground rules in which each side of the debate (represented by Hertwig versus Kahneman) was allowed to design one follow-up experiment, so a total of 3 experiments were conducted. In the end, the investigators still did not agree on the interpretation. Instead, they described what they did agree upon, followed by separate discussion sections with different interpretations by Hertwig and Kahneman. Essentially, Hertwig thought that participants tend to interpret the sentences linguistically in a way different from what was intended, whereas Kahneman thought that participants tend to make fundamental logical errors. Together they concluded (p. 275) that “Our joint efforts demonstrate the benefits of adversarial collaboration as a method for conducting scientific controversy. The major benefit is that both parties are likely to recognize limitations of their claims.” We see it as a special virtue that ground rules were set for the collaboration, but it seems a shame that these ground rules also tended to terminate the collaboration. A good compromise might be the 3-experiment rule for one article (at least, unless and until reviewers request additional experiments), fairness and symmetry in the plans if they have to be modified, and plans to continue working in this mode for additional articles.

Matzke et al. (2015) carried out an adversarial collaboration on the effect of horizontal eye movements on free recall (recalling items that had been presented in a list, with the words recalled in any order that the participant wishes). This work involved preregistration of methods and expectations: listing these experimental details online with a time stamp that cannot later be altered, to help ensure that the investigators’ memory or expression of their predictions could not change with the benefit of hindsight after the data were collected and analyzed. Changing one’s views after seeing the data is potentially a very constructive process in building theories, provided that an a posteriori account is not mistaken for an a priori prediction in support of a particular theory. In this case the investigators agreed that horizontal eye movements did not affect free recall in the study, which some of them had expected. Still, the investigators disagreed on the general outcome to be expected with future variations in the methods. The main methodological advance in that study may be the concurrent preregistration of not only the method, but also conflicting expectations. We hope that the collaboration continues.

Oberauer et al. (2018) carried out a joint effort to identify phenomena that are well-established in the area of working memory in order to make the statement that any fully adequate theory of working memory must account for these phenomena. This effort should be considered an adversarial collaboration inasmuch as the many participating authors held very different theoretical orientations, carried out two successive conference meetings to discuss the rules for inclusion versus exclusion of phenomena, and reached a general agreement. In this case, the issue of different theories was circumvented by trying not to discuss theories per se; there was more agreement on how to identify replicable phenomena than there was on how to arrive at the correct theory. It seems laudable to break down a tough problem (how to agree) into an easier part to be attacked first (agreement on phenomena to be included) while omitting a harder part to be attacked at some future point (agreement on theory). Even the statuses of the phenomena identified in the article and omitted from it, however, are not universally accepted (e.g., Logie, 2018; Vandierendonck, 2018).

Our Extended Adversarial Collaboration on Working Memory

Overview

Our own adversarial collaboration arose from an attempt to resolve apparent empirical discrepancies between laboratories studying working memory. To do so, Logie, Camos, and Barrouillet settled on the idea of requesting grant funding to work together to resolve the issue. Cowan was added to the collaboration as he had another relevant theory and was already working with Logie on several related projects despite holding different views: a special journal issue on working memory introduced by Logie & Cowan (2015), and a dissertation committee resulting in joint publications (Rhodes, Cowan, Hardman, & Logie, 2018; Rhodes, Cowan, Parra, & Logie, 2019; Rhodes, Parra, Cowan, & Logie, 2017). Naveh-Benjamin, who has worked with Cowan on working memory and aging (e.g., Cowan, Naveh-Benjamin, Kilb, & Saults, 2006; Gilchrist, Cowan, & Naveh-Benjamin, 2008; Naveh-Benjamin et al., 2014), was included on the grant proposal to help us enrich our comparison of the theories via research on cognitive aging.

We will first describe our collaboration in enough detail to convey a feeling for what it is like to work in a collaboration of this sort. The purpose of presenting it is to allow readers to understand the features of collaboration that, we believe, have made it work well. We summarize these features in Table 1 and recommend that other investigators planning adversarial collaborations adhere to as many of these points as possible. With some careful soul-searching, at the end of our description we also try to identify potential shortcomings of our collaboration that might be improved upon in the future.

Table 1.

Important and Helpful Characteristics of the Present Adversarial Collaboration

Characteristic	Importance
Three theoretical views represented	Goes beyond binary decisions. Possibility of Views A, B aligned on some issues but B, C aligned on other issues.
Three laboratories reflecting the different theoretical views	Possibility of replicating and comparing effects across laboratories representing different theoretical views.
Concurrent replication of each study in two laboratories	Allows a comparison of the same method and analyses across laboratories coming from different theoretical orientations.
Definition of terms in discussions (e.g., working memory; small effect size)	Different theoretical views often come with different implied definitions of terms, and this must be made explicit in order to determine when views conflict and adjudicate between them.
Listening for meaning	Important to go beyond what investigators say to understand what they are trying to say, given different vocabulary usage.
Reformulation of theories after listening for meaning	Important to use the feedback to articulate each theory more precisely, for better understanding and communication.
Common methodology (cognitive behavioral testing; verbal and graphic theory)	Possible to discuss specific issues and plan joint experiments without getting bogged down in discussions about what methodology is legitimate.
Preregistered methods, predictions, analysis plans	Make it difficult to overlook the fact that the results do not match the predictions, when that occurs.
Bayesian inferential statistics	Possible to find support both for and against a hypothesis, always important but especially so when some theoretical views predict null effects.
Funding for research with excellent postdoctoral fellows across several years	Incorporation of new methods not always familiar to senior researchers; early career colleagues who are not as committed to one view; fair amount of consistency in personnel over years. Financial support for regular meetings, exchange visits, and joint commitment to complete the planned research program so to be accountable to the funding agency.
Pre-existing working relationships between the senior investigators (e.g., Logie & Cowan, 2015)	Less risk in trying to work together despite differences in opinion, given that this work has tended to succeed and come to completion in the past. It is also helpful if the investigators can socialize and not take professional disagreements personally.
Extended visits of postdoctoral fellows between laboratories	Cross-fertilization of different methodology and theory, helping to bridge the gaps between laboratories. Enhances post-doctoral experience for career development.
In-person and electronic meetings with all collaborators	Frequent discussion of the issues that are most important in a timely manner to inform and guide the research.
Postdoctoral senior authorship on most of the articles	Sets up the adversarial collaboration as a high priority among collaborators while advancing everyone’s career.
Incorporation of multiple situations (e.g., aging research along with research on young adults in our work)	Can add a dimension on which the theories have not previously been compared, and can make it likely that the outcome of the work will be of practical, in addition to theoretical, value.
Researchers encouraged to propose ways they could explain new results from their respective theoretical viewpoints	Provides a path beyond the usual arguments about whose theoretical viewpoint is more appropriate, toward a situation in which each theoretical orientation can evolve, given new evidence. Ideally, the theoretical views are drawn closer together in a way that works for the investigators.

Process of Collaboration and Basic Method

In our own collaboration, each of three groups favors a different theoretical view of working memory. Many investigators have long felt that, to get a comprehensive measure of a person’s working memory, one should combine items to be remembered with problems to be solved, termed processing episodes (e.g., Case, Kurland, & Goldberg, 1982; Conway et al., 2005; Daneman & Carpenter, 1980). The reason is that this method engages both storage of items in memory and other mental activities known as processing, in order to indicate what the participant is able to remember while also doing mental work concurrently. Processing episodes leave information in a different form than that in which it had been originally encountered; depending on the assigned task, letters that had been presented in random order might be repeated by the participant in alphabetical order, presented numbers might be added together, sentences might be comprehended, and so forth. A procedure with processing episodes between items to be remembered is often called a complex working memory span task.

Here we use a simpler arrangement that we term a storage-then-processing task, in which all of the items in a list to be remembered on a trial are presented, followed by episodes of a separate processing task, and then by recall of the items in the list (first used by Brown, 1958 and Peterson & Peterson, 1959). We use this task in order to create a situation in which we can observe the effects of memory on processing and vice versa (dual-task costs) without requiring multiple switching between the two tasks in order to take in all of the materials.

The basic reason to examine dual-task costs is that the multicomponent theory (e.g., Logie, 1995, 2016) predicts that these costs should disappear when the tasks are adjusted to match the participant’s ability level, whereas the other two theories (e.g., Cowan, 1988, 2019; Barrouillet & Camos, 2015, in press) predict that the costs should remain under those conditions. After explaining the tasks more fully, we will explain the theories in relation to these tasks more fully.

In the storage-then-processing task that we have used (Doherty et al., 2019; Rhodes et al., 2019), letters are presented on a computer screen, or are spoken, one at a time, with simple arithmetic problems on the screen during a 10-second period following the last letter. After the period of arithmetic problems has ended, the letters are to be recalled. On a particular trial with three letters for the sake of illustration, with two slashes (//) indicating a new computer screen display, a trial might look like this: X // B // Q // 5+7=11? // 4+5=9? // 3+5=7? // 3+6=9? // 2+8=12? // 4+7=13? // Recall the letters. Correct answers to the arithmetic questions (no, yes, no, yes, no, no) are to be made on a button box quickly as the problems appear on the screen, and then, after the instruction to recall, the memory answer (X, B, Q) is to be made on the keyboard or is spoken aloud, depending on the experiment. Both letter memory and arithmetic are scored for correctness.

Theoretical predictions for the multicomponent approach depended critically on the performance level. To stabilize it, initially the number of letters in the list and the number of arithmetic problems in 10 seconds are both separately adjusted for each participant to achieve an estimated 80% correct, and these levels of difficulty are used in separate and combined tasks. Although the embedded-processes and time-based resource sharing approaches did not require this kind of adjustment, proponents of all three approaches agreed that it was a useful refinement of the method emerging from the collaboration.

Our focus has been on whether very different tasks, such as letter memory and arithmetic, still share some mental resource that must be split between them compared to when only one task is required, memory or arithmetic. If the tasks share a common resource under these conditions, then the storage-then-processing task should result in poorer memory for the letters and less accurate arithmetic responses compared to when the memory task is presented alone or the processing task is presented alone.

Our collaboration is unusual in coordinating the efforts of three laboratories with different theories and predictions. Through this type of collaboration, we have made progress by obtaining results using mutually-agreed-upon methods. The predictions and methods are preregistered for most experiments and each finding is examined in parallel in two of the three laboratories. Our experience in this collaboration indicates that, under these circumstances, differences between theoretical views have not been eradicated and, indeed, remain rather entrenched. Nevertheless, we advocate extended adversarial collaboration as a path toward scientific progress because details of each theoretical view tend to shift gradually in response to the data. The new, jointly-collected data push on the theoretical accounts. Provided that the theorists trust these new data, which we happily have found to be the case, the views must be constrained so as to be capable of accounting for the new evidence. The resulting changes to the theories can create areas of new overlap between the different theories.

Three Views in Competition, Illustrating the Need for an Adversarial Collaboration

It is clear that there are important limits on how much information one can keep in mind, and that such limits importantly influence the quality of comprehension and problem-solving (e.g., Cowan, 2001). Different groups have traditionally proposed different fundamental causes of the limit in working memory. The theories make different predictions as to what should be expected in storage-then-processing tasks like the one described above. In addition to contrasting our predictions for this kind of task in young adults (Doherty et al., 2019), we also applied the theories to changes in storage-then-processing performance with adult aging (Rhodes et al., 2019).

The multicomponent theory.

The key feature of a multicomponent theory is that, within working memory, information about speech sounds (whether derived from actual speech or from printed language), termed phonological information, is temporarily stored in one brain module (or mental process), whereas visual and spatial nonverbal information is temporarily stored in another brain or mental process. These modules have been termed the phonological store and the visuospatial sketch pad, respectively (e.g., Baddeley, 1986; Baddeley & Logie, 1999). The latter has been subdivided into more specialized mechanisms for storage of static visual material versus movements or pathways (Logie, 1995, 2016).

In the first extended account of such a theory (started by Baddeley & Hitch, 1974; completed by Baddeley, 1986), there was also a system called the central executive, for making decisions about how and when stimuli that have been perceived are entered into one or the other kind of storage (or both kinds at once), or when or how the information is altered or recalled. It could be altered, for example, if the task were to recall letters not in the presented order but in alphabetical order. The present multicomponent theory differs from the classical ones in that it is assumed that the central executive is the name for multiple specialized systems that comprise a set of different mental tools that can operate in an integrated way in the healthy brain to support task performance, and can be impaired selectively following focal brain damage, but have not yet all been clearly identified on the basis of research (e.g., Logie, 2016; Logie, Belletier, & Doherty, in press). The other two theories (described below) also refer to executive processes, with less confidence that the field has enough knowledge to say whether it is composed of closely related or separate components.

The time-based resource sharing theory.

According to a second theory, the time-based resource sharing theory (e.g., Barrouillet & Camos, 2015, in press), information in working memory has to be maintained, lest it be lost as a function of the elapsed time; that is, it decays. There are two ways by which this maintenance occurs: by covert verbal recitation, or rehearsal, and by using attention to keep items active, a process termed refreshing. Then the persistence of an item in working memory hangs in the balance; if it decays to a certain low point, the representation of the item in working memory can no longer be revived, and will not be recalled, on that trial; thus, the speeds of rehearsal and refreshing matter, and when they are used matters.

The embedded-processes theory.

The third theory includes the notion that there is a way to hold a limited amount of currently important information in working memory by paying attention to it. One expression of this view was provided by William James (1890) in his description of primary memory, the trailing edge of the present moment in consciousness. The embedded processes view accepts the idea of a primary memory, in particular limited to retaining about 3 independent items or thoughts in the typical adult, held in the focus of attention (Cowan, 1988, 2001, 2019).

This view is termed “embedded” because the focus of attention does not act alone; it is a subset of a larger set of information that surrounds it, consisting of unorganized features of experience (colors, line orientations, sounds, tastes, meanings, and so on) that have become activated, or especially accessible to attention, through recent experiences and associations to those experiences. Activation lasts until these features decay beyond a point of no return (as in the time-based resource sharing theory) or until other, similar features of more recent stimuli cause too much interference when they are perceived. For example, a printed word might have orthographic activated features indicating how it looks in print, along with phonological features indicating how it sounds. If a spoken word is presented, it will have vivid phonological features and there is some chance that it will overwrite or erase the phonological features of the aforementioned printed word, making that word less active and recallable than it had been. The activated information is, in turn, a subset of all the information that is in the person’s long-term memory. New associations between items, such as those needed to keep in mind the trigram BLB, are formed in the focus of attention and are learned rapidly enough to be of immediate use. This new learning of the sequence becomes a part of long-term memory that is still in an active state for a while after its initial learning, allowing repetition of the trigram if it is desired or use of that trigram in further thinking and processing.

Competing Predictions Examined

The most basic predictions that have been tested in our collaboration are related to what should occur when participants receive a storage-then-processing task with the difficulty of each task having been set separately to approximate the individual’s ability level and not exceeding that level. For example, in one of our joint experiments (Doherty et al., 2019), we asked participants in the single-task conditions to remember a short, random sequence of letters over an interval of 10 seconds, and then to try to recall the sequence. We also asked participants to carry out a series of simple arithmetic verifications (e.g., 5+6=9, True or False?), and to complete as many of these as possible in 10 seconds. Finally, participants were asked in the dual-task condition to remember a random letter sequence then to complete arithmetic verifications for 10 seconds before recalling the letter sequence. The number of memory items in a list and number of arithmetic tasks in 10 seconds were set to where the participant was about 80% correct on these two tasks carried out separately. Theorists from each camp were asked to predict what the data would look like, based on their theories.

Under these conditions, if we find that memory for the letters or accuracy on the arithmetic verification drops when the tasks are performed together compared with being performed separately, then this is referred to as a ‘dual-task cost’, that is, a ‘cost’ in performance when the two tasks are combined. The three theories differed regarding the conditions under which a dual-task cost might or might not appear. As reported in Doherty et al. (2019), there was little to no change in performance on the arithmetic task whether participants were, or were not, asked to remember a set of letters at the same time (i.e., little or no dual-task cost). This was consistent with the multi-component theory, which predicted that behaviors controlled by separate brain modules for memory and arithmetic would not interact. It was not consistent with embedded processes or time-based resources sharing theories, for which attention should have to be divided between memory and arithmetic. However, there was a decline in recall of the letter sequence when participants had to perform mental arithmetic in between seeing the letters and recalling them, compared with doing the memory task without interpolated arithmetic (i.e., a dual task cost). This was consistent with embedded processes and time-based resource sharing, but not with multi-component theory. So, no one theory predicted all of the data patterns, but each theory predicted some of the data obtained.

We further attempted to distinguish between theories based on an extension of research to changes that may occur across the adult lifespan, ages 18–81 years (Rhodes et al., 2019). We screened individuals to exclude from the participant sample those with mild cognitive impairment or dementia. Here, not only did we compare performance on memory for letters alone and performance on mental arithmetic alone with both tasks when they were combined to form a storage-then-processing task; we also asked participants to prioritize one task or the other when performing them together, based on the number of points awarded for each task. In this instance, the time-based resource sharing theory predicted, as before, that there would be a dual-task cost to performance, but this theory predicted that the dual-task cost and the ability to prioritize the tasks would be the same regardless of age. This was because the difficulty of each task, performed as a single task, was adjusted according to the ability of each participant, and this should cancel out individual and age differences in the ability to combine the memory and processing (arithmetic) tasks. The same was true of the multicomponent theory, based on evidence that two tasks can be performed together with little or no drop in performance of either task at any adult age in the absence of dementia or other neuropathology (e.g., Kaschel, Logie, Kazén, & Della Sala, 2009), and that the dual task effect should not change according to whether one or other task was prioritized when performing them together (e.g., Logie, Cocchini, Della Sala, & Baddeley, 2004). The embedded-processes theory, on the other hand, predicted that letter memory and arithmetic were expected to interfere with one another when the two tasks were carried out on the same trial. Also, it was expected by this theory that the young adults would be better than older participants at adjusting the relative priorities to letter memory or to mental arithmetic. The results showed that both younger and older adults were equally good at prioritizing the two tasks, consistent with time-based resource sharing and the multi-component theory, but with increasing age, the size of the dual task cost increased, consistent only with embedded processes. In sum, once again there was no one theory that perfectly predicted the outcome of the experiment.

From Doherty et al. (2019) and Rhodes et al. (2019) taken together, it seems clear that a successful theory of working memory will look a bit different from any of the three theories (or, indeed, any current theory) and might incorporate elements from all three theories. It is interesting that, for the young-adult study of Doherty et al., the predictions of the time-based resource sharing and embedded processes theories were most similar, whereas for the potential aging effects of Rhodes et al., the predictions of the multicomponent and time-based resource sharing theories were most similar. This realignment of theories from one situation to the other shows one potential benefit of having more than two theories to work with, which encourages subtle thinking about details of each theory rather than simpler oppositional thinking. Moreover, the experiments within this collaboration prompted the need to generate predictions in situations not previously considered. For example, the time-based resource sharing and embedded process theories were adapted to make new predictions for adult aging.

The challenges to each of the theories from these jointly generated data patterns are prompting the development of minor modifications to each theory that do not make the three views identical, but they do lead to more similar predictions. They also lead to additional questions for study, and we continue to pursue experimentation to distinguish between these theories or their reformulated versions, or perhaps to establish a compromise model that all can accept (see updated discussions in Logie, Camos, & Cowan, in press).

Observed Limits and Benefits of Collaborating

The conclusion that no current model is to be judged adequate based on these results is one reached by consensus in order to agree on a general discussion section for the paper. We suspect that, if these same results were collected by any one of our groups, that group’s discussion would be tilted much more in the favor of the group’s theory. We take this to be human nature; for various reasons, investigators do not easily abandon their favored theory. Even the design of the experiment, as well as the argument in favor of one theory, might be bolder in one group working alone, and the curtailing of bold arguments may be considered a drawback.

Through the collaboration, however, there have been many distinct advantages, we think outweighing any disadvantages, especially because it is still possible for each group to carry out its own research. Advantages include but are not limited to the following. (1) In each experiment that we have conducted, we have managed to agree on an experimental design including many procedural details. Consequently, we claim that we all basically trust the results and can turn our attention to arguing about the theoretical interpretation and the best follow-up studies to be conducted. This situation is beneficial compared to the often-found situation in which proponents of different theories advocate different methods. (2) In the attempt to account for each common data set on the basis of each theory, the theories probably must become more similar to one another, gradually as evidence accumulates. This gradual change in the theories does not depend upon voluntarily making them more similar, only upon efforts to account for common data, which theorists from opposing camps sometimes do not do. More usually, some key phenomena are accounted for on the basis of auxiliary types of evidence that can differ from one theory to another (e.g., a stronger focus on neuropsychology by multicomponent theories, versus neuroimaging by the embedded-processes theory). Instead, in our articles published in common, each theoretical camp often found itself struggling to account for data that it might actually have preferred to ignore. (3) Even though the contending theorists may not be able to agree, the collaboratively published research can serve as a forum that others in the field, not so committed to any particular view, might use to arrive at a more informed opinion based on what was found and what was claimed about it.

A more complete compendium of important points about the present, extended competing collaboration are shown in Table 1 and can be used to understand how, in the process of holding each other to high standards, we benefit from this collaboration. These are features that we recommend as important or helpful for any adversarial collaboration.

In sum, we feel that it is no small achievement to have succeeded fairly well in the agreement on findings in order to allow us the luxury of having a solid basis on which to mull over and debate theories. Carrying out agreed-upon research with methods and predictions preregistered cannot usually produce immediate theoretical converts, but it can achieve several things: (1) It can help to clear up misunderstandings about one another’s theories, and can help to point out to researchers inconsistencies or ambiguities in our own theories. This interaction can lead to more carefully-stated, specific statements of each theory. (2) It can make the leading varieties of the opposing theories more like one another, at least in so far as is needed to account for the jointly-collected evidence. (3) It can force complications in the models that make some or all of them less elegant, reducing their magical appeal and turning attention more toward actual adequacy in a variety of situations. (4) It can serve as grounds to generate interesting ideas for new experiments that might be well-positioned to force further changes in the theories, in the process of trying to choose among them. Ideally, (5) it would result in a new theory that includes the most successful aspects of each theory. Although we do not believe that we have reached that point and do not know if that goal is realistic, it seems worth striving for.

Further Thoughts about Collaboration in Hindsight

Stasis and change of theoretical views.

Why does an investigator adopt a particular view, and what persuades her or him to change that view? An answer that works well for a simple hypothesis does not seem to work for an extensive theoretical framework, which may result from a lifetime of experience and investment in certain ideas. To illustrate this point in a second domain of inquiry, consider the issue of whether eye witnesses to a crime can display reliable, high-confidence answers in a police lineup (Wixted & Wells, 2017). Without going into detail, we would note that the situation of a police lineup can vary in terms of what the witness is told in advance, how the suspects are presented (one at a time or all at once), who the non-suspected volunteers added to the lineup are (e.g., people similar to the suspect or not), and other factors. Let us refer to some hypothetical conditions of the lineup as Situations A through E that are ordered from most to least supportive of accurate memory. Suppose that an investigator has a theoretical view in which eye witnesses are rarely reliable. When a prediction must be made, the investigator may predict that adequate reliability should occur in Situations A and B but not C, D or E. A finding that there is, actually, reliable judgment in Situation C clearly contradicts one of the investigator’s hypotheses. This contradiction alone, however, would not necessarily change the investigator’s theoretical point of view fundamentally; with a little fine-tuning, its main premises might survive, altered slightly to predict that reliable memory should occur in Situations A, B, and C, but not D or E. For example, Conditions B and C might differ only on how many people are included in the lineup, and the theorist might be able to change the theory to allow a more efficient use of memory to consider all of the suspects in the lineup. The investigator may wish to preserve the theory with the minimal change because it seems consistent with many other ideas that that investigator strongly supports, or findings upon which the investigator often dwells. It seems fine for each investigator to try to preserve their favorite theory (provided that the process does not become degenerative in the sense of Lakatos 1968–1969, mentioned earlier), inasmuch as the process of seeing how far each theory can or cannot go advances the field and allows better comparisons of theories by readers and listeners to the theorists presenting findings together within a competing collaboration.

Settling large issues.

We suspect that the issue of the difficulty of deciding among theoretical frameworks is much broader than our own area, and afflicts most scientific endeavors (cf. Lakatos, 1968–1969; Newell, 1973). Part of this diversity of opinions could be a disagreement about actual facts, given conflicting findings. In our area, for example, no agreement has emerged on the best theory of working memory, or even on the best definition of it (Cowan, 2017), despite almost 50 years of experimentation on this topic in the field of cognitive psychology and the pertinence of working memory to very diverse cognitive tasks (e.g., see Baddeley, 2007; Baddeley & Hitch, 1974; Cowan, 2005/2016; Logie, 1995, 2016; Logie, Camos, & Cowan, in press). However, in our area, as noted earlier, progress has been made by establishing an extensive set of findings upon which many investigators do seem to agree (Oberauer et al., 2018; but see Logie, 2018; Vandierendonck, 2018).

By analogy, is your theory of Person A that she is basically a good person or bad? If good, suppose that person does something bad. You could switch your theory, or you could suppose that this good person was just having a bad day. Similarly, in our own area of research described earlier, multicomponent theorists can suppose that ideal conditions were not achieved for observing the fundamental absence of dual-task costs when verbal memory is combined with nonverbal processing within a storage-then-processing task, or the time-based resource sharing or embedded-processes theorists can suppose that conditions were not ideal to observe dual-task costs in both memory and processing in this situation. More varied and extensive observation is needed to support or disconfirm a theoretical view.

To understand how science might move forward beyond the individual experiment, it helps to adopt a view proposed by Kuhn (1962) and further developed by Lakatos (1968–1969, 1978). In this view, a theoretical stance is unlikely to be overturned by the results of a single, crucial experiment. Instead, the investigator brings to the scientific endeavor a worldview based on a large number of formative experiences and fundamental beliefs and biases, which resist change. The view can evolve slowly, but usually does not change radically except after considerable, varied evidence has accumulated. When two investigators with very different worldviews see the same evidence, they can have different favored interpretations, and it is often not an easy matter to discern whose worldview is most appropriate to account for all of the available evidence.

In our own adversarial collaboration with preregistration of methods, predictions, and analysis plans, based on three different theoretical views in three countries carrying out research together for several years, we hope that we can get beyond agreement on the results. We aspire to reach a point at which our theoretical views will at least begin to shift toward one another as the corpus of jointly-obtained findings and publications increases. A new view could result.

Lessons for Best Scientific Practices

Maintaining a Useful and Practical Attitude Toward Collaboration

Provided that we can continue to come up with test situations that differentiate our views, we endorse the suggestion (Lakatos, 1968–1969, 1978) that a progressive and feasible option isto build on and modify existing theories, based heavily on past evidence, to incorporate data patterns that the theories cannot currently explain. In the field of cognitive psychology, Newell (1973) wrote that “you can’t play 20 questions with nature and win,” and the concern he had then still rings true. He meant that progress in cognitive psychology cannot be made by examining separately various basic, binary oppositions, like the existence of capacity limits or not, the existence of decay or not, or parallel versus serial processing (or, we would add, the conditionsin which dual-task costs are found). He lamented that years of such testing did not add up to a coherent model of how participants operate on a variety of tasks. If Newell’s concern were not on target, we would have agreed-upon answers to the questions by the time of this writing, nearly a half-century later.

Newell’s suggested solution to the problem was a comprehensive, computational or computer model of the entire information storage and processing system, so that theoretical models of various tasks could help constrain one another. It is possible to construct such holistic systems, but computer-based models still produce multiple, alternative solutions (e.g., Anderson, 1983; Laird, 2012; Newell, 1990; Taatgen & Anderson, 2008) based on different fundamental assumptions. We are proposing a somewhat different scientific process in which the emphasis is on forcing the modification of general, opposing theories held by different investigators through joint data collection, inevitably making those theories more like one another if they are to account for the new data along with the old. Given that the effort appears to be succeeding, it seems premature to invest the time it would take to formalize each theory computationally. We are able to make progress despite the fact that individual investigators tend not to abandon their general theoretical views formed over a lifetime.

Treating Theoretical Views Carefully

When individuals in a close working relationship argue or debate one another, they hopefully realize that their task is not only to reach a common understanding of what was said, but also of what was intended, as this may not be exactly the same. There are several important sub-goals toward reaching an understanding of one another’s theories. (1) The first is to try to look beyond what is said, toward what may be meant. That initial burden falls on the one trying to comprehend someone else’s theory. (2) Following questions and clarifications, a second important goal, falling on the one trying to express a theoretical view, is to state that view more clearly so that what is literally said will line up better with what is intended. (3) In the process of query and clarification, not only what is said but also what is intended may subtly change, given the discussion of the ideas in a way that is hopefully constructive rather than destructive.

A possible example comes from our own adversarial collaboration. Logie (2016), as a multicomponent theorist, has suggested that the central executive should be retired, given that it implies an homunculus (little person inside the head, making decisions for us) in control of cognition, and the argument that essentially it offers a label for complex aspects of cognition that we have yet to understand (Baddeley, 1996), but now are beginning to understand (and for a similar view, see Barrouillet & Camos, 2015; Eisenreich, Akaishi, & Hayden, 2017; Vandierendonck, 2016; Willshaw, 2006). Rather than simply rejecting that view, it is important for opposing theorists to query what was meant by it. Does it mean that there is no such thing as mental effort? One might assume so, but some of us heard Logie give a public lecture in which he used the terms “mental effort” and “work harder” to describe some working-memory phenomena in real life to non-specialists. From the multicomponent perspective, the implication is that autonomously-operating brain modules working co-operatively yield the illusion of self-control, a description with which, on some level, most psychologists can agree. The embedded-processes view discusses the central executive as a voluntary decider or deliberator but, as noted at the close of Cowan (1995, p. 274), the decision process still must emerge in a determinate way from the laws of physics, chemistry, and neurology, and the central executive concept represents these processes until more is known. The decisions participants make using their central executive processes can be said to be voluntary, in that they can change according to experimental instructions or pre-stated motivations. It is not clear whether we disagree on this issue and in the future, the central executive or whatever replaces it is one example of a concept that must be explained quite cautiously inasmuch as misunderstandings of views on this topic seem likely.

Even when theorists do not agree with one another at all, they may find rhetorical or conceptual use for one another’s theories. Consider the persistent usefulness of the “modal model” of memory, a term that appears to have been devised by Murdock (1967) to describe a tripartite memory system with a sensory store with information that decays, leading to a primary memory with a small amount of information that can be displaced, and leading from there to a long-term memory with a lifetime of information that suffers interference at the time when information is retrieved from it (cf. Atkinson & Shiffrin, 1968). We suspect that many of the theorists who reject such a model (e.g., Baddeley & Hitch, 1974; Baddeley, Hitch, & Allen, 2019; Logie, 1995) nevertheless present the general idea of the three-part system in an attempt to explain memory to introductory psychology students with minimal confusion. [In doing so, the sequential relationship between the three parts may be altered from Atkinson and Shiffrin: both Logie (1995) and Cowan (1988) would describe the flow of information as sensory input to long-term memory before activated long-term information could be entered into working memory (Logie) or into the focus of attention (Cowan)]. Similarly, in physics, relativity theorists and quantum theorists might usefully present Newton’s theory of gravity as an understandable approximation to the truth. As Newton built on theories from Descartes, so Einstein built on Newton’s theories, and Stephen Hawking built on Einstein’s ideas. If a non-subscriber to a theory finds it useful in communication, perhaps theorists can understand each others’ theories as being valid within a certain domain (cf. Mellers et al., 2001), and perhaps even take their own theories with a grain of salt, acknowledging how much of the theory has yet to be filled in, clarified, or modified.

Finding Possible Limits and Extensions of Adversarial Collaboration

Researchers have to have some assumptions in common to motivate an adversarial collaboration. For example, in our collaboration, all three theoretical camps have operated primarily with verbal and graphical statements of theory (Barrouillet & Camos, 2015; Logie, 1995, 2003, 2016; Cowan, 1988, 1995, 1999) and with the use of simple mathematics as tools of measurement (e.g., Cowan, 2001; Cowan, Blume, & Saults, 2013, appendices; Rhodes, Cowan, Hardman, & Logie, 2017; Rouder, Morey, Morey, & Cowan, 2011) or to express simple laws of behavior (e.g., Barrouillet, Portrat & Camos, 2011). This approach is at odds with researchers who believe that theories are valuable only to the extent that they are stated in complete mathematical form to allow quantitative predictions, even when this means specifying some parameters arbitrarily (e.g., Oberauer & Lewandowsky, 2011). Could an adversarial collaboration include both qualitative and quantitative modelers? Probably so but with the impediment that it is difficult to compare a theory for which evaluation is based on a qualitative pattern of differences between conditions with a theory for which it is based on quantitative model fit statistics. As another example, it is difficult to compare a theory of the mind with a theory of brain function, and so some differences between theories may reflect different levels of explanation rather than genuine theoretical conflict.

Nevertheless, we could imagine quite exciting collaborations between these types of theorists. A quantitative theorist can make statements that constrain more qualitative theorists, such as: I cannot find a reasonable way to obtain your pattern of results without some sort of central executive; I cannot get verbal rehearsal to work correctly to produce typical results (cf. Lewandowsky & Oberauer, 2015); Your theories sound different but produce the same results; or What plausible processes are needed to make a capacity-based theory of verbal working-memory account for probe recognition results? (cf. Cowan, Rouder, Blume, & Saults, 2012).

One could imagine an adversarial collaboration with, say, four theoretical camps involving two theoretical positions (e.g., multi-modular versus unified working memory storage) crossed with two methodological positions (e.g., primarily qualitative versus heavily invested in quantitative modeling). It would produce alliances between theoretical Positions 1 and 2 and different alliances between methodological Positions A and B, potentially to an effect that is quite innovative. In another example, the 2 × 2 grid of views could include theories of working memory crossed with behavioral versus neuroscientific methods.

Recommendations

A reviewer of a prior draft of this article asked for several specific recommendations, and we agree on the need for them. We briefly answer the reviewer’s questions here with reference to many related points already discussed above and entered into Table 1.

How should partners for collaboration be chosen?

We believe that the laboratories that collaborate must be willing to come in good faith, with the possibility of compromise, at least to the point of agreeing on a method to use together; but with no agreement needed on the anticipated outcome or the interpretation of every possible finding. It is most likely that one or even several experiments cannot resolve the differences to every side’s satisfaction, so the investigators should be willing to advance the field without becoming too frustrated that a larger reconciliation between views cannot be reached; the collaborators must be patient, and must try to be realistic. It is also helpful if one chooses collaborators who express their views precisely enough that their expectations are clear, although it seems fine for some points of view to make no prediction on some part of the experimental outcomes (unlike the method we have used).

Of course, not all scientific disagreements are right for collaboration; if the sides disagree too vociferously and emotionally then it may be impossible for collaborators to work together effectively. This level of emotionality might be an impediment, for example, in debates about whether restored memories of childhood abuse commonly occur, or about whether a nativist view of language is appropriate. These are great areas for adversarial collaboration, but only if the investigators find that they are able to work together effectively. There have been some breakthroughs of agreement, such as agreement between two investigators with different views regarding police lineups; they reached an agreement on what lineup conditions allow accurate, confident identifications of the correct suspects, though most likely not on all predictions and theoretical points (Wixted & Wells, 2017).

How many laboratories should collaborate?

Our experience tells us that in-person meetings are important and that each side will need considerable time to express their views and enter into discussions. We therefore recommend two to four laboratories, perhaps more than coincidentally similar to the suggested core limit in working memory capacity (Cowan, 2001).

How does the process of experimental design work?

The aim should be to test the most fundamental disagreement between views in a way that extracts maximally dissimilar predictions from the different views. The key is reaching an agreement about what design is acceptable to test these different views. It is possible that some potential collaborations will end before an agreed-upon design can be found. However, to prevent that unfortunate outcome, it greatly helps to have funding for experts such as postdoctoral fellows whose main focus is carrying out experiments for the collaboration. In-person and remote electronic meetings and regular communication are essential to ensure a joint commitment to complete the project successfully. Journal editors can facilitate this kind of collaboration by being willing to review the methods before data are collected, offering either in-principle acceptance (contingent on faithful execution of the stated design) or comments regarding how the design is perceived by reviewers.

Depending on the theories, it may not always be possible to come up with a perfect experimental design. For example, in comparing two theories, it would be perfect if some Effect A (like our dual-task costs) were predicted by Theory 1 but not Theory 2 (or better yet, if Theory 2 predicted an effect in the opposite direction), whereas some Effect B (which we have not found) were predicted by Theory 2 but not Theory 1. In the absence of this kind of dissociation of theories by two effects, Bayesian statistics can be used to argue for the null hypothesis predicted by a theory, as mentioned earlier.

What mistakes tend to occur, and how can they be avoided?

One grave mistake would be to forge ahead with research without preregistration of predictions, methods, and planned analyses, or without general data sharing between laboratories, in which case it would be likely that at least one group would tend to perceive shifting postdictions or lack of transparency. We did not make that mistake, but perhaps we should have openly discussed the ground rules for predictions. These often had to be made under considerable time pressure and the procedure was set up such that groups were forced to make explicit predictions to the point that it was sometimes not possible to document which were the key predictions of an approach and which were filled-in predictions that were not based on strong commitment of the theory. It would also be a mistake to underestimate the importance of including some collaborators who are not fully committed to any of the views, as they can tend to serve as arbiters when there is an impasse (a function served by the first author of Mellers et al., 2001). In our collaboration, postdoctoral fellows served this function, as they tended to be less committed to any one view than were senior investigators. Finally, it could be a mistake to adhere too severely to the preregistered analyses. These definitely should be carried out and considered carefully but, after that, if it is found that these analyses do not completely distinguish between theories, we feel it is perfectly acceptable to carry out additional, post-hoc analyses provided that the post-hoc nature is made clear.

What happens when the different laboratories disagree on the interpretation of results?

We have tended to report all of the alternative interpretations in our joint publications and have used these conflicting interpretations to help steer additional research needed to resolve the issues (as did Mellers et al., 2001). In the end, although the differences are not yet resolved by reaching a mutually palatable theory, we have provided important evidence to the field for others to judge what theoretical views work best and have established some grounds on which mutually acceptable data can be collected. As this article demonstrates, we have gained valuable experience in the process of working together.

We urge investigators to try out such adversarial collaborations and, as they do, to take careful note of what is working and what is not, and how the theoretical views are evolving. In place of continually perpetuating debates that are never resolved, adversarial collaborations, and reports on how they perhaps become less adversarial or less chaotic over time, could advance our means of accelerating scientific progress.