The Laboratory of Brain and Cognition: A Brief History

The Laboratory of Brain and Cognition (LBC) was founded over 30 years ago by Leslie Ungerleider, Jim Haxby, and myself. That makes it the second longest-lasting laboratory in the Intramural Research Program of the National Institute of Mental Health, eclipsed only by the non-human primate laboratory, the Laboratory of Neuropsychology, that was founded in 1977.

Leslie, Jim, and I had known each other since the early 1980s when we were postdoctoral fellows (known at that time as “Staff Fellows”). Jim was in the NIA (National Institute on Aging), I was in the National Institute of Neurological and Communicative Disorders and Stroke, and Leslie was in NIMH (National Institute of Mental Health). Jim and I were both trained as neuropsychologists and were working on the neuropsychological and functional brain imaging characteristics of patients with Alzheimer's disease using fluorodeoxyglucose PET (Haxby & Rapoport, 1986; Martin et al., 1986), while Leslie was engaged in her seminal work on the visual processing streams in the macaque with Mort Mishkin (Ungerleider & Mishkin, 1982). So, although we were in separate institutes, we all knew each other and were familiar with each other's work.

By the early 1990s Jim and his colleagues in the NIA, Cheryl Grady and Barry Horowitz, had teamed up with Leslie and Mort to use a faster and much more spatially specific PET scanning method (oxygen¹⁵) to evaluate, for the first time, the ventral object vision and dorsal spatial vision pathways in the human brain (e.g., Haxby et al., 1991). It would be difficult to overstate the importance of this work for the history of functional brain imaging at the NIH (National Institutes of Health) and for the neuroimaging community at large. The elegance of the experimental design and the clarity of the findings convincingly showed that one could use this method to localize neural systems and to evaluate the commonalities, and differences, between the human and monkey brain. In addition, having both Leslie and Mort as co-authors provided a stamp of approval that the broader neuroscience community could not ignore. This work also put the NIMH on the map as a potential major player in the burgeoning field of functional neuroimaging of the human brain.

Indeed, it was the success of this work that led to a mandate by Susan Swedo, at that time the Scientific Director of the NIMH Intramural Research Program, to begin an international search for someone to head up functional neuroimaging at NIMH. Mort Mishkin was recruited to chair that search. Anyone who knew Mort knew he avoided taking on this kind of administration task at all costs. Mort agreed, but with the stipulation that if we were not able to recruit a suitable candidate, Jim Haxby would be brought over from NIA to head up our functional neuroimaging efforts (https://history.nih.gov/display/history/Ungerleider%2C+Leslie+2002).

By the time that recruitment effort fell through, a new plan had been hatched. I had returned to the NIH after several years working on the neuropsychology of HIV/AIDS at Walter Reed Medical Center in Washington, DC (Grant & Martin, 1994), and was talking to Jim and Leslie about collaborating on brain imaging studies. At the same time, Mort stepped down as the Chief of the Laboratory of Neuropsychology to allow Bob Desimone to become Chief. This created a problem because Bob and Leslie were married at that time and NIH nepotism rules meant that Leslie would have to leave the laboratory. In addition, the NIMH Laboratory of Psychology and Psychopathology (the LPP) was in dire need of new blood and new directions. So, to address all these issues, the three of us were moved out of our respective laboratories into the LPP.

This occurred in the early 1990s, and within a short period, we essentially took over the LPP, changed the name to the Laboratory of Brain and Cognition, and established three independent sections. The Section on Functional Brain Imaging with Jim as Chief, the Section on Neural Circuitry with Leslie as Chief, and my Section on Cognitive Neuropsychology (in the world of the NIH intramural programs, “Sections” are equivalent to laboratories in the extramural world). Leslie also took on the role of Chief of the entire enterprise.

The search for a name for the new laboratory took a bit of work and many discussions among us three. As Leslie described it, “Well, we had to think of a name. We mulled over this for months. This was the topic of lunchtime talk. It had to be catchy. We really wanted the emphasis to be on cognition or cognitive studies. A lot of the other names were taken already. Mort came up with a very good name—Cognitive Neuroscience—for his Section that covered everything. That should have been the name of our lab! But we just thought, well, Brain and Cognition. It was short and sweet and it could cover human work, monkey work, anatomy, imaging, lesion studies. It was all-inclusive. (https://history.nih.gov/display/history/Ungerleider%2C+Leslie+2002)

So, it seemed that we were well on our way to building a world-class brain imaging program. However, we were soon met with a difficult hurdle. We had written and received approval of a protocol to do a series of studies on neural systems for perception, memory, and learning. However, we had no control over the PET scanning facility and we had limited access to the scanners, being provided with only two or at most three scanning slots per month to collect data for our studies. At this time, there was also a new Scientific Director for the NIMH who sought to obtain a critical evaluation of all of the neuroimaging work currently being done in the program. To accomplish this goal, he brought in an outside panel of experts to review the brain imaging work being done by all the Labs and Branches in the institute. So, over several days, each group presented to the review board. I do not remember all the participants, but I do remember that the board included Marc Raichle and Steve Petersen from Washington University in St. Louis, and Albert Gjedde from the Montreal Neurological Institute. Jim took care of our presentation—including handing out a flow chart describing in detail our data analysis stream (I am sure none of the other groups did that)—while Leslie and I sat behind him to handle any questions that might be directed at us. I do not remember ever seeing the final report, but it soon became known that it was highly critical of all the work being done in the institute, except for our group. In fact, the panel made a strong case that we should be given all the resources we required. So, in essence, they saved the LBC, a point that Steve Petersen has reminded me of many times since.

From that point on, things moved very fast. Between 1995 and 2001, the laboratory produced a body of work covering nearly every important topic in cognitive neuroscience, bracketed on one end by Leslie's investigations of motor learning (Karni et al., 1995) and our work on the representation of object concept properties (Martin, Haxby, Lalonde, Wiggs, & Ungerleider, 1995) and on the other end by Jim's work using multivariate analyses to reveal the functional organization of the ventral processing stream (Haxby et al., 2001). All this work was accomplished along with a remarkable group of postdocs, including, in alphabetical order, Michael Beauchamp, Linda Chao, Vince Clark, Susan Courtney, Julian Doyon, Avi Karni, Sabine Kastner, Luiz Pessoa, and Cheri Wiggs, among others.

As Jim notes in his companion piece on the history of the LBC, it was an incredibly productive and exciting period of collaborative science for the three of us. He describes it “as a golden time and a high point in my career as a cognitive neuroscientist” (Haxby, 2024). That goes double for me.

It is important to note that, although we started with PET, the entire laboratory transitioned to fMRI very quickly. As a result, it soon became apparent that we needed an MRI physicist to help run the program. The position required someone to head up their own scientific research program within the LBC and to create a fMRI Imaging Core to support the entire NIMH intramural program. To fill these roles, we were extremely fortunate to recruit Peter Bandettini, an early pioneer and innovator of fMRI. Peter joined the laboratory in 1999 and established the Section on Functional Neuroimaging Methods while also creating and serving as the Director of the Functional Magnetic Resonance Imaging Core Facility.

Jim left the laboratory in 2002 to join the faculty at Princeton University. Several years later, we were able to recruit Chris Baker to the laboratory who established the Section on Learning and Plasticity and, most recently, Eli Merriam who joined us in 2016 and established the Computational Neuroimaging and Perception Team.

As of this writing, the LBC remains a vibrant community with over 50 members. The continuation of the LBC stands as a major part of Leslie Ungerleider's legacy. From the time we established the LBC until her untimely death in 2020, Leslie was the laboratory's guiding force. Although I have highlighted the work of the LBC on the functional organization of the human brain, it is noteworthy that all during this period, Leslie's Section also included her highly acclaimed non-human primate program (e.g., Zhang, Japee, Stacy, Flessert, & Ungerleider, 2020; De Weerd, Peralta, Desimone, & Ungerleider, 1999).

For me personally, she was a teacher, collaborator, and a dear friend. She taught me about science, how to do science, and how to communicate science. Her impact on the field is celebrated in this collection of papers as well as in articles published in Neuron (Kastner, 2021), Nature Neuroscience (Behrmann, 2021), the Proceedings of the National Academy of Sciences (Van Essen, Kastner, & Bandettini, 2021), and Trends in Neuroscience (Buffalo et al., 2021). In September of 2022, we held a symposium in her honor on the NIH campus. The tributes by major members of the neuroscience community and from many of her past students, fellows, and collaborators can be viewed online (Day 1, https://videocast.nih.gov/watch=46046; Day 2, https://videocast.nih.gov/watch=46048).

Acknowledgments

This work was supported by the NIMH Intramural Research Program (ZIA MH002588). The views expressed in this article do not necessarily represent the views of the NIH, the Department of Health and Human Services, or the United States Government.

Corresponding author: Alex Martin, Laboratory of Brain and Cognition, NIMH/NIH, Bethesda, MD 20892, or via e-mail: alexmartin@mail.nih.gov.

Funding Information

NIMH Intramural Research Program, grant number: ZIA MH002588.

Diversity in Citation Practices

A retrospective analysis of the citations in every article published in this journal from 2010 to 2020 has revealed a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .408, W(oman)/M = .335, M/W = .108, and W/W = .149, the comparable proportions for the articles that these authorship teams cited were M/M = .579, W/M = .243, M/W = .102, and W/W = .076 (Fulvio et al., JoCN, 33:1, pp. 3–7). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance.