Countering the Curse of Dimensionality: Exploring data-generating mechanisms through participant observation and mechanistic modeling

Causal assumptions [..] cannot be verified even in principle, unless one resorts to experimental control. […] Statisticians can no longer ignore the mental representation in which scientists store experiential knowledge, since it is this representation, and the language used to access it that determine the reliability of the judgments upon which the analysis so crucially depends.

– Judea Pearl¹

Public-health researchers face ethical and practical barriers that often preclude attaining a sufficient degree of experimental control to formally infer causality. Ethical concerns rule out many experiments on human health, and diverse socio-ecologic mechanisms shaping disease outcomes challenge experimental design (Figure part A). Researchers are instead relying on large observational datasets to attain causal understanding, but this requires balancing populations across all potentially relevant variables for an outcome of interest. As the number and interdependence of such variables increase, the size of the required dataset rapidly exceeds plausible levels (“Curse of Dimensionality”; fig. 1 Part B).²

Figure. Facilitating causal inference through systematic mechanistic exploration.

(A) Outcomes of interest to public health are shaped by complex networks of mechanisms. For example, population circulation can affect and be driven by pathogen transmission, and pathogen transmission can alter the structure and dynamics of social networks, which in turn can influence the capacity of communities to adopt health-enhancing behavioral norms.⁴⁴ (B) Disentangling these mechanisms to ascertain causes presents serious analytical challenges. Faced with the curse of dimensionality, for instance, researchers must rely on assumptions from the outside concerning data-generating mechanisms to select key variables and thereby reduce the dimension of the problem. (C) By placing researchers inside system dynamics, participant observation promotes a process of continuous, responsive counterfactual reasoning, without predetermined variables and rigid study designs. Instead, researchers accumulate perceptions and experiences of potentially relevant mechanisms from diverse perspectives. (D) Combined with mechanistic modeling, participant observation can facilitate causal inference by guiding the exploration of candidate data-generating mechanisms across different contexts of observation (‘transportability’).⁴⁵

Over the last thirty years, it has become evident that the curse of dimensionality extends across global challenges from climate change and threats to biodiversity, to increasing disparities and insularity among social groups. To improve causal understanding requires lowering the dimension of these complex processes. This in turn requires experiential knowledge across many disciplines and thus expanded definitions of relevant research foci and methods.³ Public health has partially addressed this through a renewed commitment to investigating health outcomes as parts of socio-ecologic systems, a return to the holistic roots of the field in the nineteenth and early twentieth centuries.^4,5

Delving into the health impacts of the systems involved in climate change⁶ or social disparities,⁷ public health researchers are trying to confront causal tangles not easily solved using conventional analytical approaches such as simple linear or logistic regression models. Identifying major risk factors for disease can lead to misleading results when studying large numbers of variables interacting at multiple social, spatial, and temporal scales.^8–11 In the case of epidemiology, in particular, the risk factor approach has been critically evaluated in discussions of “black box” epidemiology.¹² There have been subsequent calls for multi-level analysis,¹³ more sophisticated analysis of causal webs,¹⁴ and the need to move away from proximate (individual) and toward ultimate (socio-ecologic) causes of disease.¹⁵

Since the late 1990s, therefore, public-health methodologists have been evaluating how researchers move between population samples, joint distributions of variables, and the “data-generating mechanisms”¹ that underlie them. Arguably their most important insight is that, beyond the confines of strict experimental control, causal inference must proceed from prior assumptions or beliefs about data-generating mechanisms.^16,17 These causal assumptions, unlike associational assumptions, are fundamentally untestable; their validity hinges on the researcher’s ability to identify and accurately map the true data-generating mechanism underlying a distribution of interest.¹

Much of the early causal inference literature, however, created its evidence only within an experimental paradigm. Some recent discussions of causal inference have called for broader types of evidence to describe data-generating mechanisms: Schwartz et al.¹⁸ (see also Kaufman¹⁹), for example, criticized the excessive attention on intervention designs as the only legitimate form of causal exploration, and Krieger and Davey-Smith called for “different strands of evidence produced by myriad methods.”²⁰

We propose that field research iterating cycles of observation and mechanistic modeling represents a useful approach to explore and evaluate candidate data-generating mechanisms. Participant observation is especially appropriate when modeling human behavioral influences on disease. Other kinds of observation related to physical or natural processes (e.g., rainfall patterns, soil composition, and animal behavior) along with mechanistic modeling can reveal the relevant causal dynamics of non-human or non-behavioral influences on disease. This is precisely the experiential knowledge that Pearl promotes as essential for causal inference. For the purposes of this commentary, we focus on participant observation of social processes to illustrate our ideas about how to link observed associations to theories about causality.

Participant observation methods are especially important for epidemiologists collecting data in communities (“shoe-leather” studies), but we think that epidemiologists doing secondary analysis of existing data would also benefit from referring to these kinds of data. As we next point out, fundamental elements of this approach were instrumental to the early development of public health, especially in studies of infectious disease.

The power of participant observation

By participant observation, I mean a technique […] of getting data, it seems to me, by subjecting yourself, your own body and your own personality, and your own social situation, to the set of contingencies that play upon a set of individuals […]. So that you are close to them while they are responding to what life does to them. […] To me, that’s the core of observation.

– Erving Goffman²¹

Modern public health relies heavily on observational evidence gathered in the field. Researchers collecting community level data spend substantial amounts of time in the communities they study in order to identify and interact with key informants, conduct focus groups, administer surveys to carefully selected samples of individuals, and measure predefined variables. They may also collect relevant information in an informal way as they are repeatedly exposed to and share in the details of daily community life. Although unconstrained experiential observation of this sort is seldom included as an explicit aim of modern public-health fieldwork, when carried out intensively it can enrich the static snapshots of system states produced through standard observational techniques. It can stimulate and guide intuition and creativity—whose role in epidemiology was recently highlighted in this journal²²—and ultimately facilitate causal inference.

Intensive engagement with a system over time stimulates researchers to consider target phenomena from diverse perspectives, promoting continuous, responsive counterfactual reasoning. When accumulating perceptions and experiences of daily life, researchers can abstract observed patterns into categories—social roles, types of objects and places, etc.—and the relations that link them in recurring types of interactions. Researchers can then systematically evaluate the coherence and operation of these categories and relations across data sets, time, and situations.²³

Anthropologists and sociologists use the term “participant observation” to refer to fieldwork designed to elicit this sort of immersive engagement with a system (Figure part C), and have written extensively about its methodologic underpinnings.²⁴ Well before the formal advent of participant observation within anthropology and sociology, however, close precursors were pioneered in nineteenth-century public health.^25,26 In the case of measles, for instance, over a century before the isolation of the virus, Peter Panum used detailed studies of diets, habits, and living conditions following an outbreak in the Faroe Islands to show its respiratory transmission, incubation period, and the lifelong immunity it confers. During 8 months, he visited 52 villages, talking with those who had contracted the disease, verifying the timeline of its progression, and showing the relationship between people’s gathering together in group fish hunts and their subsequent infection.

Similarly, important aspects of participant observation can be attributed to another foundational contribution of modern epidemiology, John Snow’s work on the transmission of cholera.²⁷ Snow isolated the causal features underlying cholera outbreaks in large part by accumulating perceptions and experiences of socio-ecologic dynamics within different contexts of transmission. His sources included informal observations in the coalmines of Newcastle, systematic conversations with residents during the Soho outbreak of 1854, and the detailed local knowledge of a clergyman who was intimately involved with the community. Unlike most of his contemporaries, Snow was able to focus “on the dynamics of transmission, on movement rather than stasis, on the verbs of actions, rather than the nouns of characteristics and things.”²⁸

Turning to contemporary public health, aspects of participant observation have been productively used to study illicit drug use and related interventions,^29,30 and to evaluate population-level records of infant mortality in Brazil in light of the perceptions and experiences of “popular death reporters” such as coffin makers, midwives, and priests.³¹ These studies illustrate the utility of immersive fieldwork for probing unexpected results from quantitative analyses and improving the validity of estimates. Although participant observation can thus be effective on its own, we believe that fully realizing its promise for systematic mechanistic exploration will require integrating it into formal mechanistic modeling techniques. We next argue that the strengths of mechanistic modeling complement those of participant observation.

Toward systematic mechanistic exploration

Where other forms of observation, such as surveys and interviews, capture static snapshots of predetermined facets of a system, participant observation injects the researcher directly into the thick of the dynamics. Participant observers often struggle to connect rigorously the details of daily lives they accumulate during fieldwork with their evolving insights regarding the macro-level socio-ecological forces that help shape them. Sociologist Mitchell Duneier argues that researchers should “try to grasp the connections between individual lives and macroforces at every turn, while acknowledging [their] uncertainty when [they] cannot be sure how those forces come to bear on individual lives”.³² Agent- and equation-based mechanistic modeling can help participant observers probe mechanistic connections, communicate assumptions and uncertainties unambiguously, and ensure the compatibility of the resulting insights with appropriate statistical techniques (Figure part D).

Mechanistic models can easily be extended from simple toy models—little more than dynamic conceptual schemas (e.g., compartmental models)—to virtual counterfactual experiments incorporating large sets of carefully estimated parameters for specific cases. This spectrum of mechanistic abstraction is ideally suited to complement participant observation iteratively. Particularly in the early stages of participatory fieldwork, models can help clarify and organize mechanistic insights as they emerge from the observer’s accumulating perceptions and experiences. Even divorced from quantitative data, models at this stage can shed light on the plausibility of potential data-generating mechanisms.

Exploratory mechanistic modeling also has a long history within public health. The gradual extension of models based on simple mathematical abstractions in the late nineteenth and early twentieth centuries, for instance, helped fuel the development of infectious-disease epidemiology. Examples include the fundamental debate concerning whether regularities observed across epidemic curves for various diseases should be attributed to mechanisms of variable pathogen infectivity or host population dynamics. This debate was resolved through the gradual development and analysis of SIR (Susceptible Infected and Recovered) models,³³ which helped establish the concept of herd immunity and demonstrate the value of mosquito control in combating malaria.^34,35

The early 20th-century epidemiologist Ronald Ross was especially clear about the importance of mechanistic modeling for the development of public health. At a time when epidemiologic research was predominantly rooted in “a posteriori” statistical analyses, Ross advocated an “a priori pathometry,” conceptualizing epidemiologic problems in terms of abstract “happenings” and identifying mechanisms that could account for observed disease patterns across multiple contexts.³⁵ He presented a strong argument for the value of simple mechanistic models and their iterative extension: “These studies require to be developed much further; but they will already be useful if they help to suggest a more precise and quantitative consideration of the numerous factors concerned in epidemics. At present medical ideas regarding these factors are generally so nebulous that almost any statements about them pass muster […]”.³⁶

Many of the model notations and structures developed from the foundational debates mentioned above have anchored a century of sophisticated forays into epidemiologic dynamics.³⁷ For example, the progressive extension of SIR and related models was instrumental in establishing the role of superspreaders as potential drivers of outbreaks. Analyses of specific chains of transmission for diseases such as gonorrhea and severe acute respiratory syndrome (SARS) suggested that interactions within small groups of high-risk individuals might sustain transmission in larger populations.^38,39 These observations promoted models allowing heterogeneous network dynamics, which have since confirmed the importance of considering superspreaders in outbreak control strategies, and the types and quality of data necessary to document superspreading events.^38,40

Simple models can lend structure to the early stages of mechanistic exploration without determining or constraining the perceptions and experiences of the observer. In turn, the observer’s evolving mechanistic insights can guide the extension of models and targeted data collection to inform them. Ultimately, we envision this iterative approach as yielding a constrained set of well-specified models representing candidate data-generating mechanisms, primed to guide inference in combination with appropriate statistical techniques.^{2, 41–43}

Obstacles in academic and public health practice create disincentives to bringing observational analysis from social sciences together with modeling analysis from the natural and physical sciences. These obstacles are curricular, financial, temporal, and cultural. They can be addressed through training epidemiologists on qualitative methods and more grant opportunities requiring multiple disciplines in research teams. As epidemiologists acquire these skills and experience, the impediments associated with disciplinary conventions about study duration and legitimate methods will grow smaller. We suggest that iteratively using participant observation, basic science, and mechanistic modeling will provide a smaller and more efficient causal representation of a system under study, eliminating unnecessary variables and ensuring we include the important variables. Thus, theory can reduce the dimension of the problem because fewer variables are considered relevant, helping to operationalize Judea Pearl’s call for statisticians to consider “the mental representation in which scientists store experiential knowledge.”¹

Biography

About the Authors:

Alan Hubbard is Professor of Biostatistics at University of California, Berkeley. Dr. Hubbard’s research focuses on the estimation of causal effects using machine learning with applications in epidemiology and biomedicine.

James A Trostle is Professor of Anthropology at Trinity College and Visiting Professor of Public Health at the University of Chile. He combines anthropology and epidemiology to study the transmission of infectious diseases within families and across regional landscapes. During three decades of global health work he has served on task forces for WHO Programs on Tropical Disease Research, Diarrheal Disease Research, and Human Reproduction.

Ivan Camgeni is a Postdoctoral Scholar in the Department of Epidemiology at the University of Michigan. Dr. Camgeni’s research focuses on agent based modeling of population processes with a focus on the dynamics of social structure and interaction.

Joseph N.S. Eisenberg is Professor and Chair of the Department of Epidemiology at the University of Michigan. Dr. Eisenberg has a particular interest in environmental determinants of infectious diseases, largely focusing on water- and vectorborne diseases, integrating field epidemiology with dynamic transmission model analysis.