Ecological Validity as a Key Feature of External Validity in Research on Human Development

Abstract

This issue addresses the topic of ecological validity in research in human development across the life span. Although recent developments in study design, data collection methods, and data analysis techniques have greatly advanced researchers’ ability to collect large amounts of data on large groups of individuals in natural settings, it is important to approach these data with a reflected understanding of their ecological validity. Just because data were collected using everyday and familiar stimuli or ecological momentary assessment methods does not mean automatically that the ecological validity of these data is guaranteed and can go unquestioned.

Recent technological advances in data collection methods and researchers’ ability to process large quantities of data have raised the question if the behavioral sciences, including research on human development, have found a way to address the issue of ecological validity (EV) of study designs and research findings in a more satisfying way. For example, the advent of handheld devices, tablet computers, miniaturized global positioning systems (GPS), and most recently smartphones and sensorbased wearables give researchers the tools to collect real-time data in real-life contexts for extended periods of time (Mehl & Connor, 2012; Reis, 2012; Wrzus & Mehl, 2015). Thus, assessments that were traditionally conducted in the confines of a laboratory, or via surveys collected at periodic intervals, are increasingly replaced by real-time, “in vivo” assessments that are collectively referred to as methods of ecological momentary assessment (EMA; Shiffman, Stone, & Hufford, 2008) or ambulatory assessment methods (Wrzus & Mehl, 2015).

Given that many researchers, including those who study human development, show a great fascination with these new methods and increasingly incorporate them into their studies, we believe it is timely to ask if these methods have solved the key issues discussed with regard to the methodological criteria of EV. Or to state things in a more provocative way: Has the advent of recent data collection technologies made many of the quasi-experimental research designs obsolete and has solved all issues of external validity, including issues of EV?

Furthermore, we believe that it is timely to evaluate to which extent recent advances in stimulus generation, data collection methods, and study design may have improved the EV of experimental findings, and the EV of data obtained from large-scale surveys that strive for “contextual” embeddedness. We would also like to mention that the common treatment of EV in current developmental science, in general, and in the context of ecological momentary assessments, in particular, in our view lacks a certain degree of critical reflection. Often the term EV is used by default and without any in-depth considerations reflecting on the concepts meaning (see Freund & Issacowitz, 2013, for a similar argument). Therefore, it is our goal with this issue of Research in Human Development (RHD) to foster an understanding that no research design or research method will automatically guarantee high EV. Rather, we try to create an understanding that the challenges for achieving high EV are far from trivial and require a great deal of conceptual and empirical reflection. To achieve this objective, we invited authors from different disciplinary backgrounds to address the issue of EV in their particular area of inquiry. The result is that the topic of EV is covered across a broad range of developmental topics.

THE CONCEPT OF ECOLOGICAL VALIDITY

The term ecological validity was first introduced by Egon Brunswik (1943, 1955) in the area of the psychology of perception. According to Brunswik, EV concerns the question of whether the stimuli included in a psychological experiment are a good representation of the organism–environment relation in the naturally occurring ecology. Brunswik considered this representation as crucial because psychological processes represent adaptations to stimuli in the natural environment and can only be validly tested if the stimuli are representative of this environment. He referred to the implementation of this approach as “representative design.”

Although Brunswik (1943, 1955) aimed for a rigorous definition of EV amenable to empirical testing (i.e., the overlap between the range of stimuli found in the natural ecology with relevance for a certain research topic and the ones selected for an experiment should be quantifiable), other scholars approached the concept of EV in a more pragmatic way. In particular, Bronfenbrenner (1977) stated with regard to research in child development that “… much of contemporary developmental psychology is the science of the strange behavior of children in strange situations with strange adults for the briefest possible of time” (p. 513). With this statement, Bronfenbrenner started a debate on how human development should be studied across the life span and what role different contextual influences and changing environments may play in life-span development. To avoid an often-repeated misrepresentation of Bronfenbrenner’s position, it is important to point out that he did not advocate per se for the use of naturalistic observation as a superior method of knowledge generation. Rather, he pointed out that the use of well-conceived, structured experiments is an extremely valuable and essential approach to the study of behavioral development. Thus, Bronfenbrenner (1977) argued, from the beginning, against a dichotomy that equated naturalistic observation with “ecologically valid research” and experimental observation with “ecologically invalid research.” We believe it is extremely important to keep Bronfenbrenner’s conceptualization in mind as it has informed a great deal of research in human development, including recent approaches such as relational developmental systems models (Lerner, 2012).

In agreement with other authors (see Cook & Campbell, 1979; Whitley, 1996), we view EV as a facet of external validity. Campbell and Stanley (1963) originally defined external validity as the generalizability of a study’s findings to (1) other similar stimuli or variables; (2) other groups of individuals, including age groups; and/or (3) settings or contexts, such as physical or social environments (e.g., effects observed in the school context may or may not generalize to the family context), or time frames (e.g., effects observed in the short-term may not be observable over longer time spans). As this definition shows, EV can be a function of (1) the stimuli or variables included in a study, (2) the individuals/populations, or (3) the settings/contexts or time frames of investigation. In this sense the concept of “ecological validity” is closely linked to the concept of representative design as proposed by Brunswik (1955) because at the core of that concept lies the representative sampling of objects, subjects, settings/contexts and time frames for generalization purposes. Thus, when researchers address the topic of ecological validity, they always should ask “EV with regard to what?” and should keep in mind that EV is a multifaceted concept itself.

GOAL AND PLAN OF THIS ISSUE

Given this background, the overall objective of this issue of RHD is to stimulate further discussion on issues of external validity, in general, and EV in developmental research, in particular. Although we see a great deal of value in Brunswik’s (1943, 1955, 1956) call for a representative design approach as well as in Bronfenbrenner’s (1977) critique of decontextualized work in child psychology that produces “strange” behaviors in “strange” situations, we are also mindful of the many barriers that exist in actual research settings and in the overall culture of science production. These barriers often prevent researchers from fully implementing more ecologically valid study designs and leave them fixated on effect sizes and levels of statistical significance (Fiedler, 2011).

The sequence of articles starts with Kunzmann and Issacowitz’s (2017) treatment of EVand related laboratory research in three domains of emotion research with adults: affective information processing, emotional reactivity, and emotion perception. Specifically, the authors argue that the use of ecologically valid stimuli (i.e., the use of stimuli and tasks that are familiar and personally relevant to the individual) will contribute to a more nuanced understanding of age differences (and similarities) in emotional functioning. Dirk and Schmiedek (2017) focus in their empirical article on the general observation that the detrimental effect of noise on cognitive performance, particularly for younger children, has been demonstrated in experimental and field studies. In their ambulatory assessment study, which allowed for the capturing of school children’s daily world with high ecological validity, they indeed found that occasions with increased levels of disturbance were associated with decreased working memory performance, independent of other contextual conditions. Bielak, Hatt, and Diehl (2017) provide a conceptual account of the well-known “lab-life gap” in cognitive aging research as an important issue of EV in human development research. The authors discuss so far underdeveloped topics in this area with direct importance for EV. These topics include the relevance of everyday cognitive tests in an ever-changing world, new methods for assessing everyday cognition, and whether everyday cognition can be improved via structured practice. In their conceptual-empirical article, Wilkinson, Ferraro, and Kemp (2017) apply issues of EV to survey research. A number of approaches for contextualizing survey data are reviewed and the utility of contextual data is discussed. In fact, these authors argue that neglecting contextual factors may lead to misleading substantive conclusions, especially with regard to specific minority groups. Finally, Ram, Brinberg, Pincus, and Conroy (2017) offer a critical treatment of experience sampling methods and related data collection strategies, which usually claim high EV. As these authors recommend, more unobtrusive monitoring and person-specific analyses may provide more robust descriptions of individuals’ actual behavior.

Ultimately, we believe that the objective of EV can only be achieved in a cumulative way as the findings of studies are replicated and cross-validated across many situations (i.e., settings of stimuli), populations, and time frames. In our opinion, it is not helpful or correct to speak of ambulatory assessments or ecological momentary assessments as ecologically valid procedures per se, as is frequently done or implicitly assumed. In principle, we recommend avoiding language that suggests that research designs exist that automatically and inherently possess high EV. Furthermore, achieving maximum EV also involves that researchers have a clear understanding of the boundary conditions of their study designs, including the boundary conditions with regard to stimulus materials, participant populations, contexts, and time frames (Whitley, 1996). Such a reflected and critical understanding of EV, in our opinion, should help to make the bodies of evidence in our field more robust with regard to external validity.