Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jun 5.
Published in final edited form as: Int J Conf Violence. 2010 Jun 1;4(2):288–297.

Us versus Them in Context: Meta-Analysis as a Tool for Geotemporal Trends in Intergroup Relations

Judy Y Tan 1, Tania B Huedo-Medina 2, Carter A Lennon 3, Angela C White 4, Blair T Johnson 5
PMCID: PMC4045643  NIHMSID: NIHMS585388  PMID: 24910718

Abstract

The increasing availability of studies from many nations offers important potential insights into group-based psychology and behavior, conflict, and violence. Nonetheless, to date, few cross-national or cultural comparisons of study findings have been made, representing a gap in our understanding of the historical causes and courses of intergroup conflict in current comparative approaches. Meta-analytic methods offer researchers the ability to combine data from studies with groups as well as across time. Our review of statistical methods available for comparative analyses in intergroup research found strengths and limitations for understanding group differences, conflict, and violence, and meta-analytic methods address these limitations by exploring potential structural-level moderators and by identifying how temporal and geographical variations may relate directly to group-based variables. Such methods can contribute to our understanding of broad structural effects on group-based variables by elucidating the mechanisms underlying them.

Decades of intergroup research have amassed an extensive knowledge base from which prominent theories in intergroup relations and processes emerged. Numerous studies have tested long-standing perspectives in intergroup relations, such as the scapegoat hypothesis (Hovland and Sears 1940) and the authoritarian personality (Adorno et al. 1950). These studies were conducted in particular regions and at particular points of time. However, there is overwhelming data suggesting that attitudes, values, and behaviors are temporally and geographically clustered (e.g., Krug and Kulhavy 1973; Park and Peterson 2010; Plaut, Markus, and Lachman 2002; Rentfrow 2010). Yet little is known about how these temporal and geographical variations relate directly to group-based discrimination and conflict. Comparative analyses using data from various sources, time periods, and geographical regions have the power to elucidate mechanisms underlying group-based conflict and violence. Meta-analysis is a powerful comparative method that meets these goals yet is at present under-utilized.

The purpose of the current paper is to discuss major methodological issues involved in comparative analysis and to offer meta-analysis as a viable and practical solution in the study of intergroup relations. We begin by discussing the various methodological solutions and statistical tools for multi-level and longitudinal data, before presenting practical applications of meta-analytic methods to common methodological issues. We focus on issues of particular interest to intergroup comparative research: (a) whether group-based differences change over time, and (b) geographical area studied, (c) whether structural-level factors impact these patterns, and (d) how meta-analytic methods can be used to address these factors. Finally, we discuss the implications of such methods for structural-level theory and interventions. In order to understand how meta-analytic methods can enhance intergroup comparative analyses, it is first necessary to characterize the most sophisticated methods that are currently brought to bear on them.

1. Primary-Level Structural Comparative Analyses

Large-scale data on intergroup behavior and conflict are often multi-level or nested (e.g., groups within regions, regions within nation-states), and several advanced methods are uniquely suited to examine such data structures. For example, when intergroup differences in prejudicial attitudes and discriminatory behavior are found, researchers may have conceptual interests in discovering whether structural-level factors explain such differences. Various analytic strategies are available using either a causal or correlational approach depending on how the independent variable is operationalized, the data structure, and research questions. For example, hierarchical linear modeling (HLM) or multi-level modeling (MLM) are appropriate methods for examining changes in, say, xenophobia in relation to the emergence of conservative political parties within different nations and across separate times, when it is longitudinal (Rydgren 2003). Various statistical software programs, including HLM, Stata, SAS, and MPlus are commonly used for multi-level data analyses; the public-domain software R is increasingly used. Temporal effects add another level of complexity to structural-level analyses. In cases where there is more than one time point measured, repeated measures analysis can be conducted, considering time as another level.

Longitudinal structural equation modeling (SEM) has been extended to model intra- and intergroup variability over time and also allows estimation of causal relations among key variables and to test model fit. This strategy derives parsimonious theoretical models of causal relationships, a method useful for theory-building with temporal data. Two estimation models are available for longitudinal structural equation modeling: latent variable modeling of changes over time (McArdle 2009) and multi-group mixed-effects analyses (Ram and Grimm 2009). Such strategies could potentially be employed to examine how temporal changes (i.e., slope) in subordinate group members' level of prejudice predict changes in dominant group members' prejudicial attitudes. Finally, one of the advantages of using longitudinal structural equation modeling is its ability to deal with unbalanced or incomplete data, a common problem in longitudinal data (Judd, Kenny, and McClelland 2001).

These advanced statistical techniques allow us to fit complex causal or correlational models to available data and provide powerful ways for addressing problems arising from large quantities of longitudinal data, which are sometimes available from archives as secondary data. The main limitation of these methods that is directly relevant for cross-group comparisons is their reliance on longitudinal study designs. Such techniques also yield findings that are limited to specific participants at particular points in time and place. Because cultures are known to change along with intergroup relations, research would benefit from data gathered across a greater span of time and place. Yet, longitudinal designs are costly and suffer threats to validity, such as those due to history (Campbell and Stanley 1963). Given these limitations, alternative models of comparative analysis should be considered, such as meta-analysis.

The wealth of available studies on many group-comparison topics may seem like a good thing. Yet, beyond a certain point, very large numbers of studies can create an “evidence monster,” too large to tame with intuitive strategies (Johnson and Boynton 2008). Meta-analyses have been conducted to synthesize a wide array of social psychological topics (Richard, Bond, and Stokes-Zoota 2003), yet, to date, relatively few have been performed to compare groups. The lack of meta-analyses in this area means that the resources that have been deployed to compare groups have been underutilized and highlights a potential knowledge gap.

2. Meta-Analytic Methods

Meta-analysis organizes and integrates new findings into the currently existing information, identifies consistencies and inconsistencies within the data, determines if findings are generalizable, eliminates redundancies, and improves the “reliability and accuracy of conclusions” (Mulrow 1994, 597). By integrating findings from primary-level studies, meta-analytic methods allow us to compare results across decades, cohorts, and locales. They show whether intergroup attitudes and discrimination, behavior, and conflict operate the same way at different points in history, for example. Results from primary-level studies may also relate to structural-level factors as measured by social inequality indices. With true experiments, the effect sizes in a meta-analysis gauge a causal difference – the difference between experimental conditions – across multiple studies (see Bettencourt and Miller 1996; West and Thoemmes 2010). Yet, meta-analysis does not have to rely on experimental designs (Shadish 2010). Instead of creating experimental designs, meta-analytic techniques rely on past studies that have instantiated such designs to create experimental and control groups. Moreover, in some cases, temporal meta-analyses allow researchers to determine the temporal direction of causation, thus providing a solution to the correlation-or-causation dilemmas that often plague comparative research.

Meta-analysis for basic scientific questions

Meta-analysis is a powerful method for combining the aforementioned statistical techniques for analyzing data across time and at various levels (Johnson and Eagly 2000; Johnson and Boynton 2008) so that direct comparisons of study effects across different studies and populations can be made (see Cohen et al. 1999). Moreover, like other analytical techniques, meta-analytic methods can answer questions about the data across multiple studies (Johnson and Eagly 2000): (1) What are the statistically significant relationships among the data? (2) What is the level of variability in the data? And (3) what are the potential moderators that explain the variability? Meta-analysis answers these questions by comparing study results on a common metric adjusted for study sample size and other biases.

Several general steps should be followed for conducting a meta-analysis (Johnson and Boynton 2008; Johnson and Eagly, 2000; see Cooper, Hedges and Valentine 2009 and Lipsey and Wilson 2001 for detailed techniques and considerations). First, the researcher should articulate a research question and well-defined hypotheses of the relationships among variables of interest. These questions and hypotheses aid the process of searching for relevant articles. Once primary studies have been retrieved and coded, effect sizes should be calculated using the appropriate statistical technique(s). At this point, the researcher fits models to the effect sizes. In meta-analysis, the average effect size is a model that gauges a comparison across a set of studies. Goodness-of-fit statistics allow us to determine whether the mean is a good depiction of the underlying effect sizes. If there is more variability than one would expect by sampling error alone, then the mean effect size is not a good description of the studies' effects and more complex models are necessary.

Meta-analysis formulates statistical models in which it is possible to explain such heterogeneity as a function of substantive and methodological characteristics of the primary studies, otherwise known as moderators (Hunter and Schmidt 2004; Johnson and Boynton 2008; Lipsey 1994). The general linear model for predicting effect sizes from moderator variables is the usual strategy for analyzing their possible association. Hedges and Olkin (1985) proposed an approach based on weighted least squares multiple regression models, a practice that has become known as “meta-regression.” Primary study characteristics such as experimental design, recruitment method, age of the sample, and intervention and control group characteristics can moderate or mediate the variability sample effect sizes. Structural variables such as social inequality indices can moderate study effects, and may be included as moderators of the final effect size. Researchers should take care to correctly report meta-analytic results according to formal guidelines (e.g., QUOROM Statement, Quality Of Reporting of Meta-analyses, Moher et al. 1994); the revision of the guidelines, renamed PRISMA, or Preferred Reporting Items for Systematic reviews and Meta-Analyses, Moher et al. 2009).

Let us consider a concrete example of intergroup contact and its effects on intergroup prejudice. Pettigrew and Tropp (2006) conducted a meta-analysis to test a basic scientific question pertinent to intergroup research: Does intergroup contact reduce intergroup prejudice (Allport 1954)? Using 515 independent studies conducted across 38 countries over the past three decades, the meta-analysis tested the association between intergroup contact and prejudice, alternative explanations for effects of intergroup contact on prejudice, and effect-moderators (e.g., optimal context specified by Allport's conditions for positive contact). As predicted, intergroup contact generally resulted in prejudice reduction across various types of samples and contact settings, and effects were not attributable to alternative explanations. Contact-prejudice effects were not significantly moderated by any single contextual condition alone; rather, effects were moderated by a global indicator of optimal contact, suggesting the importance of considering Allport's optimal contact conditions altogether rather than independently. Pettigrew and Tropp's analyses (2006) provided a seminal test of an influential hypothesis in intergroup research by using accumulated data from across time and geography to advance scientific knowledge in this area.

In order to estimate the most accurate mean effect sizes and examine effects of the other moderating variables, the effect sizes derived from the primary studies must be adjusted or weighted accordingly (Lipsey and Wilson 2001). A pooled effect size across studies or a regression model needs to be weighted by the appropriate variance. Two basic models based on fixed- or random-effects assumptions may be employed to determine the weights. Fixed-effects models assume that differences between studies are due only to sampling error. In general, studies with larger sample sizes are weighted more than those with smaller samples. Fixed-effects models should be employed when researchers expect that no more than sampling error will remain after the model is applied, whether overall or in combination with moderators (Hedges and Vevea 1998; Overton 1998); strictly speaking, the results may be generalized only to conditions very similar to those observed in the underlying studies. Random-effects models, on the other hand, incorporate a source of variability in addition to sampling error, derived from the distribution of the observed phenomenon. In other words, the main assumption under random-effects model is that every individual effect size is estimating a parametric effect size with a conditional variance produced by random sampling. Findings from such models may generalize to conditions that differ from the underlying studies. When studies exhibit no more than sampling error, random effects models reduce to fixed effects models because the population variance is zero. Fixed-effects models tend to be relatively more likely to produce statistically significant results, whereas models that incorporate random effects tend to be relatively conservative, especially when studies lack homogeneity (Overton 1998).

In conducting secondary and archival data analyses, issues that arise and decisions that need to be made can affect how the data is treated (e.g., using random- versus fixed-effects model). Because meta-analysis uses secondary and/or archival data, the statistical assumptions applied to each type of data require careful consideration of the research question(s). Secondary data analysis uses data that other researchers have collected in multiple studies, while archival data analysis is based on data continuously collected over time to identify trends in a single source. Researchers must weigh the different assumptions associated with each analysis method and decide the appropriate approach for the research questions at hand (see Hedges and Vevea 1998 for discussion). The existence of archival data in meta-analysis permits researchers to incorporate important indicators into meta-analytical data.

3. Applications of Meta-Analytic Methods in Intergroup Comparative Analyses

Meta-analytic methods may be used to model explanatory mechanisms underlying changes in intergroup conflict, prejudice, and discrimination across time, space, and cultures. In the United States, population-based databases such as American National Election Studies, the General Social Survey, and the Federal Bureau of Investigation's Uniform Crime Reports and Hate Crime Statistics, as well as various survey polls (e.g., Gallup) can be used to examine conflict between groups geographically and historically. An added advantage of meta-analysis is that it allows structural-level moderators such as social inequality indices to be included in the model. Moderation by structural-level factors is often invaluably informative in accounting for group differences across time and space. For moderation analyses, researchers may consider structural features derived from the United Nations Development Program's Human Development Reports, the World Bank's World Development Indicators, the Schwartz Values Survey, and the Cingranelli-Richards (CIRI) Human Rights Dataset (http://ciri.binghamton.edu/index.asp). The Schwartz Values Survey dataset provides an overview of basic intercultural values from over 60,000 individuals in 64 nations across the world in samples taken as early as 1988, with further samples routinely added to the database. The CIRI database contains yearly measures of fifteen internationally recognized human rights from 195 nations, commencing with 1981 and updated annually. Coupled with comparative analyses of regional effects, meta-analytic methods offer important insights into how prejudicial attitudes are created and sustained.

3.1. Temporal Trends and Cohort Effects

In temporal analysis of prejudicial attitudes, researchers should test whether significant trends in the data over time reflect true change in prejudice or cohort differences. Differences between birth cohorts are driven by historical events, and by differences in cultural values and worldviews, formal education, and peer-group socialization (Ryder 1965; Stewart and Healy 1989; Twenge 2008). Reductions in prejudice with increasing age are related to changes in individuals, while reductions in prejudice with increasing time (or cohorts) are related to societal or cultural changes.

Both longitudinal and cross-sectional studies have been used to examine temporal trends (Woodruff and Birren 1972). Longitudinal studies identify changes due to maturation, while cross-sectional studies identify changes due to both maturation and generational differences (Costa and McCrae 1982; Schaie 1965). Thus, cross-sectional studies often confound age and cohort effects (Costa and McCrae 1982; Schaie 1965; Twenge and Campbell 2001; Woodruff and Birren 1972), making it difficult to determine the specific effects of age and cohort. Studies that find age differences in group-based prejudice and discrimination cannot be generalized if the studies were conducted at a particular time and did not examine potential cohort differences (Twenge 2001).

Tracking temporal trends is a key component for understanding intergroup conflict. In a meta-analysis that examined the association between intergroup contact and conflict, Hall, Matz, and Wood (2010) found that the relationship between religiosity and racism decreased over time between 1964 and 2008. In this case, tracking trends over time helped identify a variable that contributed to racism. Furthermore, numerous studies (e.g., Avery et al. 2007; Hicks and Lee 2006; Loftus 2001) have shown that attitudes toward gays and lesbians have become more positive since the 1970s. Tracking temporal trends is a key component for understanding intergroup conflict as it can serve as a clue that relations are improving or degrading and can aid in identifying factors that drive these trends.

To examine cohort differences more rigorously, researchers should compare the same age group at more than one time (Donnellan and Trzesniewski 2009), a challenge that cross-temporal meta-analyses can address (Twenge 2001, 2008; Twenge et al. 2008). Cross-temporal meta-analyses examine birth cohort differences by comparing individuals of the same age at different time points and reporting the relation between mean scores of a measured characteristic (such as ingroup and outgroup attitudes) and the year of measurement. Cross-temporal analyses should include examinations of individual-level and aggregated data to determine if age, cohort, or an interaction of age and cohort are associated with changes in prejudicial attitudes (Trzesniewski and Donnellan 2010). For example, Malahy and colleagues (2009) predicted that increasing levels of income inequality disparities would cause undergraduate students to maintain (and even strengthen) their belief in a just world (Rubin and Peplau 1975). The results of the cross-temporal meta-analysis support the authors' hypothesis. Over the 34-year period of analysis, increases in income inequalities were associated with an increase in the number of individuals who reported a strong belief that the world was just and that people received the outcomes that they deserved (Malahy et al. 2009).

3.2. Regional Comparisons

An array of nationally representative data is available to examine intergroup conflict. Several researchers have used this approach to understand how attitudes toward gay, lesbian, and bisexual individuals are related to hate crimes (Alden and Parker 2005; Avery et al. 2007; Hicks and Lee 2006; Loftus 2001). For example, individuals who live in the South Central region of the United States are more likely to hold negative attitudes toward gays and lesbians (Loftus 2001), which may be related to the incidence of hate crime. Meta-analysis could discover whether or this trend remains valid across studies and in other regions of the United States, or if the finding is unique to a few studies.

Different geographical regions foster different political, cultural, and social climates that may affect intergroup conflict. For example, political ideology is a major force behind social dominance orientation (SDO; Pratto et al. 1994) and right-wing authoritarianism (RWA, Altemeyer 1981, 1988, 1998), which contribute to racism, sexism, and other forms of prejudice and discrimination (Sibley, Robertson, and Wilson 2006). Also, group-based conflict may differ depending which region of a country is being examined. In Sri Lanka, Schaller and Abeysinghe (2006) found that the Sinhalese are less willing to engage in conflict resolution and more likely to stereotype the Tamils in regions where the Sinhalese comprises the majority group than where they are the minority group.

One problem in regional comparative studies is non-random missing data from regions where frequent intergroup conflicts and violence occur. Data from countries with high risks and greater violence are less likely to appear in data archives and may therefore be inadvertently omitted from studies that focus on these sources. However, their risk and violence are assessed in international datasets such as CIRI with representative measures suggest possible reasons for missing data. Thus, they need not be completely omitted from such research.

4. Limitations of Meta-Analysis

Meta-analysis is a method to synthesize extant research, and as such, suffers from much the same limitations as primary studies. First, the results of a meta-analysis depend entirely on the quality of available primary studies (Coyne et al. 2009; Eysenck 1978, Wilson and Rachman 1983). As such, it is important to evaluate methodological quality when selecting studies, and to include any estimation of selection criteria in order to gauge its possible impact on the final results. Second, although meta-analysis may be employed in hypothesis-testing, we caution researchers against making causal statements on the basis of meta-analysis. If the meta-analysis includes only primary studies employing true experiments, then the effect size is gauging a causal difference (Bettencourt and Miller 1996; Johnson and Eagly 2000); even here, moderator values are likely to be correlational, qualifying any findings.

Often, limitations to meta-analysis are related to misapplication of the method or its basic assumptions (Ioannidis and Lau 2001). Although meta-analytic methods are often criticized as combining “apples and oranges,” or comparing phenomena from qualitatively different studies, a meta-analytic perspective would turn the question into a moderator (e.g., Cooper et al. 2009; Johnson et al. 2007). For example, do studies assessing social dominance orientation (Pratto et al. 1994) with various ethnic groups obtain the same results when one ethnic group is compared to another?

5. Conclusion

Human societies comprise various components that interact over time at multiple levels of organization. As such, a comprehensive, interdisciplinary approach to understanding intergroup conflict may include modeling of multi-level factors and interrelations that underlie human group-based processes (Diez-Roux 2007). Such an approach may consider incorporating qualitative reviews to inform or explicate meta-analytic findings; qualitative methods offer a richer and more comprehensive understanding of quantitative findings. Longitudinal designs are not the only designs by which researchers may examine temporal changes. Cumulative sources of data are available today, including, for example, indices of inequality and development (e.g., Gini coefficient, Human Development Index), prejudicial attitudes (e.g., the Eurobarometer), and frequency of war and violence (e.g., from the World Health Organization) from regions all over the world across various time points. Meta-analysis of these data provides an alternative approximation to longitudinal designs.

There are several important methodological issues associated with analyzing trends using archival and secondary data, the most obvious being temporal dependency. Statistical models must be correctly specified in order to account for data dependency (Kenny and Judd 1986). Multi-level data structures, which provide insights into the level at which changes occur, also require specialized methods for treating nested data (Bryk and Raudenbush 1987; Kenny, Kashy, and Bolger 1998). Methodological advances in analytical strategies such as multi-level modeling and time-series analyses allow researchers to answer questions pertaining to time effects and higher-level socio-structural factors. Meta-analytic methods offer similar solutions. Their strength over other quantitative methods lies in their routine ability to examine whether group comparisons vary across decades, cohorts, and generations, something that is extremely difficult using primary-data-collection strategies. In other words, meta-analysis allows moderator analyses of temporal data. The use of this strategy allows researchers to examine broader, higher-level moderators of intergroup phenomena such as social inequality indices. This feature is critically important for advancing knowledge and informing structural interventions and policies.

References

  1. Adorno Theodor W., Frenkel-Brunswik Else, Levinson Daniel J., Sanford R. Nevitt. The Authoritarian Personality. Harper; New York: 1950. [Google Scholar]
  2. Alden Helena L., Parker Karen F. Gender Role Ideology, Homophobia and Hate Crime: Linking Attitudes to Macro-Level Anti-Gay and Lesbian Hate Crimes. Deviant Behavior. 2005;26(4):321–43. [Google Scholar]
  3. Altemeyer Bob. The Other “Authoritarian Personality.”. In: Zanna Mark P., editor. Advances in Experimental Social Psychology. Academic Press; San Diego: 1998. pp. 47–92. [Google Scholar]
  4. Altemeyer Bob. Right-Wing Authoritarianism. University of Manitoba Press; Winnipeg, Canada: 1981. [Google Scholar]
  5. Altemeyer Bob. Enemies of Freedom: Understanding Right-Wing Authoritarianism. Jossey-Bass; San Francisco: 1988. [Google Scholar]
  6. Avery Alison, Chase Justin, Johansson Linda, Litvak Samantha, Montero Darrel, Wydra Michael. America's Changing Attitudes Toward Homosexuality, Civil Unions, and Same-Gender Marriage: 1977–2004. Social Work. 2007;52(1):71–9. doi: 10.1093/sw/52.1.71. [DOI] [PubMed] [Google Scholar]
  7. Becker Betsy J. Multivariate Meta-Analysis. In: Tinsley HEA, Brown SD, editors. Handbook of Applied Multivariate Statistics and Mathematical Modeling. Academic Press; 2000. pp. 499–525. [Google Scholar]
  8. Begg Colin B., Mazumdar Madhuchhanda. Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics. 1994;50(4):1088–1101. [PubMed] [Google Scholar]
  9. Bettencourt B. Ann, Miller Norman. Gender Differences in Aggression as a Function of Provocation: A Meta-Analysis. Psychological Bulletin. 1996;119(3):422–47. doi: 10.1037/0033-2909.119.3.422. [DOI] [PubMed] [Google Scholar]
  10. Bryk Anthony S., Raudenbush Stephen W. Application of Hierarchical Linear Models to Assessing Change. Psychological Bulletin. 1987;101(1):147–58. [Google Scholar]
  11. Campbell Donald T., Stanley Julian C., Gage Nathaniel. L. Experimental and Quasi-Experimental Designs for Research. Rand McNally; Chicago: 1969. [Google Scholar]
  12. Cheung Mike W.-L. A Model for Integrating Fixed-, Random-, and Mixed-Effects Meta-Analyses into Structural Equation Modeling. Psychological Methods. 2008;13(3):182–202. doi: 10.1037/a0013163. [DOI] [PubMed] [Google Scholar]
  13. Cohen Patricia, Cohen Jacob, Aiken Leona S., West Stephen G. The Problem of Units and the Circumstance for POMP. Multivariate Behavioral Research. 1999;34(3):315–46. [Google Scholar]
  14. Cohn Ellen G., Rotton James. Assault as a Function of Time and Temperature: A Moderator-Variable Time-Series Analysis. Journal of Personality and Social Psychology. 1997;72(6):1322–34. [Google Scholar]
  15. Cooper Harris, DeNeve Kristina, Charlton Kelly. Finding the Missing Science: The Fate of Studies Submitted for Review by a Human Subjects Committee. Psychological Methods. 1997;2(4):447–52. [Google Scholar]
  16. Cooper Harris M., Hedges Larry V. The Handbook of Research Synthesis. Russell Sage Foundation; New York: 1994. [Google Scholar]
  17. Cooper Harris M., Hedges Larry V., Valentine Jeff C. The Handbook of Research Synthesis and Meta-analysis. Russell Sage Foundation; New York: 2009. [Google Scholar]
  18. Costa Paul T., McCrae Robert R. An Approach to the Attribution of Aging, Period, and Cohort Effects. Psychological Bulletin. 1982;92(1):238–50. [Google Scholar]
  19. Diez-Roux Ana V. Integrating Social and Biologic Factors in Health Research: A Systems View. Annals of Epidemiology. 2007;17(7):569–74. doi: 10.1016/j.annepidem.2007.03.001. [DOI] [PubMed] [Google Scholar]
  20. Donnellan M. Brent, Trzesniewski Kali H. How Should We Study Generational “Changes” – Or Should We? A Critical Examination of the Evidence for “Generation Me.”. Social and Personality Psychology Compass. 2009;3(5):775–84. [Google Scholar]
  21. Duval Sue, Tweedie Richard L. Trim and Fill: A Simple Funnel Plot Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics. 2000;56(2):455–63. doi: 10.1111/j.0006-341x.2000.00455.x. [DOI] [PubMed] [Google Scholar]
  22. Eagly Alice H, Johnson Blair T. Gender and Leadership Style: A Meta-Analysis. Psychological Bulletin. 1990;108(2):233–56. [Google Scholar]
  23. Eagly Alice H., Steffen Valerie J. Gender and Aggressive Behavior: A Meta-Analytic Review of the Social Psychological Literature. Psychological Bulletin. 1986;100(3):309–30. [PubMed] [Google Scholar]
  24. Egger Matthias, Smith George D., Schneider Martin, Minder Christoph. Bias in Meta-Analysis Detected by a Simple, Graphical Test. British Medical Journal. 1997;315(7109):629. doi: 10.1136/bmj.315.7109.629. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Eysenck Hans J. An Exercise in Mega-Silliness. American Psychologist. 1978;33:517. [Google Scholar]
  26. Feingold Alan. Gender Differences in Personality: A Meta-Analysis. Psychological Bulletin. 1994;116(3):429–56. doi: 10.1037/0033-2909.116.3.429. [DOI] [PubMed] [Google Scholar]
  27. Gleser Leon, J., Olkin Ingram. Stochastically Dependent Effect Sizes. In: Cooper Harris M., Hedges Larry V., Valentine Jeff C., editors. The Handbook of Research Synthesis and Meta-Analysis. Russell Sage Foundation; New York: 2009. pp. 357–76. [Google Scholar]
  28. Hedges Leon V., Olkin Ingram. Statistical Methods for Meta-Analysis. Academic Press; New York: 1985. [Google Scholar]
  29. Hedges Leon V., Tipton Elizabeth, Johnson Matthew C. Robust Variance Estimation in Meta-Regression with Dependent Effect Size Estimates. Reseach Synthesis Methods. 2010;1(1):39–65. doi: 10.1002/jrsm.5. [DOI] [PubMed] [Google Scholar]
  30. Hedges Leon V., Vevea Jack L. Fixed- and Random-Effects Models in Meta-Analysis. Psychological Methods. 1998;3(4):486–504. [Google Scholar]
  31. Hedges Leon V., Vevea Jack L. Selection Method Approaches. In: Rotstein HR, Sutton AJ, Borenstein M, editors. Publication Bias in Meta-analysis. John Wiley; Chichester: 2005. [Google Scholar]
  32. Hepworth Joseph T., West Stephen G. Lynching and the Economy: A Time-Series Reanalysis of Hovland and Sears (1940) Journal of Personality and Social Psychology. 1988;55:239–47. [Google Scholar]
  33. Herek Gregory M. Hate Crimes and Stigma-Related Experiences Among Sexual Minority Adults in the United States: Prevalence Estimates from a National Probability Sample. Journal of Interpersonal Violence. 2009;24(1):54–74. doi: 10.1177/0886260508316477. [DOI] [PubMed] [Google Scholar]
  34. Hicks Gary R., Lee Tien-tsung. Public Attitudes Toward Gays and Lesbians: Trends and Predictors. Journal of Homosexuality. 2006;51(2):57–77. doi: 10.1300/J082v51n02_04. [DOI] [PubMed] [Google Scholar]
  35. Hovland Carl I., Sears RR. Minor Studies of Aggression: Correlations of Economic Indices with Lynchings. Journal of Psychology. 1940;9(301):10. [Google Scholar]
  36. Hunter John E., Schmidt Frank L. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Sage; 2004. [Google Scholar]
  37. Hyde Janet S., Fennema Elizabeth, Lamon Susan J. Gender Differences in Mathematics Performance: A Meta-Analysis. Psychological Bulletin. 1990;107(2):139–55. doi: 10.1037/0033-2909.107.2.139. [DOI] [PubMed] [Google Scholar]
  38. Ioannidis John, Lau Joseph. Evolution of Treatment Effects over Time: Empirical Insight from Recursive Cumulative Metaanalyses. Proceedings of the National Academy of Sciences of the United States of America. 2001;98(3):831. doi: 10.1073/pnas.021529998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Johnson Blair T., Boynton Marcella H. Cumulating Evidence about the Social Animal: Meta-Analysis in Social-Personality Psychology. Social and Personality Psychology Compass. 2008;2(2):817–41. [Google Scholar]
  40. Johnson Blair T., Eagly Alice H. Quantitative Synthesis of Social Psychological Research. In: Reis HT, Judd CM, editors. Handbook of Research Methods in Social and Personality Psychology. Cambridge University Press; London: 2000. pp. 496–528. [Google Scholar]
  41. Johnson Blair T., Scott-Sheldon Lori A. J., Snyder Leslie B., Noar Seth, Huedo-Medina Tania B. Contemporary Approaches to Meta-Analysis of Communication Research. In: Slater MD, Hayes A, Snyder LB, editors. The Sage Guide to Advanced Data Analysis Methods for Communication Research. Sage; Thousand Oaks, CA: 2008. [Google Scholar]
  42. Judd Charles M., Kenny David A., McClelland Gary H. Estimating and Testing Mediation and Moderation in Within-Subject Designs. Psychological Methods. 2001;6(2):115–34. doi: 10.1037/1082-989x.6.2.115. [DOI] [PubMed] [Google Scholar]
  43. Kalaian Sema A., Kasim Rafa M. Multilevel Methods for Meta-Analysis. In: O'Connell Ann A., McCoach D. Betsy., editors. Multilevel Modeling for Educational Data. 2008. pp. 315–43. Information Age. [Google Scholar]
  44. Kenny David A., Judd Charles M. Consequences of Violating the Independence Assumption in Analysis of Variance. Psychological Bulletin. 1986;99(3):422–31. [Google Scholar]
  45. Kenny David A., Kashy Deborah A., Bolger Niall. Data Analysis in Social Psychology. In: Gilbert D, Fiske S, Lindzey G, editors. Handbook of Social Psychology. McGraw-Hill; New York: 1998. pp. 233–65. [Google Scholar]
  46. Kirk Roger E. Experimental Design: Procedures for the Behavioral Sciences. Brooks/Cole; Belmont CA: 1995. [Google Scholar]
  47. Krug SE, Kulhavy RW. Personality Differences across Regions of the United States. The Journal of Social Psychology. 1973;91(1):73–79. doi: 10.1080/00224545.1973.9922648. [DOI] [PubMed] [Google Scholar]
  48. Light Richard J., Pillemer David B. Summing Up: The Science of Reviewing Research. Harvard University Press; Cambridge, MA: 1984. [Google Scholar]
  49. Lipsey Mark W. Identifying Potentially Interesting Variables and Analysis Opportunities. In: Cooper HM, Hedges LV, editors. The Handbook of Research Synthesis. Russell Sage Foundation; New York, NY: 1994. pp. 111–23. [Google Scholar]
  50. Loftus Jeni. America's Liberalization in Attitudes toward Homosexuality, 1973 to 1998. American Sociological Review. 2001;66:762–82. [Google Scholar]
  51. Malahy Lori W., Rubinlicht Michelle A., Kaiser Cheryl R. Justifiying Inequality: A Cross-Temporal Investigation of U.S. Income Disparities and Just-World Beliefs from 1973 To 2006. Social Justice Research. 2009;22:369–83. [Google Scholar]
  52. McArdle John J. Latent Variable Modeling of Differences and Changes with Longitudinal Data. Annual Review of Psychology. 2009;60:577–605. doi: 10.1146/annurev.psych.60.110707.163612. [DOI] [PubMed] [Google Scholar]
  53. Moher David, Liberati Alessandro, Tetzlaff Jennifer, Altman Douglas G., the PRISMA Group Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(6):e1000097. doi: 10.1371/journal.pmed.1000097. doi:10.1371/journal.pmed1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Moher David, Cook Deborah J., Eastwood Susan, Olkin Ingram, Rennie Drummond, et al. Improving the Quality of Reporting of Meta-Analysis of Randomized Controlled Trials: The QUOROM statement. Lancet. 1994;354:1896–1900. doi: 10.1016/s0140-6736(99)04149-5. [DOI] [PubMed] [Google Scholar]
  55. Mulrow CD. Systematic Reviews: Rationale for Systematic Reviews. British Medical Journal. 1994;309(6954):597. doi: 10.1136/bmj.309.6954.597. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Overton Willis F. Developmental Psychology: Philosophy, Concepts, and Methodology. Handbook of Child Psychology. 1998;1:107–88. [Google Scholar]
  57. Pettigrew Thomas F., Tropp Linda R. A Meta-Analytic Test of Intergroup Contact Theory. Journal of Personality and Social Psychology. 2006;90(5):751. doi: 10.1037/0022-3514.90.5.751. [DOI] [PubMed] [Google Scholar]
  58. Pratto Felicia, Sidanius Jim, Stallworth Lisa M., Malle Bertram F. Social Dominance Orientation: A Personality Variable Relevant to Social Roles and Intergroup Relations. Journal of Personality and Social Psychology. 1994;67(4):741–63. [Google Scholar]
  59. Ram Nilam, Grimm Kevin J. Methods and Measures: Growth Mixture Modeling: A Method for Identifying Differences in Longitudinal Change among Unobserved Groups. International Journal of Behavioral Development. 2009;33(6):565. doi: 10.1177/0165025409343765. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Raudenbush Stephen W., Bryk Anthony S. Applications in Meta-Analysis and other Cases where Level-1 Variances are Known. In: Raudenbush SW, Bryk AS, editors. Hierarchical Linear Models: Applications and Data Analysis Methods. Sage; Thousand Oaks, CA: 2002. pp. 205–27. [Google Scholar]
  61. Reis Harry T., Gable Shelly L. Event-Sampling and other Methods for Studying Everyday Experience. In: Reis HT, Judd CM, editors. Handbook of Research Methods in Social and Personality Psychology. Cambridge University Press; Cambridge, MA: 2000. pp. 190–222. [Google Scholar]
  62. Richard FD, Bond Charles F., Stokes-Zoota Juli J. One Hundred Years of Social Psychology Quantitatively Described. Review of General Psychology. 2003;7(4):331–63. [Google Scholar]
  63. Robinson William S. Ecological Correlations and the Behavior of Individuals. American Sociological Review. 1950;15(2):351–57. [Google Scholar]
  64. Rosenthal R, DiMatteo M. Robin. Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews. Annual Review of Psychology. 2001;52(1):59–82. doi: 10.1146/annurev.psych.52.1.59. [DOI] [PubMed] [Google Scholar]
  65. Ryder Norman B. The Cohort as a Concept in the Study of Social Change. American Sociological Review. 1965;30(6):843–61. [PubMed] [Google Scholar]
  66. Schaie K. Warner. A General Model for the Study of Developmental Problems. Psychological Bulletin. 1965;64(2):92–107. doi: 10.1037/h0022371. [DOI] [PubMed] [Google Scholar]
  67. Shadish William R. Campbell and Rubin: A Primer and Comparison of Their Approaches to Causal Inference in Field Settings. Psychological Methods. 2010;15(1):3–17. doi: 10.1037/a0015916. [DOI] [PubMed] [Google Scholar]
  68. Shavelson Richard J. Statistical Reasoning for the Behavioral Sciences. Allyn & Bacon; 1996. [Google Scholar]
  69. Simonton Dean K. Qualitative and Quantitative Analyses of Historical Data. Annual Review of Psychology. 2003;54(1):617–40. doi: 10.1146/annurev.psych.54.101601.145034. [DOI] [PubMed] [Google Scholar]
  70. Sterling TD, Rosenbaum WL, Weinkam JJ. Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa. American Statistician. 1995;49:108–12. [Google Scholar]
  71. Stewart Abigail J., Healy Joseph M. Linking Individual Development and Social Changes. American Psychologist. 1989;44(1):30–42. [Google Scholar]
  72. Trzesniewski Kali H., Donnellan M. Brent. Rethinking “Generation Me”: A Study of Cohort Effects from 1976–2006. Perspectives in Psychological Science. 2010;5:58–75. doi: 10.1177/1745691609356789. [DOI] [PubMed] [Google Scholar]
  73. Trzesniewski Kali H., Donnellan M. Brent, Robins Richard W. Is “Generation Me” Really More Narcissistic than Previous Generations? Journal of Personality. 2008;76:903–17. [Google Scholar]
  74. Twenge Jean M. Birth Cohort Changes in Extraversion: A Cross-Temporal Meta-Analysis, 1966–1993. Personality and Individual Differences. 2001;30(5):735–48. [Google Scholar]
  75. Twenge Jean M. Generation Me, the Origins of Birth Cohort Differences in Personality Traits, and Cross-Temporal Meta-Analysis. Social and Personality Psychology Compass. 2008;2(3):1440–54. [Google Scholar]
  76. Twenge Jean M., Campbell W. Keith. Age and Birth Cohort Differences in Self-Esteem: A Cross-Temporal Meta-Analysis. Personality and Social Psychology Review. 2001;5(4):321. [Google Scholar]
  77. Twenge Jean M., Konrath Sara, Foster Joshua D., Campbell W. Keith, Bushman Brad J. Egos Inflating over Time: A Cross-Temporal Meta-Analysis of the Narcissistic Personality Inventory. Journal of Personality. 2008;76(4):875–902. doi: 10.1111/j.1467-6494.2008.00507.x. [DOI] [PubMed] [Google Scholar]
  78. Vevea Jack L., Woods Carol M. Publication Bias in Research Synthesis: Sensitivity Analysis Using a Priori Weight Functions. Psychological Methods. 2005;10:428–43. doi: 10.1037/1082-989X.10.4.428. [DOI] [PubMed] [Google Scholar]
  79. West Stephen G., Hepworth Joseph T. Statistical Issues in the Study of Temporal Data: Daily Experiences. Journal of Personality. 1991;59(3):609–62. doi: 10.1111/j.1467-6494.1991.tb00261.x. [DOI] [PubMed] [Google Scholar]
  80. West Stephen G., Thoemmes Felix. Campbell's and Rubin's Perspectives on Causal Inference. Psychological Methods. 2010;15(1):18–37. doi: 10.1037/a0015917. [DOI] [PubMed] [Google Scholar]
  81. Wilson G.Terence, Rachman SJ. Meta-Analysis and the Evaluation of Psychotherapy Outcome: Limitations and Liabilities. Journal of Consulting and Clinical Psychology. 1983;51:54–64. doi: 10.1037//0022-006x.51.1.54. [DOI] [PubMed] [Google Scholar]
  82. Woodruff Diana S., Birren James E. Age Changes and Cohort Difference in Personality. Developmental Psychology. 1972;6(2):252–59. [Google Scholar]

RESOURCES