New Trends in Gender and Mathematics Performance: A Meta-Analysis

Abstract

In this paper, we use meta-analysis to analyze gender differences in recent studies of mathematics performance. First, we meta-analyzed data from 242 studies published between 1990 and 2007, representing the testing of 1,286,350 people. Overall, d = .05, indicating no gender difference, and VR = 1.08, indicating nearly equal male and female variances. Second, we analyzed data from large data sets based on probability sampling of U.S. adolescents over the past 20 years: the NLSY, NELS88, LSAY, and NAEP. Effect sizes for the gender difference ranged between −0.15 and +0.22. Variance ratios ranged from 0.88 to 1.34. Taken together these findings support the view that males and females perform similarly in mathematics.

Policy decisions, such as funding for same-sex education, as well as the continuing stereotype that girls and women lack mathematical ability, call for up-to-date information about gender differences in mathematical performance. Such stereotypes can discourage women from entering or persisting in careers in science, technology, engineering, and mathematics (STEM). Today women earn 45% of the undergraduate degrees in mathematics (NSF, 2008a), but women make up only 17% of university faculty in mathematics (NSF, 2008b). We report on a meta-analysis of recent studies of gender and mathematics. We estimate the magnitude of the gender difference and test whether it varies as a function of factors such as age and the difficulty level of the test.

Stereotypes about Gender and Mathematics

Mathematics and science are stereotyped as male domains (Fennema & Sherman, 1977; Hyde, Fennema, Ryan, Frost, & Hopp, 1990b, Nosek, et al, 2009). Stereotypes about female inferiority in mathematics are prominent among children and adolescents, parents, and teachers. Although children may view boys and girls as being equal in mathematical ability, they nonetheless view adult men as being better at mathematics than adult women (Steele, 2003). Implicit attitudes that link males and mathematics have been demonstrated repeatedly in studies of college students (e.g., Kiefer & Sekaquaptewa, 2007; Nosek, Banaji, & Greenwald, 2002).

Parents believe that their sons' mathematical ability is higher than their daughters'. In one study, fathers estimated their sons' mathematical “IQ” at 110 on average, and their daughters' at 98; mothers estimated 110 for sons and 104 for daughters (Furnham et al., 2002; see also Frome & Eccles, 1998). Teachers, too, tend to stereotype mathematics as a male domain. In particular, they overrate boys' ability relative to girls' (Li, 1999; but see Helwig, Anderson, & Tindal, 2001).

These stereotypes are of concern for several reasons. First, in the language of cognitive social learning theory, stereotypes can influence competency beliefs or self-efficacy; correlational research does indeed show that parents' and teachers' stereotypes about gender and mathematics predict children's perceptions of their own abilities, even with actual mathematics performance controlled (Bouchey & Harter, 2005; Frome & Eccles, 1998; Keller, 2001; Tiedemann, 2000). Competency beliefs are important because of their profound effect on individuals' selection of activities and environments (Bandura, 1997; Bussey & Bandura, 1999). According to an earlier meta-analysis, girls report lower mathematics competence than boys do, although the difference is not large (d = +.16, Hyde et al., 1990b). In recent studies, elementary-school boys still report significantly higher mathematics competency beliefs than girls do (Else-Quest, Hyde, & Linn, 2010; Fredrick & Eccles, 2002; Lindberg, Hyde, & Hirsch, 2008; Watt, 2004).

A second concern is that stereotypes can have a deleterious effect on actual performance. Stereotype threat effects (Steele, 1997; Steele & Aronson, 1995) have been found for women in mathematics. In the standard paradigm, half the participants (talented college students) are told that the math test they are about to take typically shows gender differences (threat condition), and the other half is told that the math test is gender fair and does not show gender differences (control). Studies find that college women underperform compared with men in the threat condition but perform equal to men in the control condition, indicating that priming for gender differences in mathematics indeed impairs girls' math performance (e.g., Ben-Zeev, Fein, & Inzlicht, 2005; Cadinu, Maass, Rosabianca, & Kiesner, 2005; Johns, Schmader, & Martens, 2005; Quinn & Spencer, 2001; Spencer, Steele, & Quinn, 1999). Stereotype threat effects have been found in children as early as kindergarten (Ambady, Shih, Kim, & Pittinsky, 2001). Other research, measuring implicit stereotypes about gender and math, has found that these implicit stereotypes predict performance in a calculus course (Kiefer & Sekaquaptewa, 2007).

Stereotypes play a role in policy decisions as well as personal decision-making. For example, schools and states may base decisions to offer single-sex mathematics classes on the belief that these gender differences exist (Arms, 2007).

Gender and Mathematics Performance

The stereotypes about female inferiority in mathematics stand in distinct contrast to the scientific data on actual performance. A 1990 meta-analysis found an effect size of d = 0.15, males scoring higher, for gender differences in mathematics performance averaged over all samples; however, in samples of the general population (i.e. national samples, classrooms – as opposed to exceptionally precocious or low ability samples), females scored higher but by a negligible amount (d = −0.05; Hyde, Fennema, & Lamon, 1990a). Hedges and Nowell (1995), using data sets representing large probability samples of American adolescents, found d = 0.03 to 0.26 across the different data sets. Moreover, girls earn better grades in mathematics courses than boys through the end of high school (Dwyer & Johnson, 1997; Kenney-Benson et al., 2006; Kimball, 1989). In short, previous research showed that gender differences in mathematics performance were very small and, depending on the sample and outcome measure, sometimes favored boys and sometimes favored girls.

Several features of the 1990 meta-analysis (Hyde et al., 1990a) warrant more detailed description. Using computerized literature searches, the researchers identified 100 useable studies, which yielded 254 independent effect sizes representing the testing of more than 3 million persons. One key moderator analysis examined the magnitude of gender differences as a function of age and cognitive level of the test (computation was considered the lowest level, understanding of concepts was considered intermediate, and complex problem-solving was considered the highest level). Girls performed better than boys at computation in elementary school and middle school but the differences were small (d = −0.20 and −0.22, respectively) and there was no gender difference in high school. There was no gender difference in understanding of mathematical concepts at any age. For complex problem solving, there was no gender difference in elementary or middle school, but a gender difference favoring males emerged in high school (d = 0.29). This last gender difference, although small, is of concern because complex problem solving is crucial for STEM careers.

A second moderator analysis examined the magnitude of gender differences in mathematics performance as a function of the ethnicity of the sample (Hyde et al., 1990a). The striking finding was that the small gender difference favoring males was found for Whites (d = 0.13), but not for Blacks (−0.02) or Latinos (0.00).

Depth of Knowledge

Traditionally, researchers maintained that girls might do as well as, or even better than boys on tests of computation, which require relatively simple cognitive processes (e.g., Anastasi, 1958). These same researchers concluded that male superiority emerged for tests requiring more advanced cognitive processing, such as complex problem solving. The 1990 meta-analysis by Hyde and colleagues provided some support for these ideas, although the gender difference in complex problem solving did not appear until the high school years and was not large even then.

Current mathematics education researchers conceptualize this issue of complexity of cognitive processes as a question of item demand and the depth of knowledge required to solve a particular problem. Webb (1999) developed a 4-level Depth of Knowledge framework to identify the cognitive difficulty of mathematics items on standardized assessments. In this framework, Level 1 (Recall) includes the recall of information such as facts or definitions, as well as performing simple algorithms. Level 2 (Skill/Concept) includes items that require students to make decisions about how to approach a problem. These items typically ask students to classify, organize, estimate, or compare information. Level 3 (Strategic Thinking) includes complex and abstract cognitive demands that require students to reason, plan, and use evidence. Level 4 (Extended Thinking) requires complex reasoning, planning, developing, and thinking over an extended period of time. Items at Level 4 require students to connect ideas within the content area or among content areas as they develop one problem-solving approach from many alternatives. This depth of knowledge framework was used to rate the cognitive demands of the tests that assess mathematics performance in the studies reviewed here.

New Trends

Cultural shifts have occurred since the 1980s that call for the reexamination of gender differences in mathematics. In the 1980s, a prominent explanation of male superiority in complex problem solving beginning in high school was gender differences in course choice (Meece, Eccles-Parsons, et al, 1982). Girls were less likely than boys to take advanced mathematics courses and advanced science courses. Because mathematical problem solving is an important component of chemistry and physics courses, students may learn those skills in science courses as much as in mathematics courses. Today, however, the gender gap in course taking has disappeared in all areas except physics. For the high school graduating class of 2005, 7.7% of boys and 7.8% of girls took calculus; 57.8% of girls and 50.6% of boys took chemistry; and 32.8% of girls and 36.8% of boys took physics (NSF, 2008c). Insofar as courses taken by students influence their mathematics performance, we would expect that the gender difference in complex problem solving in high school would have narrowed.

In addition, cross-national data show that the gender gap in mathematics performance narrows or even reverses in societies with more gender equality (e.g., Sweden and Iceland), compared with those with more gender inequality (e.g., Turkey) (Else-Quest, Hyde, & Linn, 2010; Guiso, Monte, Sapienza, & Zingales, 2008). Insofar as the United States has moved toward gender equality over the past 30 to 40 years, the gender gap in mathematics performance should have narrowed.

Findings from a recent analysis of data from state assessments of mathematics performance provide evidence that the gender gap in mathematics performance in the U.S. has indeed diminished or even vanished (Hyde, Lindberg, Linn, Ellis, & Williams, 2008). Those data had several limitations and raised some questions that deserve analysis in a larger study. First, they were based just on tests administered by the states to satisfy the requirements of No Child Left Behind Legislation. Items tapping Levels 3 or 4 depth of knowledge were notably absent. Second, the data were derived only from students in grades 2 through 11; trends in gender differences beyond grade 11 (age 17) could therefore not be assessed. Third, the distributions of male and female performance were available for only part of the sample. The available results raised intriguing questions about gender differences in complex problem solving since in some subgroups females outperformed males at the high end of the distribution.

Gender and Variability

Most of the research has focused on mean-level gender differences, but variability (variance) remains an issue even when means are similar. The greater male variability hypothesis was originally proposed in the 1800s and advocated by scientists such as Charles Darwin and Havelock Ellis, to explain why there was an excess of men both in homes for the mentally deficient and among geniuses (Shields, 1982). In modern statistical terms, the hypothesis is that, independent of mean-level differences, males have a greater variance than females do on the intellectual trait of interest. Thus, the hypothesis states that men are more likely than women to be at both the top and the bottom of the statistical distribution of mathematics performance. Typically the statistic that is computed is the variance ratio (VR), the ratio of the male variance to the female variance. Thus values > 1.0 indicate greater male variability. Based on test norming data, Feingold (1992a) found a VR of 1.11 for the DAT numerical ability, 1.20 for the SAT-Math, and 1.02 for the WAIS Arithmetic subtest. Hedges and Nowell (1995) found VR's ranging between 1.05 and 1.25 for mathematics tests administered to national samples of adolescents such as the NLSY and NELS:88. The analysis of recent state assessment data, described earlier, found VR's ranging between 1.11 and 1.21 (Hyde et al., 2008). Thus there is some evidence of greater male variability in mathematics performance, although the variance ratios are not terribly lopsided. The greater male variability hypothesis, of course, is a description of the data, not an explanation for it, but if true, it could partially account for findings of an excess of males at very high levels of mathematical performance (Hedges & Friedman, 1993). One goal of this meta-analysis was to re-assess the greater male variability hypothesis for mathematics performance using contemporary data.

The Current Study

Several factors warrant a new meta-analysis of research on gender and mathematics performance. First, approximately 18 years of new data have accumulated since the 1990 meta-analysis (Hyde et al., 1990a). Second, cultural shifts have occurred over the last two decades. Specifically, girls are now taking advanced mathematics courses and some science courses in high school at the same rate as boys are, closing the gap in course choice. The magnitude of gender differences in mathematics performance is expected to be even smaller than it was in the 1990 meta-analysis; of particular interest is the gender difference favoring boys in complex problem-solving in high school, and whether this difference has narrowed in recent years. Third, statistical methods of meta-analysis have advanced. At the time of the 1990 meta-analysis, only fixed-effects models were available. Fixed-effects models have since been criticized, and random-effects and mixed models have been developed (Hedges & Vevea, 1998; Lipsey & Wilson, 2001). The current meta-analysis used a mixed-effects model, the advantages of which are detailed below.

Our goals in these meta-analyses were to provide answers to the following questions:

1. What is the magnitude of the gender difference in mathematics performance, using the d metric?

2. Does the direction or magnitude of the gender difference vary as a function of the depth of knowledge tapped by the test?

3. Developmentally, at what ages do gender differences appear or disappear?

4. Are there variations across U. S. ethnic groups, or across nations, in the direction or magnitude of the gender difference?

5. Has the magnitude of gender differences in mathematics performance declined from 1990 to 2007?

6. Do males display greater variance in scores and, if so, by how much?

Study 1 addressed these questions using traditional methods of meta-analysis that involve identifying all possible studies using article databases. Study 2 addressed the questions using an alternative method advocated by Hedges and Nowell (1995), which involves the analysis of recent large, national U.S. data sets based on probability sampling. Cross-national, probability sampled data sets have been analyzed by others (Else-Quest, Hyde, & Linn, 2010; Guiso et al., 2008; Penner, 2008).

Study 1

Method

Identification of studies

Computerized database searches of ERIC, PsycINFO, and Web of Knowledge were used to generate a pool of potential articles. To identify all articles that investigated mathematics performance, the following search terms were used: (math* or calculus or algebra or geometry) AND (performance or achievement or ability) NOT (mathematical model). This broad term was selected to capture the widest range possible of research conducted on this topic while avoiding studies that used computational modeling methodology to study unrelated phenomena. Search limits restricted the results to articles that discussed research with human populations and that were published in English between 1990 and 2007. The three database searches identified 10,816, 9,577, and 18,244 studies, respectively, which were considered for inclusion. Given the tremendous volume of studies identified, we decided to rely solely on this method of identification, a potential limitation.

Abstracts and citations were imported into RefWorks citation manager, and then each article was evaluated for inclusion based on the following criteria: (a) the title or abstract alluded to a measure of mathematics performance; (b) the study appeared to contain original data; (c) the study was conducted on a human population; (d) the sample included at least five males and five females. 3,941 studies met the aforementioned criteria. These articles were then printed and examined to determine whether they presented sufficient statistics for an effect size calculation. The final sample of studies included in study 1 utilized data from 242 articles, comprising 441 samples and 1,286,350 people.

Although the final sample of studies is large, some readers may wonder why so many potential studies were lost during the coding processes. Most cases of exclusion were based on one of three reasons: (1) Many of the articles identified in our search reported on data from the large national datasets included in Study 2 (NAEP, NELS, LSAY, NLSY) or from large international datasets (SIMS, TIMSS, PISA) that had been covered by other meta-analyses of gender differences in mathematics performance (Else-Quest, Hyde, & Linn, 2010; Guiso et al., 2008; Penner, 2008). Those articles were excluded from study 1 to avoid redundancy. (2) The searches also picked up reports published by state, local, and federal educational organizations, few of which contained individual-level data. (3) Finally, many articles did not contain enough data to compute effect sizes (e.g., results were not disaggregated by gender) and additional data could not be obtained from the study authors.

Coding the studies

If studies reported data from large datasets or longitudinal studies that were likely to create multiple publications, this was noted so as to avoid inclusion of non-independent effect sizes.

Several characteristics of each sample were coded as potential moderators: (a) age of the participants; (b) nationality of participants (American, Canadian, European, Australian/New Zealander, Asian, African, Latin American, or Middle Eastern); (c) for US samples, majority ethnic group of participants (Euro-American, African American, Asian American, Hispanic, other; mixed; or unreported), and (d) ability level of the participants (low ability, general ability, selective, highly selective).

Several aspects of the mathematics tests were also coded as potential moderators: (a) whether the test was time-limited; (b) whether the test included each of several problem types (multiple choice, short answer, open ended); (c) which types of mathematical content were included in the test (numbers and operations, algebra, geometry, measurement, data analysis and probability); (d) depth of knowledge (level 1 recall, level 2 skills/concepts, level 3 strategic thinking, level 4 extended thinking; levels are described in greater detail above, see also Webb, 1999) and (e) whether the test was specific to the local curriculum (i.e. based on published curricular standards for the region or developed in collaboration with local teachers, textbooks, and syllabi) or was relatively independent of the curriculum. Publication year was also coded as a potential moderator.

50 articles were double coded to determine inter-rater reliability. Percentage agreement for coding of the moderators was > 95% for all variables.

Effect size computation

Cohen's d (Cohen, 1988) is the effect size for the standardized mean difference between two groups on a continuous variable (e.g., the mean difference between males and females on a continuous measure of mathematics performance). Thus, for each independent sample within an article, d was computed, with d = (M_m − M_f) / s_w and M_m = the mean for males, M_f = the mean for females, and s_w = the pooled within-gender standard deviation. Whenever possible, separate effect sizes were computed for independent groups within each sample (e.g., different age groups, Blacks and Whites). If means and standard deviations were not available, the effect size was computed from other statistics such as t or F, using formulas provided by Lipsey and Wilson (2001). The complete list of samples, with corresponding effect sizes and variance ratios, can be found in the supplemental online material for this article.

In a few cases, more than one effect size was available for the same sample. When that happened, the following decision rules were used to select or calculate a single effect size for inclusion in the analyses: (1) If a study used a longitudinal design, we used the effect size from the most recent measurement. (2) If multiple effect sizes were available from the same time point but missing data produced different sample sizes for different measures, then the effect size with the largest corresponding sample size was selected for inclusion in the analyses. (3) If there were multiple effect sizes from the same time point with the same sample size, means and standard deviations were pooled, and the pooled values were used to compute the composite effect size that was included in the analyses.

Raw effect sizes were corrected for bias, and standard errors were calculated. Specifically, estimated population effect sizes were used, which adjust for upward-bias of effect sizes among small samples (Hedges, 1981). In addition, inverse variance weights were calculated for each effect size so that analyses could be weighted by inverse variance, a procedure that allows large samples to have more leverage in the analyses than small samples have (Lipsey & Wilson, 2001).

Variance ratio computation

Variance ratios were also computed for each study, with VR = variance_males / variance _females. Thus, a VR >1 denotes greater male variability, and a VR < 1 denotes greater female variability. Standard errors and inverse variance weights were calculated for each variance ratio. Analyses of the variance ratios were conducted as outlined by Katzman and Alliger (1992).

Data analyses

Results were analyzed using a mixed-effects model (Lipsey & Wilson, 2001) and SPSS macros by Wilson (2005). The mixed-effects model assumes that variability among effect sizes can be explained by both fixed factors (i.e. systematic differences due to moderators) and random factors (i.e. error variance). This approach is preferable to fixed-effects or random-effects models, which assume that all variability among effect sizes is accounted for by moderators or by error, respectively. In mixed-effects models, a random-effects variance component is computed based on the residual homogeneity after moderator effects have been accounted for. Then, inverse variance weights are recalculated with the random-effects variance component added, and the model is refit.

To measure heterogeneity of variance of the effect sizes, Q statistics were computed using weights based on the mixed-effects model. When homogeneity analyses indicated that there was significant heterogeneity among the effect sizes, moderator analyses were conducted to test whether characteristics of the mathematics assessments or characteristics of the samples could explain the variability among effects. These moderator analyses allowed us to explore whether systematic differences among the studies led to reliably different effects. Moderators were tested for significance using an analog to ANOVA in the case of categorical moderator variables (e.g., ethnicity), or an analog to multiple regression in the case of continuous moderator variables (e.g., publication year). The level of missing data for moderator variables (due to vague descriptions of mathematics measures and study procedures) made it untenable to conduct a simultaneous analysis of all moderators. The piece-wise approach is not ideal, but we believe that it is warranted in this case, because we were testing a priori hypotheses based on the findings of previous meta-analyses.

Results

Magnitude of gender differences

The overall weighted effect size, averaged over all studies, was d = +0.05, representing a negligible gender difference. Figure 1 shows the distribution of effect sizes, which is approximately normal and centered around 0. Heterogeneity analysis revealed that the set of effect sizes was significantly heterogeneous, Q_t (427) = 11478.74, p < .001. The random effects variance component was .070.

Moderator analyses

Given the heterogeneity among the effect sizes, we conducted analyses for suspected moderator variables. Table 1 displays the analyses for variations in effect size as a function of characteristics of the tests. Problem type (presence of multiple choice, short answer, and open ended questions) was the only aspect of the tests that significantly predicted heterogeneity among the effects, Q_b (3, 178) = 8.11, p < .05. The presence of multiple choice questions on exams predicted relatively better performance by males (β = .16), whereas the presence of short answer and open ended questions predicted relatively better performance by females (βs = −.02 and −.06, respectively).

The magnitude of the gender difference did not depend on whether there was a time limit for the test Q_b (1, 136) = 2.21, p = .14, or whether the test was curriculum-focused, Q_b (1, 422) = 3.01, p = .08. Similarly, there were no variations in the magnitude of the effect size as a function of problem content (numbers and operations, algebra, geometry, measurement, probability), Q_b (5, 204) = 8.55, p = .13, or depth of knowledge, Q_b (3, 119) = 1.51, p = .68.

Table 2 displays the analyses for variations in effect size as a function of characteristics of the sample. The magnitude of the gender difference varied significantly as a function of the selectivity of the sample, Q_b (3, 424) = 27.06, p < .001. For samples of the general population, d = +0.07, but d = +0.40 for highly selective samples.

Nationality was not a significant predictor of effect sizes, Q_b (7, 421) = 10.12, p = .18. All effects were small or negligible. Among US studies, effect sizes varied as a function of ethnicity, Q_b (1, 89) = 10.00, p < .01. Samples composed mainly of Whites showed d = +0.13, whereas for ethnic-minority samples, d = −0.05. Although we would have preferred to report results from these groups separately by ethnicity, the number of samples was too small to permit this.

The analysis also indicated that age was a significant moderator, Q_b (5, 423) = 44.75, p < .001. Gender differences were negligible in elementary-school and middle-school-aged children and reached a peak of d = +0.23 in high school. The gender difference then declined for college-age samples and adults.

To test for trends over time from 1990 to 2007, an analog to multiple regression was performed, using year of publication to predict effect size. This analysis indicated that publication year was not a significant predictor of effect size, Q_b (1, 427) = 1.05, p = .31.

A final analysis explored the interaction of age and depth of knowledge. Thus, our next analysis focused on both depth of knowledge and different age groups. This analysis was limited by the fact that many articles identified in our original search provided insufficient information to be able to code depth of knowledge, so they were not useable in this analysis. Furthermore, of those that had usable information about test content, only a small proportion of tests included items that tapped complex problem solving (Level 3 or 4). Some studies located in the literature search involved problem solving at Level 3 or 4 but reported only qualitative data on students' approach to the problem, with no data on actual performance. Although these studies could not be included in the meta-analysis, they suggest the value of looking at more complex tasks. Table 3 provides a breakdown of effect sizes by age and depth of knowledge. Of particular interest in this analysis was whether a gender difference in complex problem solving would be seen among high school and college students, as was found in Hyde's 1990 meta-analysis. Our results showed that there was a small gender difference favoring high school males on tests that included problems at Levels 3 or 4 (d = +0.16), but the effect was reversed among college students (d = −0.11). However, these findings are based on small numbers of studies, and therefore cannot be considered robust.

Gender differences in variability

A mixed-effects analysis of gender differences in variance was conducted in parallel to the effect size analyses reported above. The overall weighted variance ratio, averaged over all studies, was VR = 1.07, indicating a slightly larger variance for males than for females. The residual variance component was .073.

Study 2

Method

Large United States datasets

Study 1 excluded articles that reported secondary data analyses from large national datasets, because original data from those studies were acquired directly for a separate analysis, which constitutes Study 2. Datasets were included in Study 2 if (a) they included relevant information about math performance, (b) represented data collected after 1990, (c) were nationally representative with a large sample size, and (d) provided statistics for both males and females. International datasets were excluded from Study 2 because they have been thoroughly reviewed elsewhere in the literature (see Else-Quest, Hyde, & Linn, 2010). The following large U.S. datasets were analyzed in Study 2: The National Longitudinal Survey of Youth - 1997 (NLSY97, U.S. Bureau of Labor Statistics, n.d.), The National Educational Longitudinal Study (NELS88, National Center for Educational Research n.d.a), The Longitudinal Study of American Youth (LSAY, n.d.), and the National Assessment of Educational Progress (NAEP, National Center for Education Research, n.d.b).

The NLSY - 97 began data collection in 1997 and followed students each year until 2002. At the first assessment 58.3% of participants were White, 27.0% were Black, 1.7% were Asian American, 0.8% were American Indian, and 12.4% did not report ethnicity. Math achievement was measured using the PIAT-R (Markwardt, 1998). During round one of data collection, the test was administered to all participants who were in the ninth grade or lower. During round two of data collection, the test was administered to all participants who had taken the test during round one, as well as those who were at least 12 years old on December 31, 1996. The PIAT-R consists of multiple choice items about three areas of math content: 1) foundations (i.e. number, size, and shape discretion), 2) basic facts (i.e. addition, subtraction, multiplication, division), and 3) applications (i.e. algebra, geometry, fractions, word problems, and numerical relationships). The PIAT-R math assessment begins with an age-appropriate question and increases or decreases in difficulty until the youth establishes a “basal,” that is, when the youth correctly answers five consecutive questions. Once a basal is reached the questions increase in difficulty until a “ceiling” is reached when the youth incorrectly answers five of seven consecutive questions. The ceiling is then adjusted for incorrect responses given between the basal and the ceiling and is standardized with a mean of 100 and a standard deviation of 15. The current study determines math achievement using the standardized score of the PIAT-R math for each assessment.

The NELS88 is a longitudinal study which began examining 8th graders in 1988 and followed these youth in 10th and 12th grade in 1990 and 1992. At the first assessment 67.0% of participants were White, 12.2% were Black, 12.7% were Latino, 6.3% were Asian American, and 1.6% did not report ethnicity. The NELS math assessment was developed by Educational Testing Services and consisted of multiple choice questions. Item content included arithmetic, algebra, geometry, data and probability, and advanced topics. One version of the test was administered at the base year, and three versions of the test were administered at the first and second follow up. Based on their performance on the math test in the base year, students were divided into three groups (low, moderate, and high ability) and were assigned versions of the math test at the first and second follow up in accordance with their ability. Each test, regardless of ability, assessed skill/knowledge, comprehension, and problem solving in all five content areas described above. In addition to the multiple choice test, a constructed response test was given to 12th graders at the 1992 assessment; this test involved items examining measurement, geometry, and data analysis.

The LSAY followed youth from 1987 to 1992 with assessments at each year from grades 7 to 12. At the first assessment 70% of participants were White, 11% were Black, 9.2% were Latino, 3.5% were Asian American, 1.5% were American Indian, and 4.8% did not report ethnicity. At each assessment students completed a multiple-choice math test that assessed skills in geometry, measurement, data analysis, algebra, and simple operations (for a complete list of problems, see http://lsay.msu.edu/instruments_006.html).

The NAEP math assessment was the only large U.S. database in the current study that was not longitudinal. It consisted of two different studies: the long-term trend assessment, and the main assessment.

The NAEP long-term trend assessment was given every four years from 1992 to 2004 to students aged 9, 13, and 17. Ethnic diversity was different for each assessment and age group, but consisted of Whites (M = 74.07%, SD = 4.87), Blacks (M = 14.78%, SD = 1.31), Latinos (M = 7.82%, SD = 3.31) and those who did not report ethnicity (M = 3.29%, SD = 1.38). The math assessment in this study has remained virtually unchanged since its inception in 1978. It included both multiple-choice and short constructed response items which focus on math skills including number operations, measurement, algebra, and geometry.

In contrast to the long-term trend data, the NAEP main assessment selected students by grade rather than by age. The main assessment was given to 4th and 8th graders every two years from 1990 to 2007. Twelfth graders were included in 1990 through 2000. Ethnic diversity was different for each assessment and age group, but consisted of Whites (M = 67.07%, SD = 6.43), Blacks, (M =15.94%, SD = 0.99), Latinos (M = 11.89%, SD = 4.89), Asian Americans (M = 3.17%, SD = 1.66), American Indians (M = 0.94%, SD = 0.42) and those who did not report ethnicity (M = 3.29%, SD = 1.38). The math portion of the NAEP main assessment used multiple choice, short constructed response, and long constructed response items. In addition to the math skills assessed in the long-term trend analysis, the main analysis also included skills in data analysis and probability.

A unique aspect of the NAEP assessments is that performance scores are constructed via IRT (item response theory). Thus, they are not a direct reflection of the number of problems any student got right or wrong. Rather, the pattern of correct responses is used to construct a “probable value” score that reflects the student's overall understanding of mathematics. See Mislevy, Johnson, and Muraki (1992) for a further discussion of this approach and its advantages in estimating population values. Our analyses are based on these probable value scale scores, along with the corresponding weights and standard errors reported by NAEP.

Data analysis

Data analysis for Study 2 was similar to Study 1. All effect sizes were calculated, and a mixed effects model was used to determine whether the effect sizes within each dataset were heterogeneous. If effect sizes were heterogeneous, a weighted ordinary least squares regression was applied to predict gender differences in math performance.

Moderating variables used in Study 2 were age, publication year, percentage of each type of problem (number sense, algebra, geometry, measurement), percentage of problems in each type of format (multiple choice, short answer, open ended), and percentage of Whites, Blacks, and Latinos in each sample. Similar to Study 1, variance ratios were also computed. More information about sample and test characteristics were available for the large datasets than were available for the studies uncovered in the literature reviews in Study 1. Therefore, with the exception of depth of knowledge, we were able to code moderator variables with more detail and many moderators that were coded as categorical variables in Study 1 were considered continuous variables in Study 2. For example, Study 1 coded whether the tests used multiple choice, short answer, or open ended questions, whereas Study 2 coded the percentage of question in each format.

Results

Across all datasets in Study 2, the average weighted effect size was d = +0.07. The average weighted variance ratio across all datasets was 1.09. The effect sizes were heterogeneous, Q_t(55) = 393.04, p < 0.001, with a random effects variance component of .001. Differences among the national datasets were a significant source of heterogeneity Q_b(4, 55) = 43.12, p < .001; therefore we describe findings from each dataset in turn.

Effect sizes for each assessment of the NLSY are presented in Table 4. The mean weighted effect size for all six assessments of the NLSY was d = +0.08. The average weighted variance ratio was 1.05. Effects sizes for NLSY-97 were homogenous, Q_w(5) = 5.72, p = .33.

Effect sizes for each assessment of the NELS:88 are presented in Table 5. The average weighted effect size across all eight assessment was d = +0.10. The average weighted variance ratio for the NELS:88 was 0.94. Effect sizes within the NELS:88 were heterogeneous, Q_w(7) = 18.66, p < 0.01.

Effect sizes for each assessment of the LSAY are presented in Table 6. Results for the LSAY indicated small or negligible gender differences for each assessment. The average weighted effect size for all six assessments was d = −0.07. The weighted average variance ratio was 1.26. Effect sizes were homogenous, Q_w(5) = 2.50, p = .78.

Effect sizes for each assessment of the NAEP are presented in Table 7. Results for NAEP indicated small or negligible gender differences at all grades. The average weighted effect size across all 18 assessments of the long-term trend data was d = +0.09. The average weighted effect size across all 18 of the main assessments was d = +0.06. The average weighted variance ratio for the long-term trend data was 1.13 and for the main assessment was 1.04. Effect sizes for both the long-term trend data and the main assessment were homogenous, Q_w(17) = 16.05, p = .52 and Q_w(17) = 12.88, p = .74, respectively.

The heterogeneity of the effect sizes across datasets indicates that these studies are not replications of each other but rather vary along some dimension(s). We therefore conducted additional moderation analyses to examine whether sample characteristics or test characteristics could explain the heterogeneity among effect sizes across datasets. These analyses were conducted using an analog to multiple regression, in which each level of each moderator was entered as a separate predictor of the studies' effect sizes. The resulting betas can be interpreted much like correlation coefficients, with positive values indicating an increase in effect size as the value of the moderator increases (relative advantage for males) and negative values indicating a decrease in effect size as the value of the moderator increases (relative advantage for females).

As seen in Table 8, two aspects of the tests accounted for heterogeneity among studies: problem type and mathematical content. With regard to problem type, tests with a higher proportion of multiple choice and open-ended items yielded smaller gender effect sizes, whereas tests with a higher proportion of short answer items yielded larger gender effect sizes. This finding surprised us, given that three of the big datasets (LSAY, NLSY, NELS) were 100% multiple choice, and only one of them had a negative overall effect size; if multiple choice items confer a significant female advantage, we might have expected negative effects across all three of those studies. Therefore, we conducted an additional analysis, looking just at the 36 NAEP effect sizes, which have variation in the proportion of multiple choice, short answer, and open-ended questions. When looking at just the NAEP effect sizes, we found a different pattern of results, such that males did better on tests with a greater proportion of multiple choice items (β = +.29), and females did better on tests with a greater proportion of short answer and open-ended items (βs = −.32 and −.19, respectively). Thus, problem type had a similar effect on gender differences in the NAEP as were found in study 1.

With regard to mathematical content, tests with a higher proportion of algebra items yielded smaller effect sizes (females performed relatively better), and tests with a higher proportion of measurement items yielded larger effect sizes (males performed relatively better). The other three types of mathematical content were not significant predictors of effect size in this analysis.

With regard to depth of knowledge, tests containing items at levels 3 or 4 yielded larger effect sizes (males performed better or females performed worse). All of the tests contained items at levels 1 and 2, and therefore we were not able to examine the specific effects of items at those levels.

The ethnic composition of the samples did not have an effect on the magnitude of the gender difference in mathematics performance. However, age was a marginally significant predictor of effect size (p = .0516), with older samples yielding relatively larger gender differences favoring males.

Discussion

We proposed to answer six questions with these meta-analyses. We take up each question in turn.

First, what is the magnitude of the gender difference in mathematics performance, based on contemporary studies? Taking Study 1 and Study 2 together, the answer appears to be that there is no longer a gender difference in mathematics performance. For Study 1, d values averaged +0.05 based on data from 1,286,350 persons. For Study 2, d values averaged +0.07 based on data from 1,309,587 persons. These results are consistent with a recent analysis of U.S. data from state assessments of youth in grades 2 through 11, which found that girls had reached parity with boys in math performance (Hyde, Lindberg, Linn, Ellis, & Williams, 2008).

Second, does the direction or magnitude of the gender difference vary as a function of the depth of knowledge tapped by the test? By itself, depth of knowledge was not a significant predictor of differences in effect sizes in study 1. However, study 2 indicated that there may be a modest effect of depth of knowledge on gender differences in mathematics performance, with those containing a greater proportion of items at levels 3 or 4 favoring males. These results are inconsistent with a recent analysis of NAEP data, examining gender differences for items categorized as difficult by NAEP, and as at Level 3 or 4 Depth of Knowledge; at 12^th grade, the average d = +0.07, or a negligible difference (Hyde et al., 2008). However, an examination of depth of knowledge and age simultaneously in Study 1 (Table 3) indicates a male advantage (d = 0.16) in Level 3 or 4 problems in high school, a finding that is consistent with the earlier meta-analysis by Hyde and colleagues (1990). This finding, however, is based on only 3 studies, so it should be interpreted with caution. Very few studies used items requiring this greater depth of knowledge, yet it is precisely the skill that is required for high-level STEM careers.

Third, developmentally, at what ages do gender differences appear or disappear? Consistent with previous meta-analyses, are gender differences larger in high school than in elementary or middle school? The data sets reviewed in Study 2 showed a marginally significant increase in effect sizes as age increased (β = +.24). This is consistent with the results of Study 1, which found gender differences close to 0 for elementary and middle school students, and small effects favoring males for high-school and college students (ds= +0.23 and +0.18, respectively). These results, too, are inconsistent with the Hyde et al. (2008) analysis of data from state assessments, which showed no gender difference in performance at any grade level through grade 11. Again, though, it is important to consider age and depth of knowledge required by the test simultaneously.

Overall, we conclude that a small gender difference favoring males in complex problem solving is still present in high school. Multiple factors may account for this gender gap. As noted earlier, girls are less likely to take physics than boys are, and complex problem solving is taught in physics classes, perhaps even more than in math classes. Gender differences in patterns of interest may play a role (Su, Rounds, & Armstrong, 2009), although these patterns, too, are shaped by culture. Moreover, even in very recent studies, parents and teachers give higher ability estimates to boys than to girls (Lindberg, Hyde, & Hirsch, 2008), and the effects of parents' and teachers' expectations on children's estimates of their own ability and their course choices are well documented (Eccles, 1994; Jacobs, Davis-Kean, Bleeker, Eccles, & Malanchuk, 2005).

Fourth, are there variations across U.S. ethnic groups, or across nations, in the direction or magnitude of the gender difference? In regard to ethnicity, Table 2 shows that, for Study 1, a small gender difference was found favoring males among Whites, d = +0.13, but for all ethnic minorities combined, d = −.05. This result is similar to the one found in the 1990 meta-analysis (Hyde et al., 1990a), which found a small gender difference favoring males among Whites, but no difference for ethnic minority groups. Table 2 also shows variation in effect sizes according to nationality or region of the world. The largest gender difference favoring males was found in studies from Africa, d = +0.21, but even this difference is small. The largest difference favoring females was found in Central/South America and Mexico, d = −0.06. These variations in the magnitude and direction of the gender difference in math performance are consistent with those found in analyses of international data sets such as PISA and TIMSS. These other studies have, in addition, found that values of d for nations correlate significantly with measures of gender inequality for those nations (e.g., Else-Quest et al., 2010; Guiso et al., 2008; Penner, 2008).

Fifth, has the magnitude of gender differences in mathematics performance declined from 1990 to 2007? Study 1 found no relation between year of publication and effect sizes, indicating no discernible trend over time toward smaller gender differences. This may be because of the fact that, even in 1990, gender differences were already small (Hyde et al., 1990a), leaving little room for further decline.

Sixth, do males display greater variance in scores and, if so, by how much? The overall variance ratio in Study 1 was 1.07. That is, males displayed a somewhat larger variance, but the VR was not far from 1.0 or equal variances. In Study 2, the average variance ratio was 1.09, again not far from 1.0. In addition, the NELS:88 data (Table 3) show several VR's that are < 1.0, indicating that greater male variability is not ubiquitous. Variance ratios less than 1.0 have also been found in some national and international data sets (Hyde et al, 2008; Hyde & Mertz, 2009).

Overall, to put these findings in a broader context, gender can be conceptualized as one of many predictors of mathematics performance. Other factors include socioeconomic status (SES), parents' education, and the quality of schooling. Melhuish and colleagues (2008) compared the effect sizes of 9 predictors of children's mathematics performance at age 10: birth weight, gender, SES, mother's education, father's education, family income, quality of the home learning environment, preschool effectiveness, and elementary school effectiveness. The striking finding was that gender was the weakest of these 9 predictors, i.e., it had the smallest effect size. Mother's education, quality of the home learning environment, and elementary school effectiveness were far stronger predictors. Our findings are consistent with those of Melhuish and colleagues; gender is not a strong predictor of mathematics performance.

Implications

Overall, the results of these two studies provide strong evidence of gender similarities in mathematics performance. The heterogeneity of the findings suggests that there are moderator variables that might clarify the pattern of effect sizes. Detecting consistent moderators of gender differences would be strengthened by measures that tap the full range of mathematical reasoning, including items that require sustained reasoning about complex problems. The existence and magnitude of gender differences in mathematics performance varies as a function of many factors, including nation, ethnicity, and age.

These findings have several policy implications. First, these findings call into question current trends toward single-sex math classrooms. Advocates of single-sex education base their argument in part on the assumption that girls lag behind boys in mathematics performance and need to be in a protected, all-girls environment to be able to learn math (e.g., Streitmatter, 1999). The data, however, show that girls are performing as well as boys in mathematics, based on 242 separate studies (Study 1) and 4 large, well-sampled national U. S. data sets (Study 2). The great majority of these girls and boys did their learning in coeducational classrooms. Thus, the argument that girls' mathematics performance suffers in gender-integrated classrooms simply is not supported by the data. If we wish to improve students' mathematics performance, we would do better to focus not on gender, but on factors that have larger effects, such as the quality and implementation of the curriculum (Tarr et al., 2008) as well as the quality of the elementary school and the quality of the home learning environment (Melhuish et al., 2008).

Second, the dearth of Level 3 or 4 items in assessments has a serious consequence. Given the importance of mathematics tests for school evaluation under the No Child Left Behind legislation, it is common for teachers to teach to the test (Au, 2007). If the test fails to emphasize the skills that citizens need, American students are disadvantaged. In addition, without evidence concerning student progress on these important forms of mathematical reasoning, teachers, administrators, and policy makers cannot determine which curriculum materials or teaching strategies contribute to mathematical proficiency. Finally, tests that fail to emphasize complex problem solving or sustained reasoning communicate an inaccurate picture of mathematics to students.

These findings also have implications for dispelling stereotypes. Overall, it is clear that, in the U.S. and some other nations, girls have reached parity with boys in mathematics performance. It is crucial that this information be made widely known, to counteract stereotypes about female math inferiority held by gatekeepers such as parents and teachers, and by students themselves.