Measuring Education and Skill

Abstract

This article reviews recent developments in measuring education and skill that need to be taken into account in any new initiative to monitor social mobility. Over the past half-century, patterns of educational participation and attainment have become more heterogeneous, a trend that has been accompanied by increases in assessment and testing practices, and the availability of electronic data sources and other administrative records, including official school transcripts that are generally held indefinitely. This article describes the most promising approaches to measuring education and discusses some of the possible challenges for using the information to study social mobility. Measures of educational concepts fall along at least one of several dimensions: credentials earned, qualities of the schools attended, the amount and nature of curricular exposure, and the development and acquisition of skills. Selected data sources, with an emphasis on school transcripts and administrative records, and their possible uses are described.

This article reviews recent developments in measuring education and skill that need to be taken into account in any new initiative to monitor social mobility. Over the past half-century, patterns of educational attainment have changed in ways that are important for our understanding of the role of education both as an outcome and as an antecedent of life course events and trajectories. While high school completion rates have arguably been declining (Warren and Halpern-Manners 2007), there has been a considerable growth in postsecondary enrollments. For example, between 1972 and 1992 the proportion of high school graduates going on to some sort of postsecondary education increased dramatically, from 58 percent to 77 percent (Adelman 2004). However, this increased postsecondary participation has not been matched by an increase in the rate of degree attainment among those who enter college. Indeed, alternative pathways to and through higher education have proliferated, such as dual and concurrent enrollments and online coursework and degree programs.

The combined increases in college attendance rates and the static rates of postsecondary degree completion suggest substantial growth in the population who attends college but fails to achieve any credential for its efforts. Importantly, nondegree completers are disproportionately non-white and from lower socioeconomic status (SES) backgrounds. Among those in the high school graduating class of 1992 who entered postsecondary study, more than 60 percent of white students had completed some kind of degree or certificate eight years later, compared to only about 40 percent of African American and Hispanic students (Adelman et al. 2003). A similar pattern is observed for individuals with lower SES origins, who are also less likely to complete college. In sum, educational experiences are becoming more varied and less well measured with conventional indicators of educational attainment such as degree completion or years of schooling. This creates a challenge for researchers seeking to measure and understand educational processes and their consequences.

Fortunately, the greater heterogeneity in educational experiences has been accompanied by increases in assessment and testing practices and the availability of electronic data sources. Notably, administrative records, such as those held in state educational data systems (which are linked to labor force data systems in some states), as well as records compiled at the federal level (e.g., National Student Clearinghouse) or by companies such as the College Board are becoming more comprehensive and of higher quality. With careful processing, these records can be coupled with survey data to characterize the educational experiences and performance of students in schools throughout the United States. One of the major challenges in measuring aspects of the educational process is to obtain uniform measures that allow for comparison across settings (such as classrooms, schools, and states), time, and an individual's age or developmental stage. Administrative records also come with other challenges, from those associated with data cleaning to state differences in student privacy requirements and willingness to share data with other states or the federal government. Nonetheless, administrative records can fill in valuable longitudinal data for any period that an individual is enrolled in school—a significant advantage.

School transcripts also fill in data for enrollment periods and may be especially useful for studying social mobility; in theory, it may be possible to collect and code these records for both parents and their children. Because transcripts are an official record of coursework and degrees earned, almost all high schools and postsecondary institutions retain transcripts indefinitely. Transcripts contain extensive information about exposure to curriculum (coursework taken), performance in the courses (grades received), and degrees earned that could provide valuable information. For instance, as the occupational demands for analytic and STEM (science, technology, engineering, and mathematics) skills are an increasingly important driver of labor force participation, specific knowledge about the coursework may also become more useful for understanding social mobility. The main information needed to collect transcripts, either secondary or postsecondary, is the school name and location, dates of enrollment, and identifying information about the person (e.g., name, date of birth). Transcript studies have been regularly conducted since the early 1980s and therefore have well-established protocols.

This article describes approaches to measuring education along multiple dimensions: credentials earned, qualities of the schools attended, the amount and nature of curricular exposure, and the development and acquisition of skills. Selected data sources, with an emphasis on school transcripts, and their possible uses are described.

Measuring Education

Increased participation in postsecondary education among U.S. students has been coupled with high stakes debates over opportunity, access, curriculum, and associated outcomes. Although controversy over access to higher education of traditionally excluded population subgroups (e.g., women and racial and ethnic minorities) has been publicly divisive for over a century, at no time in the past has the issue of providing access impacted as large a portion of our population as it does now. With high rates of attrition from postsecondary institutions, particularly among minority youths and first-generation college-goers, the hurdles associated with college completion are central to the debate about social mobility.

The unparalleled flow of students into higher education has changed the patterns of U.S. educational attainment in fundamental ways. First, the female advantage in degree attainment that emerged in the 1980s has continued to grow, such that in 2005 women earned 58 percent of all bachelor's degrees awarded in the United States (Snyder, Dillow, and Hoffman 2007). The advantage is greater among children of fathers who are either absent or high-school educated and weaker among African American and Latinos than among whites (Buchmann and DiPrete 2006). The academic readiness of college-going youths has also changed; postsecondary enrollment growth between 1972 and 1992 was disproportionate among students with lower levels of academic achievement, who are also less likely to complete a degree (Bound, Lovenheim, and Turner 2007). Finally, beginning in the mid-1990s, we entered a period in which colleges and universities severely curtailed their affirmative action programs (Grodsky and Kalogrides 2008), with a reduction in the share of minority students attending elite public and private colleges and universities. All of this has occurred in an environment in which the price of attending college has risen sharply. Family economic resources have become much more important for college attendance decisions in recent years, particularly among those with lower levels of secondary school achievement (Belley and Lochner 2007), contributing to the low completion rates of students from lower SES backgrounds.

In combination with survey data, transcript-based measures of enrollment histories provide valuable information about students’ school enrollments. Additionally, the detailed measures from the enrollment and transcript data provide information about inequality in the kinds of schools students attend, the courses they take, the majors they pursue, and the grades and degrees they earn in college. As mentioned above, because transcripts are held as official records, schools typically hold them indefinitely. Even when schools dissolve, transcript records are often held at an alternative location. There are well-established procedures for collecting and coding both high school and college transcripts. Obtaining older transcripts may be slightly more challenging and slightly more expensive because they may be held in less accessible storage facilities; however, in most cases it is possible to access older transcripts as well.

Credentials: Degree attainment

One of the most commonly used indicators of education is degree attainment. For example, we might record whether an individual graduated from high school and whether he or she received an associate's degree from a sub-baccalaureate program (such as a community college), a bachelor's degree from baccalaureate awarding institution, or an advanced post-baccalaureate degree. It is worth noting that the U.S. Census Bureau changed the emphasis of its measure of education from years of enrollment to degree attainment in 1990. Data on degree attainment is usually self-reported in surveys, although it is possible to obtain selected information about degrees earned from other sources. Specifically, the American Council on Education houses the records of all people who take GED examinations, whether they attempted and passed the exam, and the scores on the exam. It is worth noting that states have different thresholds for what is a passing score. Thus, a student with a given score may pass or fail the GED—and therefore earn a GED or not—depending on the state. For postsecondary institutions, the National Student Clearinghouse (NSC) tracks degrees earned at a large portion of colleges and universities.

Many people obtain certificates and professional licenses by passing professional tests, attending specialized schools and training programs, or a combination of these approaches. Most of these certificates and licenses are granted by states or even local agencies, and there is no official central data source that compiles such information for all occupations. Catalistⁱ is a company that has compiled many licensure records across states and will make these records available for research purposes. In some cases, a license to practice is accompanied by a degree (for example, an MD or JD). However, many certificates and licenses are not necessarily reflected in degree attainment. Additionally, there are data sources about postsecondary institutions’ degree programs, which are described below. Finally, as part of transcript coding, or with carefully crafted survey items, the field of postsecondary degree can be coded according to a system (there are several) developed by NCES. These coding procedures and taxonomies, such as the Classification of Instructional Programs (CIP), classify fields of study and can be linked to the Occupational Information Network (O*NET), a database of standardized information characterizing hundreds of occupations.

Schools

Schools are the most common setting for educational activities, and there are vast differences between schools in their quality, resources, cost, populations served, faculty and other professional staff, students’ achievement, and organization and policies. These factors are crucial for understanding the kinds of opportunities to learn that schools provide as well as the advantages that some students enjoy because of family resources and background. For example, rates of student postsecondary persistence are in part endogenous to college choice. Some evidence suggests that college selectivity or quality matters, with those attending more selective colleges enjoying higher rates of persistence to degree (Alon and Tienda 2005; Small and Winship 2007), shorter times to degree (Bound, Lovenheim, and Turner 2007), and a greater likelihood of enrolling in graduate school and completing a graduate degree (Bowen and Bok 1998; Zhang 2005), all else equal. Fry (2004) argues that differences in the graduation rates of the colleges well-prepared Hispanic and white students attend go a long way toward explaining the disadvantage Hispanics face in degree attainment. Likewise, Small and Winship (2007) find that, while attending a more prestigious institution improves the probability of earning a degree for all students net of secondary school achievement and social background indicators, African American students benefit more than white students. Other research, however, concludes that students are most likely to complete a degree when they are well-matched to their college (e.g., when their entrance exam scores approach the college mean) (Light and Strayer 2000), attributing much of the variation in degree attainment to student self-selection (Manski and Wise 1983). However, recent work attributes part of the blame for college attrition to the costs of attendance {Dynarski, 2005 #3088}. Unpacking these factors through the improved measurement of family and school processes is imperative. Databases that include information about the universe of schools and districts are valuable because they can be linked to individual students’ records (assuming that the school attended is known) to enhance the measurement of the quality of education received and the selectivity of the institution.

K–12 institutions

. The U.S. Department of Education, National Center for Education Statistics (NCES) compiles information about school organization, resources and finances, staff, and student characteristics annually for public schools and districts that serve elementary, middle, and high school students in the Common Core of Data (CCD) and biannually for private schools in the Private School Survey. The CCD data, which are easily available from the NCES website, are reported by administrators and other informants at the school; however, the data vary in quality, even for the same institution across years. There is some evidence that the quality of the data have recently improved because they are more readily available to (and used by) the public and thus there is greater incentive for schools to produce accurate reports. When using school district data, it is important to recognize that states differ considerably in how they organize districts. Some follow county lines (e.g., Florida) while others may have districts with nonoverlapping boundaries that serve different types of school (for example, California schools may have some partially overlapping districts that serve elementary schools and others that serve middle or high schools).

The NCES website also has a database on districts with information from U.S. census data for each school district (the School District Demographics System). A recent project, the School Attendance Boundary Information System (SABINS), mapped attendance boundaries of schools in districts in certain metropolitan areas. For a number of reasons, district and school boundaries can be challenging to align to census boundaries, so these preconstructed databases are useful. Once district boundaries are aligned with census areas, it is possible to characterize economic and demographic conditions (for instance, the percentage of the male population of a certain age that is not employed) of the population within the district boundaries.

The Office of Civil Rights also collects valuable information about public schools and districts with the Civil Rights Data Collection (CRDC), which measures indicators of equality of educational opportunity, such as available programs and services disaggregated by race/ethnicity, sex, limited English proficiency, and disability. Participation in the CRDC is mandatory under the Civil Rights Act of 1964. Unfortunately, most rounds of data collection (which are generally biannual) involve only a sample of districts representing each state; the universe of districts was included in 1976 and 2000, and has recently been collected more frequently than in the past.

Private companies also hold valuable information about school characteristics. The Quality Education Data (QED) is a privately sourced census of K–12 schools in the United States. The QED is collected primarily for use in marketing, for example it is sold to textbook vendors, and is relatively expensive. However, it is comprehensive and is sometimes used as a sampling frame for nationally representative studies of schools (which is often the first stage in drawing a sample of students), and is generally of high quality. Several companies hold interesting information about high school quality related to the students who prepare for college. As part of the administration of the SAT and AP tests, the College Board collects a large amount of information about students in high schools that can serve as an excellent indicator of school quality when aggregated from the student to the school level. As an example, beyond the scores on college entrance and other exams that are administered by the College Board, they record all of the colleges to which the student sends test scores. Similarly, the ACT collects such information about students and schools.

Finally, states now collect rich information about students. Although scholars are working on strategies to compare indicators across states, no systematic data source is currently available for such a purpose. Nonetheless, many states provide information about the student composition of schools and quality as measured by the pass rates of students on the state assessments. It is important to recognize that states’ assessments and thresholds for passing vary considerably across states and even across years within states.

Postsecondary institutions

The main source of information about postsecondary institutions, including colleges, universities, and vocational and technical institutions, is the Integrated Postsecondary Education Data System (IPEDS), collected annually by NCES from all institutions that participate in federal student financial aid programs. Participation of such institutions is mandatory under the Higher Education Act of 1965. Administrators report on enrollments, program completions, graduation rates, faculty and staff, finances, institutional prices, and student financial aid (NCES 2013). NCES has also compiled a comprehensive database of the Barron's college selectivity rankings for the years that most students in their longitudinal studies—National Longitudinal Study of 1972 (NLS-72), High School and Beyond (HS&B), National Education Longitudinal Study of 1988 (NELS: 1988), Educational Longitudinal Study of 2000 (ELS-2000), and Beginning Postsecondary Study—first attended a postsecondary institution (1972, 1982, 1992, 2004, 2008).

The NSC, mentioned above as a source of information about degree attainment, collects information at the student-level about enrollments and degrees earned from participating colleges and universities. Gaining access to these data must be negotiated. Although it is voluntary for postsecondary institutions to work with NSC, NSC receives data from a large (and increasing) share of institutions. NSC links students across institutions, making it possible to empirically derive information about flows of students between institutions (i.e. networks of institutions based on student flow), and other patterns of enrollment such as feeder schools or institutions where students enroll concurrently. It is also possible to characterize institutions on the basis of the share of students who enroll and complete a degree at the institution, or never complete a degree.

The College Board and ACT not only collect information that is valuable for measuring the quality of high schools, but these companies also collect information about applicants to postsecondary institutions that can be used to characterize colleges and universities. Such information can be aggregated from the student to the school level. In addition to indicators about the applicant base, in theory it is possible to use network methodology to compile information about sets of colleges that share applicants. Additionally, there are more readily available indicators of college selectivity that can be compiled from Barron's, Princeton Review, U.S. News, and other such commercial sources of information about colleges and their rankings.

Curricular exposure and mastery

Measuring education through exposure to curriculum is often used by education researchers but is less frequently used by researchers interested in stratification. Because a major function of educational institutions is to award credentials, such as degrees, schools typically do a very good job of keeping accurate records of enrollment in institutions and courses through the maintenance of transcripts. Beginning in the 1980s, a standard taxonomy was developed to make it possible to compare student transcript records (e.g., coursework, grades) across high schools in the United States. A similar taxonomy was developed to code postsecondary transcript records for all U.S. postsecondary institutions. Although these coding strategies have been and continue to be amended over time, there is also an effort to create crosswalks or some way of comparing across systems. At the high school level, the best practices have been developed in conjunction with the National Assessment of Education Progress (NAEP)-High School Transcript Studies (HSTS), produced by NCES. Practices for coding transcripts and institutional enrollments at the postsecondary level are less well developed and standard. However, NCES has a Postsecondary Transcript Education Studies program that offers rich information about historical and current practices in transcript coding. As mentioned above, a major advantage of using transcript-based data is that they systematically fill in information for the years that a student is enrolled in school. What follows is a brief overview of the information that can be gleaned from transcripts and other administrative records to measure the nature of the students’ curricular exposure and mastery of the curriculum.

Enrollment patterns

Students enroll in schools, and within schools they enroll in courses. In the past it was unusual for high school students to enroll in courses at more than one school; however, with the rise in home schooling, online courses, and dual credit and off-campus course options, this pattern is becoming more common. Both the NAEP-HSTS and states’ education agencies are working to measure how often this occurs and the nature of courses taken under these alternative circumstances. At the postsecondary level, concurrent institutional enrollments have been more common for many years. Some of these patterns show up on transcripts, especially when courses taken at one institution are transferred to another institution for credit toward a degree.ⁱⁱ A good source of information about an individual's history of enrollment in educational institutions is still self-report (although see Adelman [2006] about reporting error based on comparisons of self- and transcript reporting).

School enrollments

During the K–12 years, students usually attend one school at a time. Non-normative transfers between school, especially those during an academic year, but also even those that take place in the summer, are useful to note as they have negative implications for students’ achievement. Thus, transitions between schools—when they occur during the life cycle and if they involve a routine transition between schools (e.g., the transition from middle to high school)—are noteworthy because they represent possible inflection points for a students’ susceptibility to academic risk. Such transfers may be evident through school records or would require student reports. They are not necessarily evident on the students’ transcript. Similarly, gaps in attendance and suspensions are not recorded reliably. As a general principle, when the information is needed to document the credentialing process—as it is for measuring the accumulation of course credits and grades—it tends to be accurately recorded on the transcript. Other information is at the discretion of the school, and many schools prefer not to record information that could place the student in a negative light.

At the postsecondary level, students may also follow more (or less) normative pathways of enrollment, with varying levels of risk of degree noncompletion. However, the normative pathways in higher education are not as easy to identify, and there is greater heterogeneity in enrollment patterns, overall. Adelman (2006) developed some indicators for postsecondary enrollment patterns, and these have been adapted for the planned National Longitudinal Study of Youth-1997 (NLSY-97) postsecondary study. One can also construct other indicators from transcripts that indicate enrollment patterns, such as whether the student has been enrolled continuously or has had breaks in enrollment. This type of coding allows researchers to evaluate the coherence of enrollments; a normative or classical pathway to a bachelor's degree would be uninterrupted (except for summers) and begin either in a two-year or four-year college or university, culminating in degree completion at a four-year institution. Students often deviate from this pathway by interrupting enrollment, by transferring from a four-year to a two-year institution, or otherwise moving between institutions, and these students are typically at greater risk of never completing a postsecondary degree. Thus, school enrollment patterns characterize a trajectory or pattern of stability or instability in the pursuit of human capital.

Course enrollments and performance

Course enrollment data gathered from transcripts can also be used to characterize an individual's progression through school. In addition, because transcripts contain information about the curricular content of courses and the students’ performance, rich indicators of the students’ experiences can be constructed from elements of the students’ transcripts. At a basic level, most transcripts contain information about the course, the credits attempted, credits earned, whether the course was completed or if the student withdrew, and the grad earned in the course. Information from the transcripts is used in combination with course catalogs and other information received from each institution to code these elements to be comparable across institutions using NCES-developed systems. Constructing indicators for course enrollments and performance will likely be different for high school and postsecondary transcripts because there is so much more variation in course enrollment patterns at the postsecondary level.

In high school, math and foreign language courses are highly sequenced across years; for example, French II follows French I. To a lesser extent, it is also possible to characterize progression in high school science courses. This type of organization of coursework is a major source of stratification, separating and sorting students according to the curricular demand and rigor of their classes. Consequently, completing advanced high school courses in core subjects such as math is an extremely powerful predictor of who matriculates to college (Adelman 1999; Schneider, Swanson, and Riegle-Crumb 1998). For any subject, even physical education, it is possible to develop indicators of the accumulation of credits attempted and earned across the years (Muller et al. 2007). Beyond subject-specific course taking trajectories, students’ overall grade point averages, failures, and subject-specific grades show academic progress that can be used with other knowledge about the students’ life.

At the postsecondary level, the progression through sequenced courses is less well understood and more heterogeneous, thus it is not possible to construct postsecondary indicators that are parallel to those developed for high school. As mentioned above, in higher education, many students enroll in postsecondary institutions but do not complete a degree. Furthermore, institutions and degree programs vary considerably. Nonetheless, it is valuable to characterize the students’ curricular progression toward a degree. For instance, for each academic term or year, indicators could include the curricular breadth (subject range) of the students’ coursework for each year, the curricular depth (subject specialization), and indicators of academic performance (course grades, course failures, credits attempted, credits earned).

Curricular content

Although school quality and coursework are usually used to characterize high school curricular exposure, researchers are exploring new ways to further understand stratification in academic outcomes through the analysis of course textbook content (Brown et al. 2013). Early results indicate that curricular content is linked to students’ test scores in the subject matter that is tested. For example, more rigorous algebra or geometry curriculum in a course is associated with the students’ content knowledge on the subject test. Because knowledge gained in math courses is used in the more advanced courses, understanding students’ progress through math is crucial for college preparation, admissions to more selective institutions, and college completion.

Cognitive and noncognitive skills

A full description of the measurement of cognitive and noncognitive skills is beyond the scope of this article. Nevertheless, measuring these skills is vital to understanding stratification and social mobility. People with higher levels of cognitive skills get more education, and education contributes to the growth of cognitive skills (Heckman and Vytlacil 2001). At the same time, noncognitive skills contribute to educational performance (Borghans et al. 2011; Cunha and Heckman 2008; Duckworth and Seligman 2005), and educational attainment contributes to the development of noncognitive skills (Bowles and Gintis 2000; Heckman Stixrud, and Urzua 2006). Cognitive and noncognitive skills together constitute the core of the human capital with which young people begin their schooling careers (Heckman, Stixrud, and Urzua et al. 2006; Heckman and Carneiro 2003). They are also the most fundamental and influential products of schooling (Becker 1962; Bowles and Gintis 1976; Heckman, Stixrud, and Urzua 2006).

Cognitive skills can be measured through a host of aptitude tests across the life course. The joint Cognitive Economics Project at the University of Michigan and the University of Southern California is an excellent source of information on adults. In addition to aptitude tests, young people in school are tested frequently as part of states’ testing and accountability programs. Standardized tests such as the SAT and ACT are also taken by college-bound students. It is important to use tests that are appropriate to the individual's age and developmental stage. At the moment, state tests are not comparable across states, however, the tests are generally designed to measure learning growth. Knowing something about the rate that students’ learn in school may be useful for understanding outcomes of education, since they predict labor market outcomes in early adulthood (Rose 2006). The Big Five Personality traits measure noncognitive skills and predict labor market outcomes (Barrick and Mount 1991). And Duckworth's Grit Scale (Duckworth and Quinn 2009; Duckworth and Seligman 2005), which was developed more recently, is also predictive of academic persistence and success.

Conclusion

The foregoing review makes it clear that questionnaires and face-to-face data collection are important strategies for collecting measures of education, especially for gathering information about cognitive and noncognitive skills, institutions attended, and for permissions to collect transcripts, test scores, and other administrative records. Administrative records are useful because they are reliable and in many cases are or can be coded to be comparable across educational settings, for example, across districts, states, or over time.

The limited nature of traditional measures of education, such as years of schooling or degree attainment, means that these measures were most likely never adequate for characterizing education. With the rise in educational attainment and increase in heterogeneity and stratification in education, particularly in higher education, the need for better measurement has only increased. For adults who are parents of adult children, more nuanced indicators of educational attainment and selectivity of school may be a sufficient first step for evaluating intergenerational mobility. Because school extends into early adulthood for the majority of young adults, more detailed indicators of their current or recent academic progress could provide richer indicators of their educational success and attainment.

Biography

Chandra Muller is the Alma Cowden Madden Centennial Professor in Sociology at the University of Texas at Austin. She is principal investigator (in collaboration with co-principal investigators Sandra Black, Eric Grodsky, and John Robert Warren) of studies currently following up the High School and Beyond sophomore and senior cohorts. She led a study, the Adolescent Health and Academic Achievement Study, that added an education component to the National Longitudinal Study of Adolescence (Add Health), and a study that added a postsecondary education component to the National Longitudinal Study of Youth 1997 (NLSY-97).