Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes

Abstract

There has been perennial interest in personal qualities other than cognitive ability that determine success, including self-control, grit, growth mindset, and many others. Attempts to measure such qualities for the purposes of educational policy and practice, however, are more recent. In this article, we identify serious challenges to doing so. We first address confusion over terminology, including the descriptor “non-cognitive.” We conclude that debate over the optimal name for this broad category of personal qualities obscures substantial agreement about the specific attributes worth measuring. Next, we discuss advantages and limitations of different measures. In particular, we compare self-report questionnaires, teacher-report questionnaires, and performance tasks, using self-control as an illustrative case study to make the general point that each approach is imperfect in its own way. Finally, we discuss how each measure’s imperfections can affect its suitability for program evaluation, accountability, individual diagnosis, and practice improvement. For example, we do not believe any available measure is suitable for between-school accountability judgments. In addition to urging caution among policymakers and practitioners, we highlight medium-term innovations that may make measures of these personal qualities more suitable for educational purposes.

Measurement matters. While reason and imagination also advance knowledge (Kuhn, 1961), only measurement makes it possible to observe patterns and to experiment—to put our guesses about what is and is not true to the test (Kelvin,1883). From a practical standpoint, intentionally changing something is dramatically easier when you can quantify with precision how much or how little of it there is (Drucker, 1974).

In recent years, scholars, practitioners, and the lay public have grown increasingly interested in measuring and changing attributes other than cognitive ability (Heckman & Kautz, 2013; Levin, 2013; Naemi, Burrus, Kyllonen, & Roberts, 2012; Stecher & Hamilton, 2014; Tough, 2013; Willingham, 1985). These so-called “non-cognitive” qualities are diverse and collectively facilitate goal-directed effort (e.g., grit, self-control, growth mindset), healthy social relationships (e.g., gratitude, emotional intelligence, social belonging), and sound judgment and decision making (e.g., curiosity, open-mindedness). Longitudinal research has confirmed such qualities powerfully predict academic, economic, social, psychological, and physical well-being (Almlund, Duckworth, Heckman, & Kautz, 2011; Borghans, Duckworth, Heckman, & ter Weel, 2008; Farrington et al., 2012; Jackson, Connolly, Garrison, Levin, & Connolly, 2015; Moffitt et al., 2011; Naemi et al., 2012; Yeager & Walton, 2011).

We share this more expansive view of student competence and well-being, but we also believe that enthusiasm for these factors should be tempered with appreciation for the many limitations of currently available measures. In this essay, our claim is not that everything that counts can be counted, nor that everything that can be counted counts. Rather, we argue that the field urgently requires much greater clarity about how well, at present, we are able to count some of the things that count.

A Rose by any Other Name: Naming and Defining the Category

Reliable and predictive performance tasks to assess academic aptitude (i.e., the capacity to acquire new academic skills and knowledge) and academic achievement (i.e., previously acquired skills and knowledge) have been available for well over a century (Roberts, Markham, Matthews, & Zeidner, 2005). The influence of such measures on contemporary educational policy and practice is hard to overstate.

Yet parallel measures for human attributes other than cognitive ability have not followed suit. Notably, pioneers in the measurement of cognitive ability shared the intuition that these other qualities were crucial to success both in and out of the classroom. For instance, the creators of the first valid IQ test wrote that success in school “admits of other things than intelligence; to succeed in his studies, one must have qualities which depend especially on attention, will, and character” (Binet & Simon, 1916, p. 254). The author of the widely used Weschler tests of cognitive ability likewise observed that “in addition to intellective there are also definite non-intellective factors which determine intelligent behavior” (Weschler, 1943, p. 103). Our guess is that the present asymmetry represents more of an engineering problem than a difference in importance: attributes other than cognitive ability are just as consequential but may be harder to measure (Stecher & Hamilton, 2014).

Of the descriptor “non-cognitive,” Easton (2013) has pointed out, “Everybody hates this term but everyone knows roughly what you mean when you use it…” Where did the term originate? Messick (1979) explains: “Once the term cognitive is appropriated to refer to intellective abilities and subject-matter achievement in conventional school areas…the term noncognitive comes to the fore by default to describe everything else” (p. 282). The term is problematic. Arguably too broad to be useful¹, this terminology also seems to imply that there are features of human behavior that are devoid of cognition. On the contrary, every facet of psychological functioning, from perception to personality, is inherently “cognitive” insofar as processing of information is involved. For example, self-control, a canonical “non-cognitive” attribute, depends crucially on how temptations are represented in the mind. Cognitive strategies that recast temptations in less alluring terms (e.g., thinking about a marshmallow as a fluffy white cloud instead of a sticky, sweet treat) dramatically improve our ability to resist them (Fujita, 2011; Mischel et al., 2011). And, exercising self-control also relies on executive function, a suite of top-down cognitive processes including working memory (Blair & Raver, 2015; Diamond, 2013). Hence, from a psychological perspective, the term is simply inaccurate.

Given such obvious deficiencies, several alternatives have emerged. Without exception, these terms have both proponents and critics. For example, some prefer—while others, with equal fervor, detest—the terms character (Berkowitz, 2012; Damon, 2010; Peterson & Seligman, 2004; Tough, 2013), character skills (Heckman & Kautz, 2014), or virtue (Kristjansson, 2013; for a review of moral character education, see Lapsley & Yeager, 2012). To speak of character or virtue is, obviously, to speak of admirable and beneficial qualities. This usefully ties contemporary efforts toward the cultivation of such positive qualities to venerated thinkers of the past, from Plato and Aristotle to Benjamin Franklin and Horace Mann to Martin Luther King Jr., who in 1947 declared, “Intelligence plus character—that is the goal of true education.”

Many educators, however, prefer terminology without moral connotations. Some have adopted the term social and emotional learning (SEL) competencies, a phrase which highlights the relevance of emotions and social relationships to any complete view of child development (Durlak et al., 2015; Elias, 1997; Weissberg & Cascarino, 2013). SEL terminology has grown increasingly popular in education, and a search on google n-gram shows that mention of the phrase “social and emotional learning” has increased 19-fold in published books since its introduction in 1994 (Merrell & Gueldner, 2010). The SEL moniker may, however, inadvertently suggest a distinction from academic priorities, even though the data show that children perform better in school when SEL competencies are developed (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011).

Psychologists who study individual differences among children might alternatively suggest the terms personality, dispositions, and temperament. But such “trait” terminology may incorrectly suggest that these attributes cannot be changed by people’s experiences, and the connotation of immutability is at odds with both empirical evidence (Caspi, Roberts, & Shiner, 2005; Roberts & DelVecchio, 2000; Roberts, Walton, & Viechtbauer, 2006) and pedagogical aims (Tough, 2011). Indeed, widespread interest in personal qualities is fueled in large part by the assumption that students can learn, practice, and improve them².

Next, the terms twenty-first century skills, twenty-first century competencies, and new basic skills have made their timely appearance (Murnane & Levy, 1996; Pellegrino & Hilton, 2012; Soland, Hamilton, & Stecher, 2013). Likewise, some authors have used the terms soft skills (Heckman & Kautz, 2012). Unlike “trait” terminology, “skill” terminology usefully connotes malleability. However, referring to skills may implicitly exclude beliefs (e.g., growth mindset), values (e.g., prosocial motivation), and other relational attitudes (e.g., trust). The narrowness of “skill” terminology is obvious when considering attributes like gratitude, generosity, and honesty. Yes, these behaviors can be practiced and improved, but an authentic desire to be grateful, generous, and/or honest is an essential aspect of these dispositions. As far as the descriptor “twenty-first century” or “new” is concerned, it seems fair to question whether attributes like self-control and gratitude—of central concern to every major philosophical and religious tradition since ancient times—are of special relevance to modernity. Indeed, these may be more timeless than timely.

Finally, all of these terms—virtues, traits, competencies, or skills—have the disadvantage of implying that they are consistently demonstrated across all possible life situations. But they are not (Fleeson & Noftle, 2008; Mischel, 1968; Ross & Nisbett, 1991; Ross, Lepper, & Ward, 2010; Wagerman & Funder, 2009). For instance, self-control is undermined when people are laboring under the burden of a negative stereotype (Inzlicht & Kang, 2010) or when authority figures are perceived as unreliable (Kidd, Palmeri, & Aslin, 2013; Mischel, 1961). Learners are grittier when they have been asked to reflect on their purpose in life (Yeager et al., 2014), and organizations can create a fixed mindset climate that undermines employee motivation independently of employees’ own prior mindset beliefs (Murphy & Dweck, 2009).

We believe that all of the above terms refer to the same conceptual space, even if connotations (e.g., morality, mutability, or consistency across settings) differ. Crucially, all of the attributes of interest are (a) conceptually independent from cognitive ability, (b) generally accepted as beneficial to the student and to others in society, (c) relatively rank-order stable over time in the absence of exogenous forces (e.g., intentional intervention, life events, changes in social roles), (d) potentially responsive to intervention, and (e) dependent on situational factors for their expression.

From a scientific perspective, agreement about the optimal terminology for the overarching category of interest may be less important than consensus about the specific attributes in question and, in particular, their definition and measurement. Of course, a community of practice (e.g., a school district, a reform movement, a networked improvement community) benefits from consensual terminology (Bryk, Gomez, Grunow, & LeMahieu, 2015). Marching under the same flag, rather than several different ones, would make more obvious the fact that many researchers and educators are working to measure and improve the same student attributes (Bryk, Gomez, Grunow, & LeMahieu, 2015; Langley et al., 2009). However, because each community of practice has its own previously established concerns and priorities, the choice of a motivating umbrella term is perhaps best left to these groups themselves and not to theoretical psychologists.

Our view is pragmatic, not ideological. We suggest that the potentially interminable debate about what to call this category of student attributes draws attention away from the very urgent question of how to measure them. In this review, we refer to personal qualities as shorthand for “positive personal qualities other than cognitive ability that lead to student success” (see Willingham, 1985). Of course, this terminology is provisional because it, too, has flaws. For instance, attitudes and beliefs are not quite satisfyingly described as “qualities” per se. In any case, we expect that communities of research or practice will adopt more descriptive terms as they see fit.

Advantages and Limitations of Common Measures

No measure is perfect. We attempt here an incomplete sketch of the limitations and advantages of three common approaches to measuring this set of personal qualities other than cognitive ability: (a) self-report questionnaires administered to students, (b) questionnaires administered to teachers about their students, and (c) performance tasks. Throughout, we illustrate our points with one important and well-studied personal quality—self-control. Self-control refers to the regulation of attention, emotion, and behavior when enduringly valued goals conflict with more immediately pleasurable temptations. This is an informative example because research on self-control is burgeoning (Carlson, Zelazo, & Faja, 2013). Moreover, longitudinal research supports earlier speculation (Freud, 1920) that self-control is essential to success in just about every arena of life, including academic achievement (de Ridder et al., 2012; Duckworth & Carlson, 2013; Mischel, 2014; Moffitt et al., 2011). Where appropriate, we draw on other examples, such as grit or growth mindset. With these few brushstrokes, summarized in Table 1 and discussed briefly below, we hope to depict the contemporary landscape of measurement as we see it.

Table 1.

Serious Limitations of Questionnaires and Performance Tasks

Serious Limitations of Self-Report and Teacher Report Questionnaires

Misinterpretation by participant: Student or teacher may read or interpret the item in a way that differs from researcher intent

Lack of insight or information: Student or teacher may not be astute or accurate reporters of behaviors or internal states (e.g., emotions, motivation) for a variety of reasons

Insensitivity to short-term changes: Questionnaire scores may not reflect subtle changes over short periods of time

Reference bias: The frame of reference (i.e., implicit standards) used when making judgments may differ across students or teachers

Faking and social desirability bias: Students or teachers may provide answers that are desirable but not accurate

Serious Limitations of Performance Tasks

Misinterpretation by researcher: Researchers may make inaccurate assumptions about underlying reasons for student behavior

Insensitivity to typical behavior: Tasks which optimize motivation to perform well (i.e., elicit maximal performance) may not reflect behavior in everyday situations

Task impurity: Task performance may be influenced by irrelevant competencies (e.g., hand-eye coordination)

Artificial situations: Performance tasks may foist students into situations (e.g., doing academic work with distracting videogames in view) that they might proactively avoid in real life

Practice effects: Scores on sequential administrations may be less accurate (e.g., because of increased familiarity with task or boredom)

Extraneous situational influences: Task performance may be influenced by aspects of environment in which task is performed or by physiological state (e.g., time of day, noise in classroom, hunger, fatigue)

Random error: Scores may be influenced by purely random error (e.g., respondent randomly marking the wrong answer)

Self-Report and Teacher-Report Questionnaires

For good reason, self-report and teacher-report questionnaires are the most common approaches to assessing personal qualities among both researchers and practitioners. Questionnaires are cheap, quick, reliable, and in many cases remarkably predictive of objectively-measured outcomes (Connelly & Ones, 2010; Duckworth, Tsukayama, & May, 2010; Hightower et al., 1986; Jackson, Connolly, Garrison, Levin, & Connolly, 2015; Lucas & Baird, 2006; Roberts, Kuncel, Shiner, Caspi, & Goldberg, 2007). Furthermore, a very large literature in social and cognitive psychology confirms that people are relatively good at using questionnaires to communicate their true opinions—provided that they in fact have answers for the questions asked and feel comfortable reporting accurately on them (see Krosnick & Fabrigar, forthcoming; Krosnick,1999; Schuman & Presser, 1981). Indeed, self-report questionnaires are arguably better suited than any other measure for assessing internal psychological states like feelings of belonging.

Questionnaires typically ask individuals to integrate numerous observations of thoughts, feelings, or behavior over a specified period of time ranging from “at this moment” to “in general.” For example, the Character Growth Card includes a self-control item that reads, “During the past marking period, I came to class prepared” and provides response options ranging from “almost never” to “almost always” (Park, Tsukayama, Patrick & Duckworth, 2015).

The process by which students answer this question or any other self-report item is depicted in Figure 1: (1) students must first read and understand the question, then (2) search their memories for relevant information, (3) integrate whatever information comes to mind into a summary judgment, (4) translate this judgment into one of the offered response options, and finally (5) “edit” their response if motivated to do so (Krosnick & Presser, 2010; Schwarz & Oyserman, 2001; Tourangeau, Rips, & Rasinski, 2000). Teacher-report questionnaires work the same way, except that it is the teacher who integrates observations of the student over time and arrives at a judgment with respect to his or her own standards. Individuals can carry out this kind of self-judgment and other-judgment arithmetic with admirable accuracy and precision (Funder, 2012).

Figure 1.

The process by which students and teachers respond to questionnaire items.

A catalogue of threats to validity can be accomplished by considering potential failures at each stage. For (1) encoding the meaning of the questionnaire items, literacy is an obvious concern, particularly for younger or lower-achieving students. Beyond vocabulary, it cannot be assumed that students always understand the pragmatic meaning—the intended idea—of questionnaire items. For example, self-control questionnaires aim to assess the self-initiated regulation of conflicting impulses (e.g., wanting to get homework done because it is important but, at the same time, wanting to play videogames because they are more fun). Yet students or teachers may instead interpret items as asking about compliance with authority (e.g., following directions simply because an adult asked).

After encoding the question itself, individuals must (2) search their memories and (3) integrate recalled information into a summary judgment. For both students and teachers, mentally integrating across past observations can reduce sensitivity to how behavior is now as compared to before (for a compelling empirical example of errors in these judgments, see Bowman, 2010). Moreover, individuals tend to see themselves and others as holding consistent beliefs and attitudes over time, and this bias toward consistency can affect what information is retrieved as well as how it is evaluated (Mischel, 1968; Nisbett & Wilson, 1977; Podsakoff, MacKenzie, Lee, & Podsakoff, 2003; Ross & Nisbett, 1991; Sabini, Siepmann, & Stein, 2001).

When (3) coming to a summary judgment, teachers have the benefit of a non-egocentric perspective, as well as experience with many other same-aged students over the course of their careers. Nevertheless, end-of-year teacher reports may be colored by first impressions and therefore underestimate change (see moderation results in Raudenbush, 1984). In addition, many teachers only see their students in the classroom setting. Because behavior can vary across contexts (Mischel, 1968; Ross & Nisbett, 1991; Tsukayama, Duckworth, & Kim, 2013), teacher observations may not agree with those made by parents, who may see their child in every context except school. Not surprisingly, correlations between separate teacher ratings of student behavior tend to be higher than between parents and teachers (Achenbach, McConaughy, & Howell, 1987).

Another limitation of teacher-report questionnaires is the potential for teachers to misinterpret student behavior. People’s inferences about why others act the way they do are not always accurate (e.g., Dodge, 1980). For instance, it might seem reasonable to conclude that students who reliably complete all of their homework assignments on time are highly self-controlled. Alternatively, it is possible that some assiduous students are so intrinsically motivated to do schoolwork that they do not find alternatives like texting and videogames at all tempting. If so, it is incorrect to infer that their conscientious academic behavior represents self-control (Duckworth & Steinberg, 2015).

Teachers’ ratings of students’ specific qualities can also be colored by their top-down, global evaluations. For instance, teachers may think “This is a good kid” and then conclude “This student must be good at delaying gratification” (see Abikoff, Courtney, Pelham, & Koplewicz, 1993; Babad, Inbar, & Rosenthal, 1982; Nisbett & Wilson, 1977).

Both students and teachers must use some frame of reference to arrive at their judgments, and the problem of “reference bias” refers to frames of reference that differ systematically across respondents (Heine, Lehman, Peng, & Greenholtz, 2002). For example, the more competent an individual is in a given domain, the more stringently they tend to judge themselves (Kruger & Dunning, 1999). Frames of reference are also influenced by the norms shared within—but not necessarily across—cultures. Thus, reference bias is most readily evidenced in paradoxical inconsistencies in cross-cultural research.

Reference bias is apparent in the PISA (Program for International Student Assessment). Within-country analyses of the PISA show the expected positive association between self-reported conscientiousness and academic performance, but between-country analyses suggest that countries with higher conscientiousness ratings actually perform worse on math and reading tests (Kyllonen & Bertling, 2013). Norms for judging behavior can also vary across schools within the same country: students attending middle schools with higher admissions standards and test scores rate themselves lower in self-control (Goldman, 2006; M. West, personal communication, March 17, 2015). Likewise, KIPP charter school students report spending more time on homework each night than students at matched control schools, and they earn higher standardized achievement test scores—but score no higher on self-report questionnaire items such as “Went to all of your classes prepared” (Tuttle et al., 2013). Dobbie and Fryer (2013) report a similar finding for graduates of the Harlem Children’s Zone charter school. There can even be reference bias among students in different grade levels within the same school. Seniors in one study rated themselves higher in grit than did juniors in the same high school, but the exact opposite pattern was obtained in performance tasks of persistence (Egalite, Mills, & Greene, 2014)³.

In the final stages of responding to questionnaire items, individuals must (4) translate their judgment into one of the offered response options. Reference bias can be a problem here, too, insofar as what one respondent considers “rarely” may be what another respondent considers “often” (Pace & Friedlander, 1982).

Next individuals may (5) amend their response in accordance with any of a number of motivations other than truth-telling. Potentially biasing reports is “acquiescence bias,” the inclination, particularly among younger students, to agree with statements regardless of their actual content (Saris, Revilla, Krosnick, & Shaeffer, 2010; Soto, John, Gosling, & Potter, 2008). Individuals may also not tell the truth simply because they would be embarrassed to admit it (Jones & Sigall, 1971).

Unfortunately, many methods thought to reduce social desirability response bias instead harm validity. For example, preemptive assurances of confidentiality can backfire if they imply that questionnaires will be about sensitive and potentially embarrassing topics (Schwarz & Oyserman, 2001). Moreover, assuring individuals of their anonymity can decrease response validity by removing accountability to be honest (Lelkes, Krosnick, Marx, Judd, & Park, 2012). And attempting to make adolescents feel comfortable reporting undesirable attitudes or behaviors by suggesting that “some people do X … other people do Y” implies to adolescents that the undesirable behavior is carried out by half of their peers, and so it artificially inflates reports of that behavior through conformity processes (Yeager & Krosnick, 2011). Unfortunately, scales purporting to measure individual differences in social desirability bias do not fulfill their promise (Uziel, 2010).

Finally, there is the problem of faking. The extent to which outright faking actually reduces the validity of questionnaires in real-world situations is hotly debated (Ziegler, MacCann, & Roberts, 2011), but the possibility of deliberately inflating or deflating scores on questionnaires is incontrovertible (Sackett, 2011).

Performance tasks

As an alternative to asking a student or teacher to report on behavior, it is possible to observe behavior through performance tasks. A performance task is essentially a situation that has been carefully designed to elicit meaningful differences in behavior of a certain kind. Observing students in the identical contrived situation eliminates the possible confound of variation in the base rates of certain types of situations. For example, it is problematic to use “time spent doing homework” as an indicator of self-control if some students are assigned more homework than others (for example, when comparing students whose teachers or schools differ). But if all students are put in a situation where they have the same opportunity to do academic work, with the same opportunity to allocate their attention to entertaining diversions, then differences in time spent on academic work can be used to index self-control (Galla & Duckworth, 2015).

The most influential performance task in the large literature on self-control is the preschool delay of gratification paradigm, colloquially known as the “marshmallow test” (Mischel, 2014). At the start of the task, children are presented with a variety of treats and asked to pick their favorite. Some choose marshmallows, but others choose Oreos, chocolate candies, pretzels, and so on. Next, the less preferred treats are taken away, and the experimenter makes a smaller pile (e.g., one marshmallow) and a larger pile (e.g., two marshmallows). The experimenter asks the child whether he or she would prefer to have the small pile right away or, alternatively, to wait for the larger pile after the experimenter comes back from doing something unrelated in the hallway. The question is not which choice the child makes—in a national study of approximately one thousand preschool children, nearly all chose the larger, delayed treat (NICHD, 1999)—but rather, once the decision has been made, how long the wait to obtain the larger treat can be endured. Wait time in this standardized situation correlates positively with self-control ratings by parents and caregivers and, over a decade later, predicts higher report card grades and standardized test scores, lower self-reported reckless behavior, and healthier body weight, among other outcomes (Mischel, 2014; Tsukayama et al., 2013).

An advantage of performance tasks is that they do not rely upon the subjective judgments of students or teachers. This feature circumvents reference bias, social desirability bias, acquiescence bias, and faking. Relatedly, by assaying behavior at a moment in time, task measures could be more sensitive than questionnaires to subtle changes in behavior. Not surprisingly, several major studies examining the effects of either self-control interventions or age-related changes in self-control have used performance tasks to do so (Bierman, Nix, Greenberg, Blair, & Domitrovich, 2008; Blair & Raver, 2014; Diamond & Lee, 2011; Raver et al., 2011). Likewise, experiments that attempt to manipulate self-control in the short-term commonly measure change using performance tasks rather than questionnaire measures (e.g. Baumeister, Bratslavsky, Muraven, & Tice, 1998; Hagger, Wood, Stiff, & Chatzisarantis, 2010).

Of course, the advantages of performance tasks must be considered in tandem with their limitations. As is the case with teacher-reported questionnaires, performance tasks require drawing inferences about the internal motivations, emotions, and thoughts of students. For instance, is a child who refrains from playing with toys when instructed to do so exerting autonomous self-control, or does such behavior represent compliance with adult authority (see Aronson & Carlsmith, 1963; Eisenberg et al., 2004; Mischel & Liebert, 1967)? While the task itself is “objective,” interpreting performance is nevertheless “subjective” in the sense that behavior must be interpreted by the researcher.

Relatedly, a one-time performance task may be appropriate for assessing the capacity of a student to perform a certain behavior when maximally motivated to do so but not particularly diagnostic of their everyday behavior in typical life situations (Duckworth, 2009; Sackett, 2007). For many personal qualities (e.g., generosity, kindness, honesty), what matters most is how a child usually behaves, not how they could behave when trying their hardest. In these cases, performance tasks that assess behavior under optimally motivating circumstances miss the mark. Of course, for some personal qualities, assessing capacity may be appropriate, because the construct itself specifies an ability which may or may not be expressed in daily life. For example, performance task measures of emotional intelligence appropriately assess the ability—not the propensity—to perceive, understand, and manage emotions (Brackett & Geher, 2006; Brackett & Mayer, 2003).

Another limitation of performance tasks is their sensitivity to factors irrelevant to the attribute of interest. Miyake and Friedman (2012) call this the “task-impurity problem” (p. 8) and use as an example the Stroop task of executive function. Completing the Stroop task entails looking at the names of colors printed in variously colored ink. When the name of the color is different from the ink in which it is printed (e.g., the word “green” printed in red), then naming the ink color requires executive function. But executive function is not all that is required. Quick and accurate performance also requires color processing, verbal articulation, motivation to pay attention, and so on. Task impurity is thought to be one reason why performance tasks assessing executive function are only weakly correlated with questionnaire measures of self-control (Duckworth & Kern, 2011; Sharma, Markon, & Clark, 2014).

In addition, performance tasks may thrust individuals into situations they might have avoided if left to their own devices (Diener, Larson, & Emmons, 1984). Consider, for example, children faced with the dilemma of one treat now or two treats later in the delay of gratification task. In the test situation, children are not allowed to get up from their chair, occupy themselves with toys or books, or cover the treats with a plate or napkin. Outside of this constrained laboratory situation, any of these tactics might be employed in order to make waiting easier. In fact, more self-controlled adults say they very deliberately avoid temptations in everyday life (Ent, Baumeister, & Tice, 2015; Imhoff, Schmidt, & Gerstenberg, 2013), and as a consequence experience fewer urges to do things they will later regret (Hofmann, Baumeister, Forster, & Vohs, 2012). Thus, performance tasks foist individuals into identical circumstances so that we may assess their ability to navigate such situations, but this comes at the expense of knowing the extent to which they might have the judgment to proactively avoid or modify situations of that kind on their own (Duckworth, Gendler, & Gross, 2014).

To some extent, all performance tasks suffer from practice effects (or test-retest effects), defined broadly as the effect of repeated exposure to the same task. This is even true for the most “pure” measures of general cognitive ability (Hausknecht, Halpert, Di Paolo, & Moriarty Gerrard, 2007; Reeve & Lam, 2005). Familiarity with the procedures of a task can undermine score validity when the task is intended to represent an ambiguous or novel situation (Burgess, 1997; Muller, Kerns, & Konkin, 2012). For example, a first-time experience with the delay of gratification task is not identical to a second-time encounter because expectations of when the experimenter will return to the room are altered (McGuire & Kable, 2013). Experience with a task may also lead to boredom or increased fluency with task procedures irrelevant to the target attribute. At present, almost nothing is known about the feasibility of developing parallel forms of performance tasks assessing personal qualities for repeated administration.

Because performance tasks are standardized situations in which to observe student behavior, they must be administered under carefully controlled conditions. For example, children in the delay of gratification task wait longer if they trust that the experimenter is actually going to deliver on the promise of two marshmallows later (Kidd et al., 2013). Likewise, performance on self-control tasks can suffer when performed in sequence after other effortful tasks (Hagger et al., 2010). Error increases, and precision decreases, the more these situational influences differ across students.

Moreover, situational influences on task performance that vary systematically across groups create bias and potentially misleading conclusions about group differences. For example, a task that assesses diligence on academic work cannot be properly interpreted if administered in a school setting characterized by frequent noisy intrusions (e.g., students walking in and out of the testing room) or especially crowded conditions (e.g., students sitting so closely that they are distracted by each other) (Galla et al., 2014). While questionnaire responses, too, can be influenced by transient situational influences, these effects may be small (see Lucas & Lawless, 2013). In our experience, performance tasks are especially sensitive to differences in administration, such as time of day or presence of ambient distractions.

Even when administered under optimally controlled conditions, performance tasks generate random error—the white noise produced by stochastic influences on behavior. This is especially problematic for performance tasks because most yield a single score (e.g., in the marshmallow test, the number of seconds a child can wait). Questionnaires, in contrast, usually include several different items designed to assess the same latent construct. Using multiple items exploits the principle of aggregation, which states that uncorrelated errors across items cancel out, thus reducing noise and increasing reliability (Clark & Watson, 1995; Rushton, Brainerd, & Pressley, 1983).

An obvious solution is to create a suite of different performance tasks to assess the same construct and then to aggregate results into a composite score. There are only a handful of precedents for this multi-task approach to assessing self-control (Hartshorne & May, 1929; White et al., 1994). The rarity of these studies suggests that the time, expense, and effort entailed in administering a battery of performance tasks to the same children is at present prohibitive in most applied settings. A single performance task could take as many as 20 minutes to administer by a trained experimenter; doing so several times across separate sessions (to avoid fatigue) would likely require hours and hours of testing time.

Valid for What?

As the above exposition demonstrates, perfectly unbiased, unfakeable, and error-free measures are an ideal, not a reality. Instead, researchers and practitioners have at their disposal an array of measures that have distinct advantages and limitations. Accordingly, measurement experts have emphasized that validity is not an inherent feature of a measure itself but rather a characteristic of a measure with respect to a particular end use (AERA, APA, NCME, 1999/2014). Thus, different measures, with their unique advantages and limitations, are differentially valid depending not only on their psychometric properties, but also on their intended application.

One important end use is basic research, and indeed this is the purpose for which most of the measures reviewed here were developed. Given the litany of limitations noted above, it is notable that measures of personal qualities have been shown in basic research studies to be predictive of consequential life outcomes months, years, or decades later (Almlund, Duckworth, Heckman, & Kautz, 2011; Borghans, Duckworth, Heckman, & ter Weel, 2008; Farrington et al., 2012; Moffitt et al., 2011; Naemi, Burrus, Kyllonen, & Roberts, 2012; Roberts et al., 2007). Of course, these research studies have sought to reject the null hypothesis of no relation between personal qualities and later life outcomes, under testing conditions where incentives to distort responses were minimal—a very different project than the applied uses we consider in this final section.

We attempt to explain how the problems with extant measures of these personal qualities can create threats to validity for more applied uses. Four common examples are: program evaluation, accountability, individual diagnosis, or practice improvement. We make specific recommendations regarding each.

Program Evaluation

Many educational programs, including charter schools, in-school programming, and afterschool activities, aim to cultivate self-control, grit, emotional intelligence, and other personal qualities. Yet the above review makes it clear that in many cases self-report questionnaires have serious limitations for such evaluations. Reference bias may even produce results opposite of the truth when evaluating within-person program effects (i.e., a change from pre-test to post-test) or assessing between-program differences (i.e., mean-level differences among schools or programs), as noted above (e.g., Tuttle et al., 2013; West et al., 2015).

Teacher-report measures of personal qualities may be valid when program evaluation is occurring within schools (i.e., comparing classes in the same school, where the standard for a given characteristic is presumably held constant). However, when conducting between-school program evaluation—as is common—it seems likely that self-report and teacher-report questionnaires could be biased by a non-shared frame of reference. For example, teachers at schools with more rigorous standards of behavior may rate their students more stringently.

How then should between-school program evaluations be conducted? Performance tasks may be helpful (Blair & Diamond, 2008; Greenberg, 2010). However, they have the limitations noted above, including but not limited to: dependence on carefully controlled settings for proper administration, the need to tailor the task parameters to the age group, practice effects, and respondent burden. At the same time, performance tasks have perhaps the most important quality for program evaluation: objective, quantifiable behaviors that do not suffer from reference bias over time and across sites.

A potentially solvable engineering problem, in the medium-term, is to create a suite of brief, scalable, age-specific performance tasks designed for group administration. This possibility was foreseen as prohibitively expensive by pioneers in the assessment of personal qualities (Hartshorne & May, 1929), but they could not have predicted the proliferation of computers and wireless technology in schools. Imagine, for example, a set of web-based tasks of academic self-control accompanied by easy-to-follow protocols and checklists for administering them (e.g., Galla et al., 2014). Assuming that practice effects (i.e., test-retest effects) could be minimized, such task batteries might allow for meaningful, apples-to-apples comparisons across schools, among individuals within schools, or within individuals over time.

In sum, scalable batteries of performance tasks to assess various personal qualities would be of great value for program evaluation, especially as schools and districts seek to allocate limited funds wisely.

Accountability

Reference bias in questionnaire measures has a pernicious implication for accountability. Current data and theory suggest schools that promote personal qualities most ably—and raise the standards by which students and teachers at that school make comparative judgments—may show the lowest scores and be punished, while schools that are least effective may receive the highest scores and be rewarded for ineffectiveness (Dobbie & Fryer, 2013; O’Brien, Yeager, Galla, D’Mello, & Duckworth, 2015; West et al., 2015). Even when accountability does not carry high stakes—for instance, when between-school measures are simply used to pair high and low scoring schools to learn from one another—reference bias undermines school improvement: it would lead the practices from the worst schools to be spread to the best schools. Unfortunately, our experience suggests that re-writing items does not seem to eliminate this type of reference bias.

The reference bias problem alone suggests that questionnaires, as they currently exist, should not be used for between-school accountability. Yet accountability adds at least two additional concerns. First, it is not clear that aggregated student reports can reasonably distinguish among schools throughout the majority of the distribution. Indeed, even value-added measures based on standardized achievement test scores (which do not suffer from reference bias) fail to distinguish more effective from less effective teachers outside of the very high or very low ends of the distribution (Goldhaber & Loeb, 2013; Raudenbush & Jean, 2012).

One exception may be assessing personal qualities for the purpose of comparing teachers within schools. For example, the Tripod measure allows for students’ ratings of different teachers in the same school; ratings of one teacher can be “anchored” using ratings of others in the same school, and these anchored ratings have been shown to correlate with differences in value-added measures among teachers within schools (Ferguson, 2012; Ferguson & Danielson, 2014; Kane & Cantrell, 2013). These measures would be excellent for identifying “positive outlier” teachers within schools—for instance, those who reduce achievement gaps and maintain a strong sense of belonging in students—and then encouraging peer teachers in the same schools to learn from their practices. Unfortunately, these measures, like many others, are not very effective when comparing between schools. This mirrors analyses of state test scores, which have found that value-added measures are better at distinguishing among different teachers in the same schools than different teachers in different schools (Raudenbush, 2013).

There is a second, perhaps more problematic, issue with using measures for the sake of accountability: the potential for faking or unfairly manipulating data. Campbell (1976) observed, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (p. 49). Campbell was making a very general point about how good intentions can lead to unintended perverse outcomes. But this may be especially germane to self-report and teacher-report questionnaire measures, because students can easily be taught to mark the “right answer”, and teachers can likewise rate their students more favorably than they really perceive them to be. Even when faking does not occur, accountability pressures for qualities such as growth mindset can lead to superficial parroting of growth mindset ideas, so as to increase self-reports, rather than true, deep, mindset changes in students. We should note that accountability pressures can also affect performance tasks insofar as schools could be incentivized to alter testing situations to optimize student performance (for examples from achievement tests, see commentaries by Hollingworth, Dude, & Shepherd, 2010; Ravitch, 2012).

In sum, we have a simple scientific recommendation regarding the use of currently-available personal quality measures for most forms of accountability: not yet.

Individual Diagnosis

Schools may wish to diagnose students’ personal qualities to use in tracking or remediation decisions. This type of measurement involves decisions about the resources given to a child, and raises two major concerns: reliability at the level of the individual person and context dependency. First, existing questionnaire measures are likely not sufficiently reliable for making diagnoses. Indeed, even more well-established clinical protocols for assessing factors such as depression rely on extensive self-report questionnaires only as a screening tool, and these have only modest sensitivity and specificity for determining whether a person is clinically depressed (Kovacs, 1992). Such measures require clinical follow-ups. Without highly reliable, multi-method, multi-informant measurement batteries whose validity has been demonstrated for diagnosis, it will be difficult for a practitioner to justify the individual diagnosis of children’s personal qualities such as self-control, grit, or growth mindset.

Our second concern is the context-dependency of some measures. Take the example of self-control. Research finds that individuals have a harder time regulating themselves when they are under stereotype threat, but they show greater self-control when they do not feel under threat (Carr & Steele, 2009; Inzlicht & Kang, 2010; Inzlicht, McKay, & Aronson, 2006). Hence, cues that signal the potential to be stereotyped might impair a stigmatized student’s self-control, inappropriately supporting conclusions about the child’s ability rather than about the bias in the context. This may lead to unintended effects such as victim-blaming, rather than systemic reform.

In sum, a handful of self-report or teacher-report questions cannot (currently) diagnose an individual child’s self-control, growth mindset, grit, purpose, etc. And, even if more extensive protocols were available, it would be essential to consider the possibility of situational and group-specific biases.

Practice Improvement

The ultimate goal of much educational research is to systematically improve personal qualities across contexts—that is, to promote the improvement of practice (Bryk, Gomez, Grunow, & LeMahieu, 2015). Here, too, existing measures have important limitations, but they also have great potential.

As has been argued elsewhere (Bryk, Gomez, Grunow, & LeMahieu, 2015; Langley et al., 2009; Yeager & Bryk, 2014), the improvement of practice requires “practical measurement.” Practical measures are not measures for theory development or accountability. Instead, they are administrable in the web of daily instruction, they can be quickly reported on and communicated to practitioners, and they have direct relation to causes of student underperformance that are the explicit target of improvement efforts. They allow people to learn rapidly from practice. This means that the measures should be brief, easily collected, and also contextually appropriate. Practical measures should be sensitive to short-term changes and provide short-term feedback on progress that has or has not been made in improving personal qualities (Bryk, Gomez, Grunow, & LeMahieu, 2015; Yeager & Bryk, 2014).

Existing questionnaires demonstrate very few of these features. First, questionnaires can be quite long (Atkins-Burnett, Fernandez, Akers, Jacobson, & Smither-Wulsin, 2012). For instance, some measures of self-efficacy—a construct that could in theory be assessed with a single item—are 60 items long (Marat, 2005). Next, questionnaire measures are rarely if ever customized for different settings, and therefore their data may not be as relevant to a given teacher working in a given school. That is, issues of language and literacy, cultural norms, or even colloquialisms could compromise a practical measure. A practical measure is only useful if it helps improve practice in a given setting, not if it has reliability and validity on average in different settings. Third, conventional questionnaire measures are often not designed to be sensitive to change over time. For example, a teacher who wants to know whether classroom activities encouraged self-control during the prior week may not learn much by asking students to repeatedly respond to very general questions such as “People say that I have iron self-discipline.” At the same time, it may be possible to write optimized questions—ones that use construct-specific verbal labels to avoid acquiescence bias, use the optimal number of response options, balance bipolar choices, and so on (Gehlbach & Brinkworth, 2011; Krosnick, 1999; Schuman & Presser, 1981; Schwarz & Oyserman, 2001)—and many fewer of them, to solve this latter problem (Yeager & Bryk, 2014).

We believe performance tasks can also support practice improvement. For instance, tasks can document within-person changes over the short-term. To the extent that performance tasks can be embedded online, then they may be used to produce efficient web-based reports, facilitating teachers’ improvement efforts. At the same time, as noted, performance tasks still require that procedures be optimized to reduce systematic and random error. This can make them logistically difficult to embed in the web of daily practice. Still, this may be a solvable engineering problem in the medium-term.

In sum, a promising area for future research is to increase our knowledge of the conditions under which questionnaires and performance tasks can support the continuous improvement of educational practice.

Final Recommendations

The major conclusions of this article are summarized in Table 2. We have argued that all measures have limitations as well as advantages. Furthermore, we have observed that the applied uses of assessments are diverse, and design features that make a measurement approach helpful for one use may render it less appropriate for another. As a consequence, it is impossible to hierarchically rank measures from best to worst in any absolute sense. Rather than seek out the “most valid measure,” therefore, we advise practitioners and researchers to seek out the “most valid measure for their intended purpose.” While doing so, policymakers and practitioners in particular should keep in mind that most existing measures were developed for basic scientific research. We urge heightened vigilance regarding the use-specific limitations of any measure, regardless of prior “evidence of validity.”

Table 2.

Summary for Practitioners and Policymakers

Conclusions There is a scientific consensus in the behavioral sciences that success in school and beyond depends critically on many attributes other than cognitive ability. As shown in Table 1, measures created for basic theory development have various advantages and limitations. These limitations often undermine validity for applied uses. Self-report and teacher-report questionnaire measures may potentially produce the opposite finding of the truth if used for between-school or within-school, over time comparisons, as in program evaluation and school accountability. Existing questionnaire and performance task measures are rarely sufficiently reliable to use for diagnosis, and may produce group biases. Both questionnaire and performance tasks may be useful for practice improvement under some circumstances.

Recommendations A consensual umbrella term for this heterogeneous set of competencies may be less important than clarity about the individual constructs. A high priority for research is to improve the suite of performance tasks available for program evaluation and practice improvement. A second priority is to develop novel and innovative measures, capitalizing on advances in theory and technology.

Whenever possible, we recommend using a plurality of measurement approaches. While time and money are never as ample as would be ideal, a multi-method approach to measurement can dramatically increase reliability and validity (Eid & Diener, 2006; Rushton, Brainerd, & Pressley, 1983). As just one example, Duckworth and Seligman (2005) aggregated multiple measures of self-control, including a delay of gratification task, self-report, teacher-report, and parent-report questionnaires, finding that a composite score for self-control in the fall predicted final report card grades better than a standard measure of cognitive ability. We also encourage further innovation in measurement development. An incomplete list of promising approaches includes: opportunistically mining students’ online learning behavior or written communication in real time (e.g., Twitter feeds, Kahn Academy databases) for meaningful patterns of behavior (D’Mello, Duckworth, & Dieterle, 2014; Ireland & Pennebaker, 2010; Kern et al., 2014); the aperture method of administering random subsets of questionnaire items to respondents so as to minimize administration time while maximizing content validity (Revelle, Wilt, & Rosenthal, 2010); recording and later coding 30-second audio snippets during everyday life (Mehl, Vazire, Holleran, & Clark, 2010); presenting hypothetical situations in narrative form and asking students what they would do in that circumstance (Oswald, Schmitt, Kim, Ramsay, & Gillespie, 2004; Ployhart & MacKenzie, 2011); asking students to make observations of their peers (Wagerman & Funder, 2007); indirectly assessing personal qualities through innovative application of factor analysis to conventionally collected data (e.g., GPA, attendance, achievement test scores) (Kautz & Zanoni, 2014; Jackson, 2012); and contacting students throughout the day to assess their momentary actions, thoughts, and feelings (Wong & Csikszentmihalyi, 1991; Zirkel, Garcia, & Murphy, 2015). In general, efforts to advance measurement of personal qualities would greatly benefit from cross-fertilization with similar efforts in personality psychology, industrial and organizational psychology, neuroscience, and economics (Heckman & Kautz, 2013; Pickering & Gray, 1999; Roberts, Jackson, Duckworth, & Von Culin, 2011; Schmidt, 2013; Schmidt & Hunter, 1998).

Relatedly, it has recently been suggested that supplementing questionnaires with anchoring vignettes may help reduce reference bias (King, Murray, Salomon, & Tandon, 2004; Kyllonen & Bertling, 2013). Anchoring vignettes are brief descriptions of hypothetical persons that serve as anchors for calibrating questionnaire responses. Respondents rate each vignette and then their own behavior on the same rating scale. Adjusting scores of self-report questionnaires using anchoring vignettes has been shown to resolve paradoxical findings attributed to reference bias. However, adding vignettes to questionnaires can dramatically increase respondent burden. Moreover, at present it has been impossible to verify the extent to which vignettes fully correct for reference bias (Kyllonen & Bertling, 2013).

Finally, measuring personal qualities, although difficult, is only the first step. Scientific inquiry and organizational improvement begin with data collection, but those data must be used to inform action. Too little is known about the question of how to act on data regarding the personal qualities of students in various classrooms or schools (Bryk, Gomez, Grunow, & LeMahieu, 2015). If a classroom is low in grit, what should one do? If a student is known to have a fixed mindset, how can one intervene without stigmatizing the child (and should one intervene at all)? How can multi-dimensional data on personal qualities be visualized and fed to decision makers more clearly? The wise use of data in educational practice is another topic that will be increasingly important—and likely just as fraught with difficulty—as the collection of that data (Bryk, Gomez, Grunow, & LeMahieu, 2015).

Interest in the “other” side of the report card is not at all new. What is new is the expectation that we can measure, with precision and accuracy, the many positive personal qualities other than cognitive ability that contribute to student well-being and achievement. Quantifying, even imperfectly, the extent to which young people express self-control, gratitude, purpose, growth mindset, collaboration, emotional intelligence, and other beneficial personal qualities, has dramatically advanced scientific understanding of their development, impact on life outcomes, and underlying mechanisms. It is no surprise that policymakers and practitioners have grown increasingly interested in using such measures for diverse purposes other than theory development. Given the advantages, limitations, and medium-term potential of such measures, our hope is that the broader educational community proceeds forward with both alacrity and caution, and with equal parts optimism and humility.