Mentor Support Provisions Scale: Measure Dimensionality, Measurement Invariance, and Associations with Adolescent School Functioning

Abstract

Mentoring relationships can have important effects on adolescents’ psychosocial and academic outcomes; however, the transactions within mentoring relationships that may account for impact on psychosocial and academic outcomes are not well understood. This study investigated the psychometric properties of the Mentor Support Provisions Scale (MSPS), a tool for assessing the types of support that mentors provide. Exploratory factor analyses and confirmatory factor analyses were used to determine measure dimensionality. Findings indicated acceptable fit with a three-factor structure: Academic Support, Intimacy, and Warmth. The MSPS was found to have scalar invariance; thus, factor loadings and intercepts are the same across student sex and ethnic groups (e.g., White, Hispanic, and Black). In structural equation modeling analyses, the three latent factors predicted academic engagement and reading and math achievement, above baseline scores. Research and practical uses of the MSPS are discussed.

A natural mentor is commonly defined as a nonparental adult with whom a youth shares a close, trusting relationship in which the mentor provides guidance and encouragement (Rhodes, 2002; Rishel, Cottrell, Stanton, Cottrell, & Branstetter, 2010). Natural mentoring relationships are fostered organically, without the involvement of an agency, and mentors commonly include family members, family friends, teachers, coaches, and religious leaders. These relationships are fostered with adolescents regardless of risk status, as opposed to formal programs that utilize selection criteria. Accordingly, they have been found to occur at a greater frequency than formal mentoring relationships (DuBois & Silverthorn, 2005b; MENTOR, 2006; Zimmerman, Bingenheimer, & Behrendt, 2005). Due to their potential for longer duration and more frequent contact, mentoring relationships may be particularly influential in adolescents’ lives.

Research on natural mentoring has examined its role in positive youth adjustment (e.g., depression, anxiety, well-being, optimism, identity development, behavioral delinquency), primarily focusing on populations of at-risk youth (e.g., ethnic minorities, adolescent mothers). This research has yielded inconsistent findings. Rhodes, Ebert, and Fischer (1992) found that youth with natural mentors reported less depression, as well as more optimism, higher expectations for career attainment, and social support, than those without mentors. Similarly, Hurd and Zimmerman (2010a, 2010b) found that the presence of a natural mentor was associated with fewer depressive symptoms for adolescents over a five-year trajectory. Other studies, however, reported no association between natural mentors and adolescents’ internalizing symptoms (Chang, Greenberger, Chen, Heckhausen, & Farruggia, 2010; DuBois & Silverthorn, 2005b; Zimmerman, Bingenheimer, & Notaro, 2002). In examining delinquency, studies have demonstrated that adolescents with natural mentors exhibit less gang affiliation, violent and nonviolent problem behaviors, and risky sexual behaviors (Beier, Rosenfeld, Spitalny, Zansky, & Bontempo, 2000; DuBois & Silverthorn, 2005a, 2005b; Zimmerman et al., 2002). Findings related to substance abuse, however, have proven inconsistent (Beier et al., 2000; DuBois & Silverthorn, 2005a, 2005b; Hurd & Zimmerman, 2010b). Studies have also examined natural mentoring associations with academic outcomes. The literature indicated positive associations with high school performance, educational pursuit and attainment (e.g., high school graduation, college attendance; DuBois & Silverthorn, 2005b; Klaw, Rhodes, & Fitzgerald, 2003), as well as school attitudes (e.g., school attachment, school importance, academic efficacy; Sanchez, Esparza, & Colón, 2008; Zimmerman et al., 2002). Associations with grades, however, have been inconclusive. For instance, Chang et al. (2010) found positive associations between grades and natural mentoring relationships for adolescents, wherein Sanchez et al. (2008) failed to detect any association.

To date, the literature on natural mentoring has focused on the association of structural aspects of the relationships (e.g., mentor role, frequency of contact and longevity of the relationship) with various psychosocial and academic outcomes (DuBois & Silverthorn, 2005a; Rishel et al., 2010; Sanchez et al., 2008). The importance of quality in natural mentoring relationships has also been noted (DuBois & Silverthorn, 2005a; Rhodes et al., 1992). However, there has been a lack of attention given to the transactions within these relationships that make natural mentoring effective, specifically the various provisions or types of support garnered from mentors. The lack of research in this area may be due, in part, to the lack of established measures for assessing the support provisions that occur in mentoring relationships. To advance research on mentoring relationships, the current study evaluated the technical properties of a tool designed to evaluate the nature of support that mentors provide.

Affective and Academic Support

One would anticipate that natural mentors differ in the types of support they provide and that these variations may be differentially linked to school functioning. For example, a teacher serving as a natural mentor may be more apt to provide an adolescent with academic support (e.g., assistance with work, appreciation for behavioral engagement and motivation) and this academic support may contribute to an adolescent’s increased achievement and engagement. Although research has yet to examine such provisions, we anticipated these varying provisions to include both affective support and support for specific performance domains, including school success.

Many studies have examined the effect of students’ sense of relatedness to parents, teachers, and peers on academic achievement and engagement (Furrer & Skinner, 2003; Hamre & Pianta, 2005; Wentzel, 1997, 1998). The impact of the student-teacher relationship, specifically as it relates to pedagogical caring, most closely aligns with our conceptualization of affective and academic support within a natural mentoring relationship (Noddings, 1992).

Several authors have noted that students’ perceptions of teacher support may be especially important during the transition to middle school, when normative levels of teacher support decline (Eccles, Wigfield, Midgley, & Reuman, 1993; Niehaus, Rudasill, & Rakes, 2012). Consistent with this reasoning, Furrer and Skinner (2003) analyzed children’s sense of relatedness to parents, teachers, and peers from third to sixth grade. They found that children who reported higher relatedness demonstrated greater emotional and behavioral engagement in school, and relatedness uniquely contributed above the effects of perceived control. Teacher relatedness was most strongly associated with emotional engagement, and it was a more influential predictor during middle school, as opposed to elementary school. Sakiz, Pape, and Woolfolk-Hoy (2012) also found perceived teacher affective support was positively related to school belonging, academic enjoyment, academic hopelessness, and academic self-efficacy, which were then associated with increased academic engagement in middle school mathematics classrooms. Finally, Wentzel (1998) examined the relation between sixth grade students’ perceived parent, teacher, and peer support and academic motivation. Perceived teacher support was inclusive of affective (e.g., caring) and academic support (e.g., help with learning) and was found to be a positive predictor of school- and class-related interest, as well as prosocial academic behavior, above other support sources. Such results indicate the important role of affective and academic support provisions in teacher-student relations at varying points in childhood and adolescence and are indicative of the effects of these provisions of support in relationships with other nonparental adults.

Provisions of Support in Mentoring Relationships

Beyond understanding those outcomes associated with effective natural mentoring relationships, research must also examine what transpires within these relationships that may explain their effects. To date, the literature has largely focused on the association of structural aspects of the natural mentoring relationships, including the frequency of contact and longevity of the relationship, with various psychosocial and academic outcomes. There has been a lack of attention given to the transactions within these relationships that make natural mentoring effective, specifically the various provisions or types of support garnered from mentors. Natural mentoring research has indicated the importance of affective support (e.g., warmth, nurturance, trust, openness) in a relationship, as these social support provisions relate to increased relationship quality (DuBois & Silverthorn, 2005a; Rhodes et al., 1992). Related literature on teacher-student relationships and formal mentoring relationships suggests that in addition to affective support, academic support may also be influential in school functioning (Cham, Hughes, West, & Im, 2014). Academic support is defined as transmitting the value of education, showing an interest in school, communicating high academic expectations, assisting with school tasks, and providing support for educational pursuits. Research on teacher-student relationships has found that adolescent-reports of high teacher educational expectations predict school belonging (Cham et al., 2014) and academic motivation (Legault, Green-Demers, & Pelletier, 2006), above the effects of warmth in the teacher-student relationship. Natural mentors providing academic support have been found to have similarly positive impact on academic outcomes. Sanchez et al. (2008) analyzed high school Hispanic students and found that among those that indicated they had a mentor, higher educational support (e.g., encouragement, interest, direct assistance, financial support) was associated with higher grades, a greater sense of school belonging, and less absenteeism.

Mentor Support Provisions Scale

The Mentor Support Provisions Scale (MSPS) was developed by the third author with the aim of establishing a measure to examine the transactions that occur within mentoring relationships that may be predictive of positive school functioning. The 24-item adolescent-report questionnaire was designed for use once the adolescent has defined a single mentoring relationship. Fifteen items were drawn from the Network of Relationships Inventory (NRI, Buhrmester & Furman, 1987) and address affective support. The NRI drew on Robert Weiss’s (1974) conceptualization of social needs and social provisions (Furman & Buhrmester, 1985). Respondents rate the extent to which individuals in their social network (e.g., parents, siblings, and best friend) meet each social support provision (affection, reliable alliance, enhancement of worth, intimacy, instrumental help, companionship, and nurturance). The MSPS did not include NRI items assessing companionship or instrumental help, based on the decision that these items were not relevant to mentoring relationships in early adolescence.

Nine items pertaining to academic support were added to the 15 NRI social support items to comprise the 24-item MSPS. These items were created based on extant research on learning support (Bouchey & Harter, 2005; Malecki & Demaray, 2003) and assess social support specific to academic achievement (i.e., encouragement to try hard and to do well in school, assistance with school work, confidence in one’s choices and abilities).

Study Purpose and Hypotheses

The purpose of this study was to examine the psychometric properties of the MSPS. Specifically, analyses investigated the dimensionality, measurement invariance, and criterion-related validity of the measure. We utilized an ethnically diverse sample of adolescent students, originally recruited for having been academically at risk in first grade.

Based on prior research on elementary students’ perceptions of the provision of affective support and level of conflict in relationships with their teachers (Hughes, 2011), we expected the 15 NRI support items to load on 1 or 2 factors and the 9 learning items to load on a separate Academic Support factor. In the Hughes (2011) study, students in grades 3 and 4 reported on teachers’ provision of affective support on the same 15 items in the MSPS plus an additional support item, as well as 6 conflict items (for a total of 22 items). Exploratory factor analyses (EFA) and confirmatory factor analyses (CFA) supported a 3-factor solution, with 10 items loading on a factor named “Warmth,”6 items loading on a factor named “Intimacy,” and 6 items loading on a factor named “Conflict.” Thus, the affective support items loaded on two factors rather than a single support factor. However, with the addition of 9 learning items and removal of the conflict items, different results are certainly possible, if not likely. Therefore, we first used EFA to identify the factor structure of the MSPS, followed by CFA. We hypothesized that distinct support provisions would make non-redundant, positive contributions to the prediction of four academic outcomes: teacher-rated school engagement, student-rated behavioral engagement, and reading and math achievement, above baseline measures of each outcome. No hypotheses regarding the magnitude of these positive effects were proffered.

Method

Participants and Procedures

Participants in the current study were enrolled in a larger longitudinal study (N=784). The present analyses included data from 459 students (53.6% male) who were recruited into the larger study in the 2001–02 or 2002–03 school years, depending on cohort membership, when they were in first grade. Students were enrolled in one of three school districts (one urban and two small city districts) in Texas. Individuals were invited to participate in the larger study on the basis of having scored below the median on a district-administered test of literacy during the spring of kindergarten or fall of first grade. Additional inclusionary criteria were speaking English or Spanish, not receiving special education services other than speech and language services, and not having been previously retained in first grade. Additional detail on the recruitment of the participants in the larger study is reported in Hughes and Kwok (2006).

The 784 students recruited into the original study were re-consented to maintain participation at the completion of the first five years of the study. Parental consent for continued participation was obtained for 569 of the 784 students. Of the 215 who did not re-consent, 18 (8.4%) actively declined, and the remaining 197 did not return the consent form. Of the 569 re-consented students, 4 withdrew from the study by Year 6, when the Mentor Support Provisions Scale was administered. Of the remaining 555 students, 20 could not be located in Year 6, 76 were not administered the MSPS due to school absence on the dates of testing, and 10 had incomplete data on the MSPS, leaving a study sample of 459 students. At Year 6 of the study most students were in Grade 6. At Year 6, the 459 students had a mean age of 12.57 (SD = 0.37). An attrition analysis was conducted to determine if the 459 students in the current study differed from the 325 attrited subjects on a wide range of relevant variables at Year 1, when participants were recruited into the longitudinal study in first grade. According to t-tests and chi-square difference tests that applied the Bonferroni correction, statistically significant differences were not observed on the following variables: age (t (781) = −.132, p =.911), sex (χ²(1) = .484, p =.487), parent education level (rated on a 5 point scale from 1 to 5; χ²(4) =13.914, p =.008), scores on the school district literacy test (t (588) = −2.84, p =.776), reading achievement (t (755) = −1.523, p = .128), IQ (t (765) = −.112, p = .911), ethnicity (χ²(2) = 3.072, p =.215), and economic disadvantage status (based on eligibility for free or reduced lunch; χ²(1) = .013, p =.910). To perform a Bonferroni correction, the critical p value (α=0.05) is divided by the number of comparisons being made (i.e., 9 hypotheses are tested), and the new corrected critical p value .006 is used for the attrition analyses. A statistically significant difference was found for math achievement (t (754) = −3.99, p < 0.01), in which students included in the current study had a higher Broad Math W score (Mean = 464.43; SD=12.42) compared to attrited students (Mean = 460.53; SD=14.23). The ethnic composition of the sample of 459 was 35.9% White, 35.9% Hispanic, 25.3% Black, and 2.9% other (i.e., 1.8% Asian; 0.2% Native American, or Pacific Islander; and 0.9% other). Fourteen students, or 3%, were Spanish-speaking and completed assessment measures in Spanish. At Year 6, the participants were enrolled in 74 schools in 5 school districts. The expansion in the number of schools is due to student mobility across six years.

Assessment

Overview

The Mentor Provisions Scale was administered to students in their sixth year of participation in the study. Due to some students having been retained in grade, in Year 6, 1% of students were in grade 4, 33% in grade 5, and 66% in grade 6. Outcome measures were administered in Year 4 (baseline) and Year 7 (i.e., the year following administration of the Mentor Support Provisions Scale. Student assessments were conducted between October and May in individual sessions at school. Trained graduate and undergraduate students who had demonstrated proficiency in administration conducted all assessments, including administration of the Woodcock Johnson Tests of Achievement, Third Edition (Woodcock, McGrew, & Mather, 2001) and student questionnaires. Trainees received a minimum of 18 hours of classroom instruction and passed a practice examination on each measure prior to administering measures in the school, and their protocols were checked and corrected, as needed, on a weekly basis. Data from teachers were obtained via questionnaires administered between November and May, and teachers were paid $25 for completing each questionnaire. Sex and ethnicity were obtained from school records.

Students who spoke any Spanish were administered the Woodcock–Muñoz Language Survey (Woodcock & Muñoz-Sandoval, 1993) to determine if they were more proficient in Spanish or English. Children more proficient in Spanish were administered all tests in Spanish by bilingual examiners. Once a child demonstrated equal or greater proficiency in English for two consecutive years, they were tested in English. Spanish versions of student questionnaires, including the Mentor Support Provisions Scale, were created. Specifically, the English version of the MSPS was first translated into Spanish by a native Spanish speaker who was a student in a doctoral program in school psychology. Next, a different native Spanish speaker who was also a doctoral student in school psychology back-translated the MSPS into English. The back-translated version was found to be equivalent to the English version and no changes in the Spanish version were required.

Mentor Support Provisions Scale

A semi-structured interview was utilized to identify students with natural mentors (i.e., individuals over the age of 18 and not a parent or guardian of the student) and, thus, eligible to complete the MSPS. Specifically, students were initially asked to identify an adult (under the aforementioned parameters) who was important to them and someone with whom they feel close or on whom they can depend. If a student did not name such a person, the interviewer asked the student to think about the adults in his or her life-including family friends, teachers, relatives, pastors, coaches, and neighbors. If the student named someone, the interviewer asked the person’s age, ethnicity, and relationship to the student, whether the person was employed and, if employed, the person’s occupation.

As previously described, the MSPS is a 24-item questionnaire that assesses students’ perceptions of a current mentoring relationship and the type and level of support provided by their mentor. Nine items relate to the hypothesized provision of Academic Support (e.g., “How much does this person tell you it is important for you to do well in school;” “How much does this person help you with your school work?”). The remaining 15 support items were drawn from the Network of Relationships Inventory (NRI, Buhrmester & Furman, 1987) (e.g., “How much can you count on this person to be there for you?;” “How much does this adult have a strong feeling of affection -love or liking- toward you?”). Items were designed as statements to which students indicated their agreement on five-point Likert-type scales (1 = strongly disagree; 5 = strongly agree). Appendix A contains a complete list of the original MSPS items.

Academic achievement

The Woodcock-Johnson Tests of Achievement, Third Edition (WJ-III ACH; Woodcock et al., 2001) is an individually administered measure of academic achievement for use with individuals of age two to adulthood. For the purposes of the current study, students’ WJ-III ACH Broad Reading age-based standard scores and their WJ-III ACH Broad Math age-based standard scores were used. The WJ-III ACH Broad Reading W Scores were comprised of the Letter-Word Identification, Reading Fluency, and Passage Comprehension subtests. The WJ-III ACH Broad Math W Scores were comprised of the Calculations, Math Fluency, and Math Calculation Skills subtests. For students with Spanish language dominance, the Batería III Woodcock-Muñoz (Batería III; Woodcock, Muñoz-Sandoval, McGrew, Mather, & Schrank, 2004) was administered. The Batería III is the comparable Spanish version of the WJ-III ACH and similarly yields W scores for Broad Reading and Broad Math. WJ-III ACH Broad Reading and Broad Math W scores are computer-generated and internal consistency is not calculated. Extensive research documents the reliability and construct validity of the WJ-III ACH (Woodcock & Johnson, 1989; Woodcock et al., 2001).

Student-rated behavioral engagement

Student-rated classroom behavioral engagement was measured using the Student Engagement Questionnaire. This 18-item measure was based on Skinner, Zimmer-Gemback, and Connell (1998) and the 6-item Behavioral Engagement scale was used in the current study. An example item states, “When I am in class, I work as hard as I can.” Responses are indicated on a 4-point Likert-type scale. Cronbach’s α for the current sample at Year 4 was 0.71 and Year 7 was 0.82. High behavioral engagement has been found to predict student learning, grades, achievement, and grade retention (Skinner, Furrer, Marchand, & Kindermann, 2008).

Teacher-rated behavioral engagement

Teacher-rated classroom behavioral engagement was measured through an 11-item questionnaire. Questionnaire items were adapted from both the teacher and the student ratings of students’ engagement (Skinner et al., 2008). Items used a 4-point Likert-type scale and examined perceptions of classroom engagement, including effort, persistence, concentration and interest. An example item states, “Concentrates on doing work.” Cronbach’s α for the current sample at Year 4 was 0.92 and Year 7 was 0.92. These 11 items demonstrated good factorial validity (Wu, Hughes, & Kwok, 2010). Prior research has indicated that higher behavioral engagement is related to children’s academic expectations, long-term academic achievement and school completion (Connell, Spencer, & Aber, 1994; Skinner et al., 1998).

Data Analyses

The present study utilized EFA and CFA, as well as structural equation modeling (SEM). Analyses were conducted using Mplus (version 7, Muthén & Muthén, 1998–2012) and R packages (version 3.3.1, R Core Team, 2013). In the current study, we used the maximum likelihood (ML) estimator for the factor analyses (i.e., EFA and CFA) and SEM. In practice, most researches treat ordinal variables with five or more categories as continuous. Some studies reported that the use of ML estimator for items with many categories is not likely to result in much practical impact on results (e.g., Babakus, Ferguson, & Jöreskog, 1987; Dolan, 1994; Hutchinson & Olmos, 1998) if these ordinal items are normally distributed. The least squares means and variance adjusted (WLSMV) estimation is recommended if ordinal items are not normally distributed. However, because there is some disagreement in the literature about when the use of ML is appropriate with ordinal data (Beauducel & Herzberg, 2006) we re-ran the data using both the weighted least squares with mean and variance adjusted (WLSMV) estimator. Results were not meaningfully different; thus, only results from the ML model are reported. First, using the MSPS items measured at Year 6, we initially conducted EFA to determine the number of constructs and the factor patterns underlying these items. Second, based on the EFA results, we investigated the dimensionality and structure of the MSPS with CFA and considered correlated-factor, second-order factor, and bi-factor CFA models. Third, we conducted tests of measurement invariance of the MSPS model across students’ sex and ethnicity (White, Hispanic, and Black). For testing measurement invariance of ethnicity, we excluded Asian, Native American, or Pacific Islander, and other due to insufficient number of participants (see Participants and Procedures). Finally, using SEM, we examined the predictive criterion-related validity of the three factors in the MSPS measurement model (CFA) at Year 6 via path analysis, in which each factor directly predicts academic outcomes at Year 7 (e.g., student-rated and teacher-rated behavioral engagement and reading and math achievement). By entering the MSPS simultaneously, we are able to estimate the unique, or non-redundant, contribution of each provision on Year 7 outcomes. In the SEM analyses, we controlled for the effects of three demographic variables (i.e., sex, ethnicity, and economic disadvantage status) on the each of the three MSPS factors as well as the baseline variables on Year 7 outcomes.

Results

Characteristics of Natural Mentors

Natural mentors ranged in age from 18 to 93 (mean = 39.46, SD = 15.77, median = 35), with the majority (52%) of mentors of ages 21 to 40 (under 21 = 11%, 41–60 = 25%, over 60 = 12%). A total of 176 females (82.6%) and 124 males (51.2%) reported having a same-sex natural mentor, with statistically significant sex congruence [χ²(1) = 56.84, p < 0.01]. Similarly, students’ mentors were more likely to belong to their same ethnic group than to another ethnic group. Specifically, the numbers of same-ethnic mentors were 129 for White (95.6%), 69 for Black (73.4%), and 110 for Hispanic (80.3%) youth. Chi-square difference testing indicated these results were statistically significant [χ²(4) = 436.74, p < 0.01] and post-hoc analyses revealed that all pairwise comparisons of three ethnic groups for both mentors and students were statistically significant at the Bonferroni corrected p value of < 0.0056.

The majority of mentoring relationships were fostered with relatives (65%). Familial friends and school-related mentors accounted for a smaller number of relationships (21% and 11%, respectively), and extra-curricular adults (e.g., youth group leader, coach, club leader) rarely served as natural mentors (3%). The most common mentor occupation (37%) fell within the helping professions (e.g., teacher, doctor, nurse, youth director, member of clergy, social worker), though unskilled workers (e.g., farm laborer, menial service worker) were also a prominent occupation at 26%. Skilled workers accounted for 20% of natural mentors. Few mentors had management (13%) or other (5%) occupations.

Exploratory Factor Analysis

An EFA was conducted to determine the number of underlying constructs and factor patterns in the MSPS. A cross validation approach, in which the data were randomly split into two subsamples, was used to determine the MSPS factors. The summary statistics for random subsample 1, random subsample 2, and the combined full sample in the analyses were presented in Appendix B and Appendix C presenting descriptive statistics for the measured items and demographic characteristics (% sex and ethnicity), respectively. The items were screened for non-normality and extreme values. None of the items used in the analyses exhibited levels of skewness (ranging from −1.35 to 0.41) or kurtosis (ranging from −1.22 to 1.39) associated with problematic tests of fit or standard errors (West, Finch, & Curran, 1995). Figure 1 depicts the scree plot (Cattell, 1966) and the parallel analysis (Horn, 1965) of MSPS items of the two random samples and the combined sample to assist in determining the number of MSPS factors (Zwick & Velicer, 1986). The scree plots and parallel analyses using the package “nScree” (non-graphical Catell’s scree test; Raîche, Walls, Magis, Riopel, & Blais, 2013) and the package “paran” (Horn’s parallel analysis of principal components/factors) in R packages (version 3.3.1) showed that the two random samples were identical. The results from the scree plot (see Figure 1) and the parallel analysis (see Figure 1; Appendix D) suggested three factors might underlie the data. Table 1 presents the item correlation matrix with means and standard deviations for each of the MSPS items.

Figure 1.

Scree plot and parallel analysis. The numbers in paranthesis represents the number of factors to extract. The line with triangle represents the eigenvalues for a randomly generated correlation matrix having the same sample size in which there is no relationship between variables in the population.

Table 1.

Descriptive Statistics and Item Correlation Matrix for Scale Items

Item	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
Correlation
2	0.77
3	0.63	0.63
4	0.50	0.59	0.46
5	0.43	0.43	0.36	0.55
6	0.43	0.46	0.41	0.41	0.39
7	0.49	0.52	0.49	0.41	0.34	0.64
8	0.58	0.62	0.56	0.51	0.40	0.51	0.59
9	0.41	0.44	0.37	0.46	0.37	0.42	0.43	0.49
10	0.46	0.46	0.41	0.38	0.32	0.57	0.62	0.57	0.46
11	0.38	0.40	0.31	0.51	0.39	0.45	0.46	0.51	0.42	0.43
12	0.49	0.46	0.43	0.48	0.51	0.52	0.49	0.55	0.42	0.49	0.57
13	0.32	0.38	0.37	0.32	0.24	0.55	0.72	0.49	0.37	0.63	0.46	0.45
14	0.42	0.47	0.38	0.50	0.36	0.54	0.57	0.53	0.45	0.53	0.52	0.53	0.48
15	0.42	0.41	0.35	0.33	0.26	0.58	0.57	0.49	0.37	0.65	0.41	0.47	0.56	0.55
16	0.29	0.32	0.24	0.45	0.39	0.37	0.38	0.39	0.37	0.34	0.64	0.50	0.38	0.39	0.33
17	0.42	0.41	0.39	0.35	0.32	0.53	0.61	0.49	0.38	0.59	0.46	0.52	0.62	0.52	0.57	0.37
18	0.42	0.46	0.41	0.34	0.28	0.57	0.72	0.54	0.42	0.65	0.45	0.50	0.79	0.54	0.57	0.37	0.61
19	0.44	0.44	0.40	0.37	0.32	0.44	0.52	0.59	0.43	0.51	0.38	0.51	0.45	0.57	0.46	0.35	0.44	0.52
20	0.12	0.18	0.13	0.12	0.07	0.23	0.29	0.26	0.23	0.35	0.17	0.21	0.36	0.24	0.33	0.21	0.31	0.37	0.33
21	0.42	0.44	0.43	0.33	0.28	0.55	0.67	0.60	0.44	0.65	0.48	0.48	0.67	0.56	0.66	0.39	0.54	0.69	0.56	0.39
22	0.29	0.31	0.25	0.42	0.37	0.36	0.38	0.38	0.35	0.34	0.65	0.47	0.37	0.39	0.31	0.73	0.37	0.35	0.32	0.19	0.38
23	0.35	0.37	0.33	0.32	0.21	0.50	0.65	0.49	0.39	0.60	0.46	0.43	0.72	0.55	0.55	0.39	0.60	0.67	0.49	0.36	0.64	0.41
24	0.39	0.43	0.33	0.41	0.33	0.45	0.47	0.52	0.41	0.47	0.44	0.48	0.46	0.54	0.49	0.35	0.50	0.52	0.53	0.29	0.52	0.38	0.49
MEAN	4.08	4.10	4.09	3.07	2.74	4.24	4.43	4.20	3.76	4.25	3.00	3.68	4.29	4.14	4.29	2.66	3.97	4.39	4.18	3.87	4.44	2.80	4.07	3.87
SD	1.06	1.06	1.03	1.26	1.40	1.00	0.83	0.92	1.11	0.94	1.24	1.17	0.97	0.98	0.89	1.39	1.12	0.85	0.87	1.33	0.78	1.39	1.09	1.00

Note. A list of the scale items is presented in Appendix A. The correlations among scale items are bivariate correlations based on listwise deletion. SD is standard deviation. We used the combined sample (N=459).

We performed EFA using oblique (i.e., the underlying factors were assumed to be related) geomin factor rotation to facilitate the interpretation of factor loadings (Browne, 2001) with the random subsample 1 (n = 234). The estimator for this type of analysis is a robust weighted least squares estimator using a diagonal weight matrix. We also compared different factor rotation methods such as oblique promax, oblique oblimin, and orthogonal (i.e., the underlying factors were assumed to be unrelated) varimax in the selected three-factor solution. The results from the rotated factor patterns yielded by these methods agreed with the use of the geomin rotation. The geomin rotated factor patterns from one to four factor loading patterns were analyzed. Our determination of the number of factors to retain was based on the general principles that the number of factors in the model should explain the data better than a model with one fewer factor, a model with one more factor does not improve appreciably in fit, and that the solution was interpretable and most theoretically sensible (Fabrigar, Wegener, MacCallum, & Strahan, 1999).

Table 2 outlines the results on the fit indices of the EFA models with one to four factors. Model fit was examined across four fit indices using acceptability guidelines: χ2/df < 2:1 (Tabachnick, Fidell, & Osterlind, 2001); comparative fit index (CFI) ≥ 0.95 (Hu & Bentler, 1999); root mean square error of approximation (RMSEA) ≤ 0.06 (Hu & Bentler, 1999); and standardized root mean square residual (SRMR) ≤ 0.08 (Hu & Bentler, 1999). These four goodness of fit indices and chi-square difference testing (Δχ2) between two competing nested models provided the best fit for the four-factor model. Although for the three-factor model, the χ² difference test versus the four-factor model was significant (Δχ² (21) = 87.96, p < .01), the RMSEA (change = 0.008 ≤ 0.01, Chen, 2007; Cheung & Rensvold, 2002) fit did not increase noticeably from the four-factor model to the three-factor model. Based on interpretability of the factors, the results from the scree plot and parallel analysis, as well as goodness of fit indices and chi-square difference testing (Δχ²) between two competing nested models, suggested consideration of the three-factor solution. Closer inspection revealed similarity among items that loaded on common factors and theoretical consistency with mentorship provisions as well as empirical consistency with prior research on items drawn from the Network of Relationships Inventory (Buhrmester & Furman, 1987). Specifically, when teachers report on their provision of support to students using the Teacher version of the Network of Relationships Inventory (Hughes, Lou, Kwok, & Loyd, 2008), which includes the same 15 support items as in the MSPS plus 6 Conflict items, the Conflict items load on one factor whereas the Support items load on two factors: Intimacy (comprised of the same 3 items as those in the MSPS) and a Warmth factor (Hughes et al., 2008; Spilt, Hughes, Wu, and Kwok, 2012). Furthermore, across ages 6 to 15 years, the Warmth scale demonstrates good criterion-related validity, whereas the Intimacy scale has small and inconsistent relationships with criterion variables (Wu & Hughes, 2014). Thus, on the basis of overall fit and theoretical coherence, the three-factor solution was retained.

Table 2.

Fit Indices for Exploratory Factor Analysis Models with One to Four Factors

No. of factors	χ2a (dfb)	χ2/df (< 2:1)	CFIc (≥ 0.95)	RMSEAd (≤ 0.06)	SRMRe (≤ 0.08)
1	1088.61 (252)	4.32	0.756	0.119	0.079
2	686.95 (229)	3.00	0.866	0.092	0.053
3	426.60 (207)	2.06	0.936	0.067	0.035
4	338.64 (186)	1.82	0.955	0.059	0.030

Note. Recommended cut-offs for the four indices are shown in parentheses. Oblique geomin rotation method was used.

^aChi-square statistical test.

^bDegrees of freedom.

^cComparative fit index.

^dRoot mean square of approximation.

^eStandardized root mean square residual.

Model fit was examined across four fit indices using acceptability guidelines: χ2/df < 2:1 (Tabachnick, Fidell, & Osterlind, 2001); comparative fit index (CFI) ≥ 0.95 (Hu & Bentler, 1999); root mean square error of approximation (RMSEA) ≤ 0.06 (Hu & Bentler, 1999); and standardized root mean square residual (SRMR) ≤ 0.08 (Hu & Bentler, 1999).

Factor 1 was termed Academic Support and contained six items. Items within the factor included, “How much does this person tell you to try hard at school?” and “How much does this person talk about what you learned in school?”. Factor 2 was termed Intimacy and contained the same three items that comprised the Intimacy subscale of the Network of Relationships Inventory (Buhrmester & Furman, 1987). These items included, “How much do you tell this adult everything?” and “How much do you share your secrets and private feelings with this adult?”. Factor 3 was termed Warmth and contained 13 items. Example items include, “How much can you count on this person to be there for you?” and “How much does this person believe in you and care deeply about you?”.

Table 3 shows both factor pattern and factor structure coefficients, following the recommendations of Graham, Guthrie, and Thompson (2003). The pattern and structure coefficients are equal only if factors are perfectly uncorrelated. Thus, it has been suggested that both factor pattern and factor structure coefficients be reported when correlated factors are involved in the analysis (Graham et al., 2003). We identified and deleted the items that had low factor loadings (β=0.11–0.29 for Item 9) on all factors or substantial cross loadings on secondary factors (β=0.32 on Academic Support; β=0.33 for Intimacy Item 12) for the subsequent CFA analyses. Two factor loadings greater than one for the Warmth factor are found in item 13 and 18 with the residual variance of 0.267 and 0.243 for item 13 and 18, respectively. According to Joreskog (1999) in his article “How Large Can a Standardized Coefficient be”, loadings greater than one without negative residual variances can be reported. Additionally, using these 22 items (i.e., two items omitted in the CFA from 24 items) we conducted the exact same EFA procedure (i.e., Scree plot, parallel analysis, and EFA using Mplus) and compared the results with those of the EFA using 24 items. The results between with 22 items and 24 items were virtually identical. In summary, EFA identified three correlated dimensions underlying the MSPS items.

Table 3.

Geomin Rotated Pattern Coefficient and Structure Coefficient of Three-Factor EFA Model

Items	Pattern Coefficient			Structure Coefficient
	Factor 1	Factor 2	Factor 3	Factor 1	Factor 2	Factor 3
	Academic Support	Intimacy	Warmth	Academic Support	Intimacy	Warmth
1	0.90	−0.20	0.00	0.79	0.29	0.53
2	0.93	−0.17	0.00	0.84	0.33	0.57
3	0.88	−0.28	0.01	0.73	0.20	0.49
4	0.62	0.17	0.02	0.73	0.52	0.52
5	0.42	0.32	−0.05	0.55	0.52	0.38
6	0.18	0.00	0.56	0.56	0.35	0.68
7	0.09	−0.09	0.80	0.59	0.32	0.82
8	0.54	0.01	0.28	0.74	0.43	0.66
A*	0.27	0.11	0.29	0.52	0.38	0.52
10	0.01	0.01	0.73	0.51	0.34	0.73
11	0.06	0.54	0.35	0.59	0.73	0.63
B*	0.32	0.33	0.21	0.64	0.59	0.58
13	−0.35	−0.02	1.07	0.38	0.27	0.82
14	0.16	0.09	0.55	0.59	0.42	0.70
15	0.03	0.03	0.65	0.50	0.34	0.69
16	0.01	0.80	0.12	0.52	0.86	0.49
17	0.01	0.07	0.70	0.53	0.39	0.74
18	−0.25	−0.01	1.02	0.46	0.32	0.85
19	0.21	0.03	0.41	0.51	0.33	0.57
20	−0.12	−0.01	0.40	0.14	0.10	0.31
21	−0.12	0.05	0.83	0.48	0.36	0.77
22	−0.02	0.70	0.23	0.52	0.80	0.53
23	−0.21	0.01	0.93	0.43	0.32	0.79
24	0.14	0.09	0.50	0.53	0.39	0.64

Note. For EFA, 24 items were used. Items A* and B* were removed in the CFA because of low loadings and loadings on more than one factor. Bold factor loadings were the highest factor loadings; these items were permitted to load on the corresponding factor in the CFA analysis. A list of the items is presented in the Appendix A. Pattern coefficient is standardized factor loading.

Confirmatory Factor Analysis

A CFA was subsequently performed using random subsample 2 (n=225) to validate the EFA results. Factor loadings for each item were estimated for each of the primary factors of the three-factor EFA structure. The standardized factor loading patterns (β) were consistent with the results of the EFA. As reported, the MSPS items used in the analyses were screened for non-normality and none of the items exhibited levels of non-normality (see Appendix B for descriptive statistics for the measured items). Model fit was again examined, using the same four fit indices and acceptability guidelines as stated above. The three-factor model demonstrated marginally adequate fit (χ² (206) = 496.938, p < 0.01; CFI = 0.914; RMSEA = 0.079; SRMR = 0.067). Modification indices were examined and six paths of correlated residual variance were added sequentially, based on pairs of items that were similarly worded and theoretically related. Following the addition of the six paths of correlated residual variance, the resulting model fit was acceptable (χ² (200) = 377.42, p < 0.01; CFI = 0.948; RMSEA = 0.063; SRMR = 0.053).

For the inspection of the final validation model, we tested three hypothesized CFA models (i.e., three correlated-factor model versus a second-order factor model versus a bi-factor model). The three correlated-factor model is the resulting model with six paths of correlated residual variance. In the second-order factor model, the three first-order factors loaded on a second-order factor. In the bi-factor model, each item loaded on a general factor and a specific factor that corresponded to the highest loading in the EFA solution. The χ² statistic and the global model fit indices of the three hypothesized CFA models were reviewed. The second-order factor model with three first-order factors is just-identified; thus, model fit indices are not meaningful. However, based on the interpretability, the first-order three factor model was considered a better fit to the data than the second-order model. All fit indices showed the three correlated-factor model (CFI= 0.948, RMSEA=0.063, SRMR=0.053) fit the data better than the bi-factor model (CFI= 0.922, RMSEA=0.080, SRMR=0.182). The Satorra-Bentler χ² difference test (Satorra & Bentler, 2001) also showed that the correlated-factor model fit the data significantly differently from bi-factor model (Year 9: Δχ² (14) = 74.389, p < 0.001). Each of three factors was found to have strong internal consistency, ranging from 0.86 to 0.93. Intimacy was moderately correlated with both Academic Support (r = 0.60) and Warmth (r = 0.61), and Warmth was highly correlated with Academic Support (r = 0.82). Table 4 presents the standardized coefficients with standard errors as well as R square as an indicator of item-level reliability. The final model, including standardized factor loadings with the corresponding standard error and correlations among factors, is presented in the MSPS measurement model part of Figure 2.

Table 4.

Standardized Factor Loading and R² of Three-Factor MSPS (Mentor Support Provisions Scale) Model in Confirmatory Factor Analysis

	Academic Support		Intimacy		Warmth		R2
	β	SE	β	SE	β	SE
1	0.76	0.035					0.58
2	0.78	0.033					0.61
3	0.71	0.039					0.50
4	0.62	0.047					0.38
5	0.56	0.051					0.31
6					0.74	0.032	0.55
7					0.84	0.021	0.71
8	0.83	0.029					0.69
10					0.83	0.023	0.69
11			0.90	0.041			0.81
13					0.81	0.025	0.81
14					0.70	0.036
15					0.77	0.029
16			0.71	0.049			0.50
17					0.72	0.034	0.51
18					0.84	0.022	0.71
19					0.73	0.034	0.53
20					0.51	0.051	0.26
21					0.85	0.021	0.72
22			0.72	0.045			0.52
23					0.77	0.029	0.59
24					0.64	0.041	0.41

Note. β is standardized factor loading. SE is standard error of corresponding standardized estimates.

p value for all factor loadings < 0.001

Figure 2.

Structural Equation Modeling (SEM) to examine the predictive criterion-related validity of MSPS (Mentor Support Provisions Scale) factors on Year 7 academic functioning outcomes. MSPS measurement model refers to CFA of MSPS. A list of the items for MSPS is presented in the Appendix A. Item 9 and Item 12 used in EFA were removed in the CFA and SEM analyses.

Measurement Invariance

Measurement invariance was tested across different groups (i.e., male versus female; White versus Hispanic; White versus Black) with the full sample. Following the sequential procedure for examining invariance originally developed by Meredith (1993) and extended by Millsap (2011) and Millsap and Cham (2012), we sequentially tested configural invariance (same factor pattern), metric invariance (same factor pattern + same factor loading), and scalar invariance (same factor pattern + factor loading + latent intercept). The Mplus automated feature of estimating and comparing the fit of configural, metric, and scalar models is used with the command “MODEL = CONFIGURAL METRIC SCALAR”. Model result adequacy was based on the change in RMSEA index and the χ² difference test of the two nested invariance models. The χ² difference test determines if the more restricted model fits the data equally as well as the less restricted model. The RMSEA index is then used in the case of a statistically significant difference between models to examine the significance of the difference (change ≤ 0.015; Chen, 2007). A small change in the RMSEA is supportive of retaining the more restricted invariance model (Chen, 2007).

Invariance across student sex

The RMSEA of the configural invariance model indicated acceptable fit (0.07). The χ² difference test (Δχ²) used to subsequently examine metric invariance versus configural invariance was not statistically significant (Δχ² (19) = 24.95, p = 0.16). The scalar invariance versus metric invariance χ² difference test, however, reached statistical significance (Δχ² (19) = 42.05, p < 0.01). The minimal RMSEA difference of 0.001 between metric invariance model and scalar invariance model indicated scalar invariance. Therefore, the MSPS was found to have the same measurement structure for both males and females.

Invariance across student ethnicity

The RMSEA of the configural invariance model indicated acceptable fit (0.08). The χ² difference test used to subsequently examine metric invariance was not statistically significant (Δχ² (38) = 46.83, p = 0.15). The scalar invariance χ² difference test, however, reached statistical significance (Δχ² (38) = 76.90, p < 0.01). The minimal RMSEA difference of 0.001 between metric invariance model and scalar invariance model indicated scalar invariance. Therefore, the MSPS was found to have the same measurement structure across students who identified as White, Hispanic, and Black.

Predictive Criterion-Related Validity

Table 5 shows the zero-order correlations, means, standard deviations, and missingness of the measures that assess the criterion-related validity of the selected MSPS model. Approximately 8%, 24%, 8% and 8% of students were missing data on student-rated behavioral engagement, teacher-rated behavioral engagement, and Math and Reading achievement scores, respectively; however, analyses revealed no statistically significant differences between students with and without complete data. The models in the study were tested using full information maximum likelihood (FIML) to provide proper adjustment on outcome measures with data missing at random (Enders, 2010).

Table 5.

Descriptive Statistics and Correlations Among Measured Variables in SEM Analyses

	Y7 Outcome				Y4 Baseline variables				Y7 Covariate
	1	2	3	4	5	6	7	8	9	10	11	12
Y7 Outcome
1. S_Behavioral Engagement
2. T_Behavioral Engagement	0.23
3. WJ-III ACH Math	0.07	0.25
4. WJ-III ACH Read	0.03	0.26	0.71
Y4 Baseline variables
5. S_Behavioral Engagement	0.30	0.12	0.05	−0.04
6. T_Behavioral Engagement	0.11	0.35	0.41	0.31	0.20
7. WJ-III ACH Math	−0.01	0.18	0.81	0.61	−0.04	0.31
8. WJ-III ACH Reading	0.04	0.20	0.63	0.85	−0.02	0.25	0.59
Y7 Covariate
9. Sex	−0.16	−0.24	−0.04	−0.13	−0.14	−0.22	0.00	−0.06
10. Black	0.18	−0.13	−0.35	−0.37	0.19	−0.24	−0.31	−0.38	−0.02
11. Hispanic	−0.03	−0.02	0.00	−0.02	−0.06	0.12	−0.03	0.09	−0.02	−0.45
12. Economic Disadvantage	0.04	−0.26	−0.33	−0.37	0.04	−0.13	−0.29	−0.28	0.08	0.31	0.26
Descriptive Statistics
Mean	3.18	2.72	516	515	3.50	2.83	496	488	0.54	0.26	0.37	0.62
Standard Deviation	0.52	0.68	11.55	21.28	0.43	0.67	10.50	18.71	0.50	0.44	0.48	0.49
Missing (%)	8	24	8	8	4	21	4	4	0	3	3	18

Note. S_=Student-reported. T_=Teacher-reported. Y7 and Y4 = Year 7 and Year 4 school year when the variable was measured, respectively. Sex was coded as 1 for male and 0 for female. Hispanic is coded as 1 for Hispanic and 0 for White. Black is coded as 1 for Black and 0 for White. WJ-III ACH Read and Math are WJ-III ACH Broad Reading and Broad Math W scores, respectively. The correlations are bivariate correlations based on listwise deletion.

First, the maximum likelihood estimated correlations of the factors with each of the criterion measures are reported in Table 6. Across all bivariate relations, correlations were found to be weak. For student-rated behavioral engagement, the correlations with all three of the MSPS factors were positive whereas for other criterion measures (teacher-rated behavioral engagement, WJ-III ACH Broad Reading and Broad Math scores), the correlations with all three of the MSPS factors were negative.

Table 6.

Maximum Likelihood Estimated Correlations of Measured Variables with Each of the Three Factors of the MSPS (Mentor Support Provisions Scale) Measure in Structural Equation Modeling (SEM) to Examine Predictive Criterion-Related Validity

	Academic Support	Intimacy	Warmth
Y7 Outcome
S_Behavioral Engagement	0.07	0.03	0.06
T_Behavioral Engagement	−0.17	−0.11	−0.05
WJ-III ACH Math	−0.21	−0.18	−0.12
WJ-III ACH Read	−0.27	−0.16	−0.09
Y4 Baseline variables
S_Behavioral Engagement	0.07	0.07	0.06
T_Behavioral Engagement	−0.10	−0.05	−0.04
WJ-III ACH Math	−0.17	−0.12	−0.09
WJ-III ACH Read	−0.18	−0.11	−0.09
Y7 Covariate
Sex	0.02	−0.12	−0.10
Black	0.33	0.24	0.23
Hispanic	0.06	0.05	−0.03
Economic Disadvantage	0.35	0.22	0.12
MSPS factors
Academic Support
Intimacy	0.62
Warmth	0.78	0.66

Note. S_=Student-reported. T_=Teacher-reported. Y7 and Y4 = Year 7 and Year 4 school year when the variable was measured, respectively. MSPS factors are latent scores derived from the suggested measurement model. The correlation for Y7 Covariates and MSPS factors were averaged from the four hypothesized models as these covariates and MSPS factors were used for the four hypothesized models.

Finally, using SEM, we examined the predictive criterion-related validity of the MSPS model. In the hypothesized model, each of the three MSPS factors in the MSPS measurement model (CFA) directly predicted the criterion as an indicator of academic achievement (path analysis). Figure 2 depicts the model for the SEM analyses. Results of the SEM analyses are reported in Table 7. SEM can incorporate CFA and regression analysis (i.e., path analysis) including both measurement and structural components. The criterion measures in the current SEM analyses are Year 7 outcomes: student’s behavioral engagement (student- and teacher-rated) and actual achievement (WJ-III ACH Broad Reading and Broad Math scores). We examined association of MSPS factors together to estimate the unique (distinct) contribution of each context on Year 7 achievement related outcomes. Based on the association between sex, ethnicity, and economic disadvantage status with academic engagement and achievement (National Center for Education Statistics, 2015) we controlled for the effects of the demographic variables on the each of the three MSPS factors as well as baseline variables on Year 7 outcomes. The data had a nested structure with students nested within 64 schools during Year 7. Accordingly, the TYPE=COMPLEX procedures in combination with the CLUSTER function (i.e., school at Year 7) was employed to adjust standard errors of the estimated coefficients as well as to take into account the data dependency (i.e., students nested within schools). Standard errors typically are smaller within schools, as opposed to between schools; thus, this nested structure provided more appropriate tests of statistical significance. With the use of TYPE=COMPLEX, Mplus uses the restricted maximum likelihood (MLR) estimator which provides robust standard errors. In Table 7, we present the standardized coefficients (β) and corresponding standard errors (SE) for the parameters for each MSPS factor on four criterion measures (i.e., outcomes) in the hypothesized simultaneous SEMs.

Table 7.

Standardized Coefficient of Year 6 MSPS (Mentor Support Provisions Scale) Factors on Year 7 Academic Functioning Outcomes in SEM Analyses

Outcomes Effect	Achievement Scores						Behavioral Engagement
	Y7 Reading			Y7 Math			Y7 Student-Rated			Y7 Teacher-Rated
	β	SE	p	β	SE	p	β	SE	p	β	SE	p
MSPS factors
Academic Support	−0.27	0.08	0.001	−0.08	0.10	0.397	0.07	0.06	0.231	−0.30	0.13	0.024
Intimacy	−0.07	0.06	0.205	−0.08	0.04	0.028	−0.04	0.06	0.544	−0.07	0.14	0.631
Warmth	0.24	0.06	<0.001	0.07	0.11	0.553	0.01	0.05	0.875	0.25	0.14	0.074
Baseline measure	0.80	0.02	<0.001	0.78	0.02	<0.001	0.27	0.05	<0.001	0.34	0.06	<0.001
R2	0.72	0.03	<0.001	0.65	0.03	<0.001	0.08	0.02	0.001	0.17	0.04	0.001
Model Fit
CFI	0.942			0.943			0			0.933
RMSEA	0.053			0.052			0.052			0.055
SRMR	0.050			0.051			0.049			0.051

Note. MSPS factors were entered and analyzed simultaneously. Academic support refers students’ perception of academic support from their mentors. Intimacy refers students’ perceived intimacy with mentors, indicating higher score of this factor may indicates developmental immaturity. Warmth refers students’ perception of mentors’ provisions. Based on standardized parameter estimates (β). SE is standard error of corresponding standardized estimates. Reading and Math is Woodcock Johnson Test of Achievement Broad Reading and Broad Math W composite score, respectively.

For Reading achievement, the model fit was adequate; CFA=.942; RMSEA= 0.053; SRMR = 0.050. Students’ perceptions of mentors’ provisions of warmth predicted students’ reading achievement the following year, above baseline levels of achievement (β=0.24, SE = 0.02, p < 0.001). This positive effect of the Warmth factor score indicates that students who reported higher levels of students’ perceptions of mentors’ provisions achieved a higher level of WJ-III ACH Broad Reading scores at Year 7. That is, for a one standard deviation increase in students’ perception of mentor warmth, it is predicted that WJ-III ACH Broad Reading scores at Year 7 will increase by 0.24 standard deviation unit. In contrast, the Academic Support factor score predicted students’ reading achievement the following year, above baseline levels of achievement (β= −0.27, SE = 0.08, p = 0.001). This negative effect indicates that students who reported higher levels of academic support from their mentors achieved a lower level of WJ-III ACH Broad Reading scores at Year 7. For a one standard deviation increase in students’ perception of academic support from their mentors, WJ-III ACH Broad Reading scores at Year 7 would decrease by 0.27 standard deviation unit. Intimacy (β= −0.07, SE = 0.06, p = 0.205) was not statistically associated with students’ achieved reading scores at Year 7.

For Math achievement, the model also showed adequate fit; CFA=.943 RMSEA= 0.052; SRMR = 0.051. Intimacy (β= −0.08, SE = 0.04, p = 0.028) predicted students’ Math achievement the following year, above baseline levels of achievement whereas Academic Support (β = −0.08, SE = 0.10, p = 0.397) and Warmth (β= 0.07, SE = 0.11, p = 0.553) did not. The negative effect of the Intimacy factor score indicates that students who reported higher levels of perceived intimacy with mentors (higher scores on this factor may indicate a developmental immaturity) achieved a lower level of WJ-III ACH Broad Math scores at Year 7. For a one standard deviation increase in students’ perceived intimacy with mentors, it is predicted that WJ-III ACH Broad Math scores at Year 7 will decrease by 0.08 standard deviation unit.

For the student-reported behavioral engagement, the hypothesized model showed adequate fit; CFA=.940 RMSEA= 0.052; SRMR = 0.049. None of the MSPS factors predicted student-reported behavioral engagement at Year 7 (β= 0.07, SE = 0.06, p = 0.231 for Academic Support; β= −0.04, SE = 0.06, p = 0.544 for Intimacy; β= 0.01, SE = 0.05, p = 0.875 for Warmth).

For the teacher-rated classroom behavioral engagement, the hypothesized model also showed adequate fit; CFA=.933 RMSEA= 0.055; SRMR = 0.051. Only Academic Support (β = −0.30, SE = 0.13, p = 0.024) was negatively associated with teacher-rated behavioral engagement at Year 7 whereas Intimacy (β = −0.07, SE = 0.14, p = 0.631) and Warmth (β = 0.25, SE = 0.14, p = 0.074) was not. This negative effect of Academic Support indicates that students who reported higher levels of perceived academic support from their mentors reported a lower level of student-reported behavioral engagement at Year 7. For a one standard deviation increase in students’ perceived academic support from their mentors, it is predicted that teacher-rated behavioral engagement at Year 7 will decrease by 0.30 standard deviation unit.

Discussion

The current study examined the psychometric properties of a measure of adolescent-reported mentor support provisions. Extant literature on natural mentoring has focused on structural aspects of mentoring relationships, such as frequency of contact and relationship longevity, as well as general indications of quality. Research on the interworking of these interpersonal relationships is lacking, owing in part to the lack of measures of mentor support provisions. The Mentor Support Provisions Scale provides insight on the transactions within mentoring relationships that may account for impact on psychosocial and academic outcomes and thus has the potential to improve understanding of natural mentoring relationships. The MSPS was examined for dimensionality of mentor support, measurement invariance, and predictive criterion-related validity through measures of academic engagement and reading and math achievement.

Dimensionality of Mentor Support

Results of the EFA and CFA supported three distinct factors. The six items that aligned with Academic Support specifically assessed a mentor’s engagement in his/her mentee’s school functioning, including valuing effort, having expectations for success, providing assistance, and discussing newly learned topics, as well as the extent to which the mentor provides inspiration for effort. Research on teacher-student relationships, one possible representation of natural mentoring, has indicated the unique contributions of academic support on school functioning as adolescent-reports of teacher educational expectations predict school belonging (Cham et al., 2014) and academic motivation (Legault et al., 2006), above the effects of warmth in the teacher-student relationship. The Intimacy factor contained three items related to sharing private thoughts and feelings with a mentor. These items may indicate a level of dependence on adults that is not typical in early adolescence, a time when youths’ needs for intimacy are increasingly met by best friends (Berndt, 1999).

Finally, the Warmth factor contained 13 items that measured the extent to which the adolescent feels he/she can rely on his/her mentor, feelings of caring and respect for the adolescent, satisfaction with the relationship, and belief in relationship endurance. The development of an emotional bond is considered necessary for mentors to have a positive influence on the youth mentee (Rhodes, 2002). Accordingly, warmth has been widely regarded as the most important component of natural mentoring (Dubois & Silverthorn 2005; Greenberger, Chen, & Beam, 1998; Rhodes, 2002; Rhodes et al., 1992). In studies of teacher-student relationships, Wang and Eccles (2012) found social support from teachers reduced the decline in school compliance, school belonging, and subjective value of learning that occurs during adolescence. The three factors were also found to have strong internal consistency and demonstrated moderate to high inter-correlations. Intimacy was moderately related to both Academic Support and Warmth, whereas Warmth and Academic Support were highly related. Our data indicate these factors represent distinct but related aspects of mentoring relationships, and one would anticipate unique effects on youth outcomes, including school functioning.

Measurement Invariance

The MSPS was examined for measurement invariance across adolescent sex and ethnicity. Measurement invariance is an established procedure to ensure that group differences in means, variances, and correlational relationships of latent constructs can be directly compared. Without conducting such analyses, one cannot ensure that the scale is measuring the same constructs across various groups of individuals. Scalar invariance was found for both sex and ethnicity (White, Hispanic, and Black), which indicated that the three factors, factor loadings, and item latent intercepts were equivalent across groups. These results indicate that the MSPS measures mentor support in the same manner and results have similar meaning across the subgroups evaluated in this study, which suggests that the MSPS may be appropriate for examining sex and ethnic differences in adolescent perceived mentor support.

Predictive Criterion-Related Validity

As expected, students’ perceptions of mentors’ provisions of warmth (i.e., admiration, affection, satisfaction, nurturance, reliable alliance) predicted students’ reading achievement the following year, above baseline levels of achievement. Given the strong stability of scores on the WJ-III ACH Broad Reading Scale (r = 0.85 for baseline and Year 7), the finding that levels of mentor warmth predicted reading scores is notable and suggests that affectively supportive relationships with non-parental adults are an academic asset for youth as they enter adolescence and middle school. However, this conclusion must be considered in light of the lack of statistically significant associations with math achievement.

The significant negative predictive associations for Academic Support with both teacher-rated behavioral engagement and reading achievement were unexpected. One explanation for these findings is that students who are performing poorly in school or lack motivation to try hard at school elicit more academic encouragement from their mentors. Consistent with this explanation are findings from the parent-school involvement literature that more frequent communication with teachers about academics is associated with poorer student academic performance (Hill & Tyson, 2009), presumably because higher levels of parent-teacher communication is prompted by academic problems. This same literature finds that discussing educational aspirations and showing an interest in school activities more consistently predict students’ achievement than do more direct forms of involvement, such as checking homework or attending school meetings (Hill & Tyson, 2009; Hong & Ho, 2005). Taken together, the current study’s findings for warmth and academic support suggest that communicating affection for the youths and confidence in them (e.g., this person believes in you and cares deeply about you; treats you like you are admired and respected) may be more effective mentoring strategies than telling youth to try hard or helping them with homework.

The negative predictive association between Intimacy and Year 7 math scores was also unexpected. As described above, these items assess a type of reliance on adults (sharing secrets and private feelings) that is not normative for early adolescents. Higher scores on this factor may indicate a developmental immaturity that undermines academic achievement.

Limitations

This current study was an initial attempt to create an instrument to measure mentees’ perceptions of mentor support. The findings of this study must be considered in light of several limitations. First, the sample consisted of students previously identified as academically at-risk. Based on research reporting that at-risk students are more strongly affected by relational supports than are their lower-risk peers (Baker, 2006), mentor support provisions may be more strongly related to academic outcomes than would be the case in a sample representative of high and low risk students. Yet, these are also the students who are of higher risk of educational failure and, therefore, of great concern to educators and policy makers. Whereas this sample was diverse with regard to sex and had approximately equal representation of three main ethnic/racial groups, further replication is necessary to generalize these findings to other samples. Such replication efforts should also test for measurement invariance across ethnicity with samples that include sufficient numbers of Asian Americans and individuals of two or more races. This recommendation would also address the need to conduct measurement invariance on a sample other than the one used to test the factor structure of the MSPS, as well as testing alternative measurement models. Specifically, given the correlations among the three factors of the MSPS, it is important to further test its structure using bi-factor and higher-order models (Brunner, Nagy, & Wilhelm, 2012). A greater understanding of the structure of the MSPS dimensions could clarify their general and unique contributions to the prediction of criteria (Chen, West, & Sousa, 2006).

Future research might also re-examine the factor structure of the MSPS. We determined that a three-factor structure best fit the data, but this is a result that awaits replication. Additionally, we did not evaluate whether a second-order factor accounted for the relation among the identified factors, and it is possible that a higher-order factor structure may better represent the data.

In addition, the data analyzed in this study were collected as early as 2005. Since this time, the WJ-III ACH has been updated. The current measure, the Woodcock-Johnson Tests of Achievement, Fourth Edition (Schrank, Mather, & McGrew, 2014), maintains similar subtests comprising the Broad Reading and Broad Math W scores; thus, these study findings on the relation of mentoring support provisions influence on academic achievement remain applicable. Another measure-related limitation includes the limited scope of the provision of Intimacy on the MSPS. Whereas the colloquial conceptualization of intimacy includes reliance, care, and personal disclosure, this construct is more narrowly defined as the sharing of private thoughts and feelings. Future versions of the MSPS might expand the current items to include perceptions of a reliable and trusting alliance with the mentor.

The relatively weak associations that were observed between the mentoring provisions and reading and math achievement may have been due in part to the use of the WJ-III ACH as the only measure of academic achievement. The WJ III ACH subtests measure basic skills such as word reading, reading fluency, and solving math facts-- skills that are critical for school success but may be less influenced by mentoring. Conversely, variables such as school attendance, work completion, and achieving passing grades may be more strongly influenced by an adolescent’s interactions and relationship with a natural mentor. Future research might explore this possibility.

Finally, a limitation in the study included the inability to examine the influence of mentoring relationship duration on outcomes. One would inherently expect the strength of the mentoring relationship and associated provisions of support to change as relationships evolve over time (e.g., increased intimacy with prolonged relationships). Data for the current study did not include relationship duration and thus, this variable could vary significantly across study participants. Additional research is needed to validate the MSPS across time points and determine if mentoring relationship duration is associated with outcomes.

Implications for Research and Practice

This study yields important implications for both research and practice. With regard to research, EFA and CFA demonstrate the distinctiveness of three forms of mentor support. It is important to replicate these results in new samples. Importantly, these forms of mentor support make unique contributions to changes in students’ academic engagement and achievement from mid-elementary school to middle school. Of particular interest is the finding that not all forms of adult social support promote academic achievement, at least among a sample of youth who were academically at-risk at entrance to first grade. Telling students to try harder and to make better grades may be less helpful than assisting students with academic tasks and communicating admiration, confidence, and liking. Future research on the MSPS may add items that address more indirect forms of academic support, such as communicating high expectations for achievement and showing an interest in the youth’s school activities.

Further examination of support provisions may be used in conjunction with structural information on adolescent mentoring relationships (e.g., duration of relationship, frequency of contact, mentor role) to explain the effects of mentoring relationships on adolescent school functioning. Similarly, further research may examine whether different mentor provisions may influence academic outcomes in a synergistic versus an additive fashion. For example, perhaps academic support is more helpful when provided in the context of a mentor relationship high in warmth than one low in warmth. Again borrowing from parenting literature in adolescence (Baumrind, 1991), high parental demandingness (i.e., high standards for behavior and firm enforcement of rules) predicts child social responsibility and achievement orientation only in the context of high parental responsiveness (i.e., warmth, open communication, and respect for the developmental needs of the child). These contexts may be further influenced by characteristics of the mentors. For example, academic support may be differently effective when provided by teachers versus mentors in other roles. Additionally, one might anticipate differentiated effects on the basis of familial or non-familial mentoring relationships, with the assumption that mentors outside of the family system may increase a youth’s social capital.

Within practice, school psychologists are well positioned to utilize these findings to serve students within the context of a Response-to-Intervention framework. As proposed by Dishion and Stormshak (2007), an ecological assessment of the sources of support available to a child or adolescent serves as the foundation for decision making and case conceptualization, identifying both areas of deficit and strength to guide systemic, developmentally-appropriate intervention planning. Beginning with a prevention approach, one might use the MSPS (once validated for use in a general population), in conjunction with other academic and social-emotional screeners, to identify students who are, for any reason, at-risk for poor school functioning.

At a Tier 2 or Tier 3 level, data derived from the MSPS may also prove useful in analyzing identified problems, providing partial explanations. For those students able to identify a natural mentor, the MSPS could provide additional data regarding the nature of mentoring support. Students unable to identify a natural mentor or whose mentoring relationships are low in warmth, could be considered for an intervention aimed at natural mentoring, or effort to enhance an existing relationship. School psychologists, through their partnership with parents, can advocate for the importance of youth having access to natural mentors and explain the unique positive influence of these individuals.

School psychologists also have access to school personnel who may serve as youth mentors (e.g., teachers, coaches, counselors). These individuals may be particularly important, given their position as a non-familial helping professional and their accessibility to students. Research has noted the value of non-familial natural mentors. For instance, DuBois and Silverthorn (2005a) found that within a sample of at-risk youth, those with non-familial mentors were more likely to complete high school than those who identified a familial mentor. Natural mentoring can also be targeted within the structure of existing child interventions, such as Check and Connect (C&C, Christenson, Stout, & Pohl, 2012) or Check In Check Out (CICO, also known as the behavior education plan; Crone, Horner, & Hawken, 2003). School psychologists can support the C&C mentors or CICO coordinators, to ensure they are cultivating affectively warm and academically supportive relationships. The MSPS might be used to monitor these relationships to ensure the youth perceives these provisions.

Appendix A Mentor Support Provisions Scale

Mentor Support Provisions Scale
Items
1	How much does this person tell you it is important for you to do well in school?
2	How much does this person tell you to try hard at school?
3	How much does this person expect you to make good grades at school?
4	How much does this person talk to you about what you learned in school?
5	How much does this person help you with your schoolwork?
6	How much can you count on this person to be there for you?
7	How much does this person believe in you and care deeply about you?
8	How much does this person inspire you to do your best?
9	How much has knowing this person really affected what you do and choices you make?
10*	How satisfied are you with your relationship with this adult?
11*	How much do you tell this adult everything?
12*	How much does this adult help you with things you cannot do by yourself?
13*	How much does this adult like or love you?
14*	How much does this adult treat you like you are admired and respected?
15*	How happy are you with the way things are between you and this adult?
16*	How much do you share your secrets and private feelings with this adult?
17*	How much does this adult take care of you or protect you?
18*	How much does this adult really care about you?
19*	How much does this adult treat you like you are good at many things?
20*	How sure are you that your relationship with this adult will last in spite of fights?
21*	How good is your relationship with this adult?
22*	How much do you talk to this adult about things that you do not want others to know?
23*	How much does this adult have a strong feeling of affection (love or liking) toward you?
24*	How much does this adult like or approve of the things you do?

Note.

^*after the item number indicates the item was extracted from the Network of Relationships Inventory (Buhrmester & Furman, 1987). Students were asked to indicate the degree to which they agreed or disagreed with each of the statements using a five-point Likert-type scale (1 = strongly disagree; 5 = strongly agree). Item 9 and Item 12 were removed in the CFA because of low loadings and loadings on more than one factor.

Appendix B Summary Statistics of Random Sample for Analyses

	Random Subsample 1 (n=234)				Random Subsample 2 (n=225)				Combined Sample (N=459)
Item	Mean	SD	Skewness	Kurtosis	Mean	SD	Skewness	Kurtosis	Mean	SD	Skewness	Kurtosis
1	4.11	1.05	−1.06	0.37	4.06	1.07	−1.03	0.38	4.08	1.06	−1.05	0.36
2	4.10	1.09	−1.09	0.39	4.11	1.04	−1.15	0.80	4.10	1.06	−1.11	0.56
3	4.11	1.05	−1.08	0.39	4.08	1.02	−0.92	0.05	4.09	1.03	−1.00	0.22
4	3.09	1.28	−0.02	−1.07	3.05	1.25	0.11	−1.08	3.07	1.26	0.04	−1.08
5	2.78	1.41	0.24	−1.26	2.70	1.38	0.30	−1.15	2.74	1.40	0.27	−1.21
6	4.27	0.99	−1.22	0.65	4.20	1.01	−0.98	−0.30	4.24	1.00	−1.10	0.14
7	4.47	0.81	−1.43	1.19	4.40	0.85	−1.27	0.65	4.43	0.83	−1.34	0.89
8	4.21	0.90	−1.00	0.57	4.18	0.95	−0.97	0.19	4.20	0.92	−0.98	0.36
9	3.79	1.08	−0.61	−0.32	3.72	1.15	−0.55	−0.70	3.76	1.11	−0.58	−0.53
10	4.22	0.92	−0.99	0.20	4.28	0.96	−1.22	0.69	4.25	0.94	−1.11	0.43
11	3.04	1.22	0.05	−0.96	2.95	1.27	0.27	−1.02	3.00	1.24	0.16	−1.01
12	3.73	1.15	−0.56	−0.67	3.64	1.20	−0.54	−0.71	3.68	1.17	−0.55	−0.68
13	4.28	0.96	−1.16	0.46	4.30	0.98	−1.22	0.41	4.29	0.97	−1.19	0.41
14	4.17	0.95	−1.17	1.18	4.10	1.00	−0.86	−0.14	4.14	0.98	−1.01	0.45
15	4.27	0.89	−1.05	0.36	4.31	0.89	−1.16	0.64	4.29	0.89	−1.10	0.47
16	2.68	1.36	0.39	−1.02	2.64	1.41	0.43	−1.14	2.66	1.39	0.41	−1.09
17	3.96	1.08	−0.72	−0.38	3.97	1.16	−0.81	−0.48	3.97	1.12	−0.77	−0.43
18	4.40	0.82	−1.28	0.93	4.39	0.89	−1.31	0.68	4.39	0.85	−1.30	0.79
19	4.20	0.87	−0.84	0.06	4.15	0.87	−0.71	−0.35	4.18	0.87	−0.77	−0.16
20	3.90	1.34	−0.99	−0.26	3.85	1.33	−0.89	−0.42	3.87	1.33	−0.94	−0.35
21	4.45	0.76	−1.31	1.15	4.42	0.81	−1.39	1.61	4.44	0.78	−1.35	1.39
22	2.84	1.37	0.23	−1.15	2.76	1.42	0.23	−1.28	2.80	1.39	0.23	−1.22
23	4.11	1.06	−1.15	0.67	4.03	1.12	−0.89	−0.34	4.07	1.09	−1.01	0.11
24	3.94	0.99	−0.72	−0.12	3.79	1.02	−0.47	−0.44	3.87	1.00	−0.59	−0.32

Note. For cross validation approach, random subsample 1 (n=234) was used for EFA analyses and random subsample 2 was used for CFA analyses. Combined samples were used for examining for measurement invariance testing and the predictive criterion-related validity of MSPS scale in SEM.

Appendix C. Sex and Ethnicity Summary Statistics of Random Sample for Analyses

	Random Subsample1		Random Subsample 2		Combined Sample
	Count	Percent	Count	Percent	Count	Percent
Sex
Female	113	48	100	44	213	46
Male	121	52	125	56	246	54
Ethnicity
Native American/Pacific Islander	0	0	1	0	1	0
Asian	6	3	2	1	8	2
Black	52	22	64	28	116	25
Hispanic	86	37	79	35	165	36
White	87	37	78	35	165	36
Other	3	1	1	0	4	1

Note. For cross validation approach, random subsample 1 (n=234) was used for EFA analyses and random subsample 2 was used for CFA analyses. Combined samples were used for examining for measurement invariance testing and the predictive criterion-related validity of MSPS scale in SEM.

Appendix D. Output of Parallel Analysis with 24 items using R package

##############################################################
                                      PARALLEL ANALYSIS
##############################################################
Randomization method: generated data
Type of correlations specified for the real data eigenvalues: pearson
Type of correlations specified for the random data eigenvalues: pearson
Extraction Method: Principal Components
Variables = 24
Cases = 23
Ndatasets = 100
Percentile = 95
Compare the Real Data eigenvalues below to the corresponding
random data Mean and Percentile eigenvalues
   Real Data     Root     Mean     Percentile
1     7.869        1     3.557          4.095
2     4.667        2     2.982          3.241
3     3.179        3     2.585          2.810
4     1.391        4     2.252          2.487
5     1.018        5     1.984          2.199
6     0.954        6     1.746          1.940
7     0.721        7     1.545          1.707
8     0.590        8     1.347          1.508
9     0.555        9     1.172          1.338
10     0.470      10     0.990          1.127
11     0.430      11     0.839          0.950
12     0.367      12     0.694          0.793
13     0.321      13     0.589          0.703
14     0.294      14     0.480          0.607
15     0.219      15     0.382          0.464
16     0.207      16     0.286          0.357
17     0.184      17     0.214          0.291
18     0.175      18     0.155          0.222
19     0.115      19     0.101          0.154
20     0.113      20     0.062          0.096
21     0.101      21     0.030          0.057
22     0.058      22     0.009          0.025
23     0.000      23     0.000          0.000
24     0.000      24     0.000          0.000
Parallel Analysis: Number of Factors = 3>