Mentoring Network Questionnaire Support Scales Reliable and Valid with University Faculty

Abstract

The Developmental Network Questionnaire (DNQ) is used in business to self-assess relationships with developers, or people who support one’s career. The Mentoring Network Questionnaire (MNQ) is an online modification of the DNQ and includes two scales that rate developer’s contributions to career or psychosocial help. The psychometrics of these scales for different populations are unreported. This study analyzed the construct validity and reliability of the two scales measuring support provided by developers of university faculty. Mentors and mentees (G=156) from multiple Southwestern and Mountain West universities rated 741 developers on the MNQ’s five-item career- and psychosocial-support scales. Participants responded on a seven-point scale ranging from “never, not at all” to “to the maximum extent possible.” Multilevel confirmatory factor analysis (MCFA) using Mplus and the multi-level reliability coefficient omega assessed construct validity and internal consistency reliability, respectively. Results supported the validity of two latent constructs of career- and psychosocial support, each measured by the established five-item scale: Comparative fit index (CFI)=.93, Tucker-Lewis Index (TLI)=.91, root mean square error of approximation (RMSEA)=.06, standardized root mean square residual (SRMR): W=.09, B=.10. The measurement model was improved when the “removes barriers” item was removed from the career-support scale (CFI=.96, TLI=.95, RMSEA=.05, SRMR: W=.06 B=.09. Factor loadings at both the within- and between-levels were strong and statistically significant. Reliability omegas ranged from .85 to .92. Career and psychosocial support provided to university faculty by developers in their networks may be validly and reliably measured at both the within- and between-levels by a modified four-item career support scale and the original five-item psychosocial support scale from the DNQ and the modified MNQ. Limitations include reduced statistical power due to small sample size and lack of testing at the university level. Future work will assess the responsiveness of these scales to measuring change over time in the amount of support provided.

Introduction & Literature Review

The Developmental Network Questionnaire (DNQ) has been widely used, predominantly in business settings, to assess individuals’ perceptions of their developmental relationships. (Higgins, 2004). The Mentoring Network Questionnaire (MNQ) is an online modification of the DNQ and, like the DNQ, includes two scales to measure opinions about the quality and quantity of career and psychosocial help provided by one’s developers. The five-item Career Support Scale measures individuals’ perceptions of the support they receive for their career development. It assesses various aspects of career support, including mentoring, coaching, sponsorship, and access to opportunities. The five-item Psychosocial Support Scale measures individuals’ perceptions of the emotional and social support they receive from the network in both their professional and personal life. It assesses aspects such as friendship, emotional support, and advice.

Psychometric studies involving the careers of lawyers using six-item versions of the scales demonstrated construct validity based on principal components factor analysis, with each set of items loading on two separate factors, and convergent validity with positive correlations of support with work satisfaction, intentions to remain, and likelihood of organizational retention (Higgins & Thomas, 2001). Later studies of MBA graduates used three-item versions of each scale likewise demonstrated construct validity at three points in time and strong internal consistency reliabilities for both instruments (Cronbach’s alphas .82-.89) and (Cummings & Higgins, 2006). However, the psychometric properties for the current five-item scales, or for populations from diverse organizational and cultural contexts, including individuals from academic settings are unreported. There is a need for more extensive validation research. The purpose of this study was to analyze the construct validity and reliability of the two scales measuring support provided by developers of university faculty. Our research questions are: 1) Is the factor structure (number of factors and factor loadings) for the Career and Psychosocial Support Scales at the developer level similar to or different from the structure at the respondent (mentor/mentee) level? 2) What is the reliability of the scores from the Career and Psychosocial Support Scales at the developer and the respondent levels?

Methods

Sample

As part of a larger study, faculty (G=156; 81 mentors, 75 mentees), completed a cross-sectional survey and identified up to six developers each (N = 741). Developers were described as people who had helped their careers, either professionally or personally, including people from current or prior jobs or school, friends, family members, or broader social arenas such as community settings. Faculty were from multiple Southwest and Mountain West institutions, including the University of New Mexico (UNM) Central and Health Science campuses, Arizona State University, Oklahoma University Health Science Center, and Mountain West Clinical Translational Research Infrastructure Network institutions. The study was approved by the UNM Health Sciences Center Institutional Review Board (HRPO 18–261).

Measures

In addition to demographic data, participants completed the web-based MNQ, a modification of the DNQ (Higgins, 2004) to allow online administration via REDCap^™ (Harris et al., 2009). For each developer identified, respondents completed two scales.

Career Support and Psychosocial Support Scales

The Career Support Scale is a five-item scale developed by Higgins (2004). Respondents rate the extent to which each developer: 1) “Provides you with opportunities that stretch you professionally,” 2) “Creates opportunities for visibility for you in your career,” 3) “Opens doors for you professionally,” 4) “Acts as a sponsor for you,” and 5) “Acts as a buffer for you from situations that could threaten your career achievement.” The five-item Psychosocial Support Scale includes items about how much each developer: 1) “Cares and shares in ways that extend beyond the requirements at work,” 2) “Counsels you on non-work-related issues,” 3) “Offers you support, respect, and encouragement,” 4) “Is a friend of yours,” and 5) “Confirms and affirms your identity and sense of self (Higgins, 2004).” Response options for both scales range from 1 (never, not at all) to 7 (to the maximum extent possible). Item responses were summed to measure either career or psychosocial support, with higher scores indicating more support.

Data Analysis

Multilevel confirmatory factor analysis (MCFA) was conducted using Mplus Version 8.2 (Muthén & Muthén, 2017). MCFA allows analysis of data with a hierarchical structure, where individuals are nested within higher-level units. In this study, because each mentor or mentee respondent identified and rated multiple developers, developers (individuals) were nested within respondents (higher-level unit). MCFA allows the total sample covariance matrix to be separated into pooled within-group (within developers) and between-group (between respondents) covariance matrices to analyze the factor structure. For example, the within-level analysis examined the relationships between the observed career support items (reported by the respondents) and the latent construct of “career support” for each developer separately. Likewise, the between-level analysis examined the relationships between the latent “career support” and “psychosocial support” factors across different respondents.

Parameters were estimated using full information maximum likelihood estimation that assumes missing data is missing completely at random. Factor loadings were freely estimated except that one item in each scale was constrained to have a value of one. Overall goodness of fit for the models was assessed using the X² likelihood ratio statistic, the normed comparative fit index (CFI; Bentler, 1992), Tucker-Lewis index (TLI; Tucker & Lewis, 1973), root mean square error of approximation (RMSEA), the standardized root mean square residual (SRMR), and the Bayesian information criterion (BIC; Schwartz, 1978). Multiple fit indices were used because each has limitations. Acceptable fit was estimated by CFI and TFI values greater than .95, SRMR values close to or less than .08, and RMSEA values close to or less than .06 (Hu & Bentler, 1999). Smaller BIC values indicate better fit.

The multilevel reliability coefficient omega was used to assess internal consistency for each of the two scales (Geldhof et al., 2014). Omega is a measure of the reliability of the latent factors (career support and psychosocial support) at both the within-level (developer level) and between-level (respondent level) in the hierarchical data structure. It represents the proportion of reliable variance in the observed variables that is due to the underlying latent constructs.

Results

Sample

Table 1 displays the demographic characteristics of our faculty mentors and mentees. Mentors were an average 10 years older than mentees, 80% were associate or full professors, and half were on tenure track, as compared to approximately one-third of mentees. Both groups were predominantly female and White, non-Hispanic.

Table 1.

Participant Demographic Characteristics

Characteristic	Mentors (n = 81)	Mentees (n = 75)
	Mean (SD) / %	Mean (SD) / %
Age	50.67 (9.24)	39.97 (7.84)
Male	32.00	35.21
Female	68.00	64.79
Race/Ethnicity
White (non-Hispanic)	69.23	59.46
Hispanic (non-Black)	10.26	12.16
Black (non-Hispanic)	6.41	4.05
Asian	7.69	17.57
Other	6.41	6.76
Faculty Rank
Instructor	2.63	5.88
Lecturer	0.00	2.94
Assistant Professor	17.11	73.53
Associate Professor	50.00	13.24
Professor	28.95	0.00
Other	1.32	4.41
Faculty Track
Tenure	50.00	37.50
Clinician Educator	38.16	38.89
Lecturer	1.32	12.50
Research	2.63	2.78
Other	7.89	8.33

Item Analysis and Descriptive Statistics

Responses for each of five items ranged from 1 (never, not at all) to 7 (to the maximum response possible) for both the Career Support Scale, and Psychosocial Support Scales. Table 2 shows the descriptive statistics for both scales, including intraclass correlation coefficients (ICCs). The ICC for an item provides a measure of the amount of variability between respondents (mentor or mentee) and the degree of clustering of data within respondents. ICCs can range from 0 to 1.0, larger values indicate greater clustering effects within respondents, or greater between-level differences, and the need for MCFA. Lower ICC values suggest that individual-level differences (among developers) account for a larger proportion of the variance. Although there are no established cutoffs for the use of MCFA, values above .10 are often found in published MCFAs (Dedrick & Greenbaum, 2011; Dyer et al., 2005). In this study, the intraclass correlations coefficients for the 10 items on the two scales ranged from .17 to .38, indicating that MCFA was the appropriate statistical technique given the presence of between group variance.

Table 2.

Descriptive Statistics for Items from the Career Support and Psychosocial Support Scales

Scale and Item	n	M	SD	ICC
Career Support Scale
1. Provides you with opportunities that stretch you professionally	727	4.55	1.71	0.17
2. Creates opportunities for visibility for you in your career	741	4.20	1.94	0.19
3. Opens doors for you professionally	738	4.15	1.95	0.23
4. Acts as a sponsor for you	737	4.24	2.07	0.29
5. Acts as a buffer for you from situations that could threaten your career achievement	730	3.86	2.01	0.29
Psychosocial Support Scale
1. Cares and shares in ways that extend beyond the requirements of work	732	4.85	1.92	0.17
2. Counsels you on non-work-related issues	728	4.10	2.20	0.18
3. Offers you support, respect, and encouragement	734	5.74	1.51	0.38
4. Is a friend of yours	736	4.82	1.97	0.21
5. Confirms and affirms your identify and sense of self.	739	5.20	1.80	0.35

Note: ICC = intraclass correlation coefficient (range 0 – 1.0). Response scale ranged from 1 (never, not at all) to 7 (to the maximum extent possible).

When the Pearson correlation matrix was assessed for the 10 items, the five career support items correlated with each other (rs = .42-.78), with much lower correlations with the five psychosocial support items (rs = .09–.37). Likewise, the psychosocial support items correlated with each other (rs = .55-.72). The fifth career support item (“Acts as a buffer…”) correlated the most weakly with the other four career support items (rs = .42-.52) and had some of the stronger correlations with the psychosocial support items (rs = .20-.37). For this reason, a second MCFA model using only four of the career support items was tested.

Multilevel Confirmatory Factor Analysis

MCFA results supported the validity of two latent constructs of career- and psychosocial support at both the within- and between-levels. Table 3 shows the fit indices for the two tested models: Model 1 with the original five items in each scale, and Model 2 with the fifth career support item (“Acts as a buffer…”) removed due to weaker correlations with the other four career support items. Model 2 demonstrated more acceptable fit than Model 1, with CFI and TLI of .96 and .95 compared to .93 and .91, respectively, and RMSEA of .05 compared to .06 in Model 1. The SRMR indices for both models were better for the within level than the between level; however, the SRMR for Model 2 was best (SRMR: W=.06 B= .09), demonstrating a within level well below 0.08, although the between level remains higher than desired. The BIC for Model 2 (21,292.51) was also lower than the BIC for Model 1, indicating better fit.

Table 3.

Multilevel Confirmatory Factor Analysis: Fit Indices for Two Models

Fit Index	Model 1: Two Factors, 5-item Career Support, 5-item Psychosocial Support	Model 2: Two Factors, 4-item Career Support, 5-item Psychosocial Support
X2	252.73	145.93
df	68	52
CFI	.93	.96
TLI	.91	.95
RMSEA	.06	.05
SRMR
Within	.09	.06
Between	.10	.09
BIC	23,928.98	21,292.51

Note: CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root mean square error of approximation; SRMR = Standardized root mean square residual; BIC = Bayesian information criterion

Table 4 shows the standardized parameter estimates for Model 2. Within-level estimates (factor loadings) are all significant and large, ≥ .71, indicating stronger associations between the items and the latent factors (career support and psychosocial support) at the individual developer level. Within-level residuals are generally small, indicating adequate unexplained variation or measurement error. Between-level estimates are also all statistically significant and have magnitudes ≥ .66. These estimates represent the average associations between the latent factors across respondents. Between-level residual variances are generally higher than within-level, suggesting some variability between respondents, but not excessive heterogeneity, thereby allowing for meaningful comparisons.

Table 4.

Standardized Parameter Estimates for Model 2: Two Factors, 4-Item Career Support, 5-Item Psychosocial Support

Item	Within Developers			Between Respondents		Residual Variance Estimate
	Career Support Estimate	Psychosocial Support Estimate	Residual Variance Estimate	Career Support Estimate	Psychosocial Support Estimate
Career 1	.76		.42	.89		.21
Career 2	.90		.19	.87		.24
Career 3	.88		.23	.85		.27
Career 4	.71		.50	.66		.57
Psychosocial 1		.87	.24		.69	.53
Psychosocial 2		.86	.26		.56	.69
Psychosocial 3		.72	.49		.85	.28
Psychosocial 4		.82	.32		.79	.37
Psychosocial 5		.81	.34		.72	.48

Note: See Table 2 for item content. All loadings were significant at p < .001.

Internal Consistency Reliability

Multilevel reliability coefficient omegas (ω) for Model 2 were ω_within=.89 for career support and ω_within=.92 for psychosocial support at the developer level, and ω_between=.87 (career) and ω_between=.85 (psychosocial) at the respondent level.

Discussion

MCFA is a strong analytic approach for nested measurements because it partitions the variance into within-developer and between-respondent components. Likewise, the reliability of scales can also be calculated at each level. Prior psychometric research of the Career Support and Psychosocial Support scales has yet to fully take into account the nested data structure and has used simple exploratory or confirmatory factor analysis as support for construct validity, and Cronbach’s alpha as the reliability estimate. This study contributes to the psychometric validation of the two scales in the context of university faculty. The MCFA analysis confirmed the factor structure of the scales at both the within- and between-levels, supporting their construct validity. Likewise, both within- and between-level omega reliability values were comparable in magnitude, supporting strong internal consistency of the scales both within developers and between mentor and mentee respondents. The findings provide evidence of the applicability of these scales in assessing developmental relationships and support among university faculty.

The DNQ and the modified MNQ tested in this study had both five-item Career Support and Psychosocial Support Scales. However, the results of this analysis, showed that the measurement model fit best when the fifth career support item, “Acts as a buffer for you from situations that could threaten your career achievement” was deleted from the Career Support Scale. Whether this result is characteristic of this sample or is more generalizable will need to be tested in future research. However, it is worth noting that “acting as a buffer” is the only protective item (in response to a threatening event) in the scale and also had the lowest mean of all the items. It appears that not all respondents experience a threatening event that requires such a response from someone in their developmental network.

There are two possible limitations to this study. First, statistical power may be limited due to a relatively small sample size at the between-level. While statistical significance is not being tested and the concept of statistical power, strictly speaking, does not apply to factor analysis, the accepted technique when one is testing exploratory models is 10 participants per item plus an additional 50, so 150 participants total for the 10 items in the two scales (Knapp & Campbell-Heider, 1989). Thus, the number of respondents in this study, 156, may be adequate to test the measurement model for established scales. The second limitation, is that only one level of nesting was tested in this analysis. An even higher grouping level, that of the universities employing the mentors and mentees, was not tested due to the sample size restrictions. Differences in culture and climate between organizations may lead to differing faculty perceptions and expectations related to career and psychosocial support. Analyses of measurement models of support with faculty respondents nested by the organization would be important to pursue. In addition, now that evidence continues to accumulate supporting the reliability and validity of these two scales, additional measurement properties, such as predictive validity and responsiveness of these scales to measuring change over time in the amount of support provided to university faculty by developers, should be explored.

Conclusion

Strong mentoring in universities includes helping faculty to build stronger developmental networks. To assess these networks, and to evaluate the effectiveness of interventions designed to improve networks, reliable and valid measures of network characteristics are needed. This study assessed the psychometric properties of two such measures in an academic setting: career support and psychosocial support. This research demonstrated that support provided to university faculty by developers in their networks may be validly and reliably measured by a modified four-item career support scale and the original five-item psychosocial support scale from the DNQ and the modified MNQ.