Multi-Relational Measurement for Latent Construct Networks

Abstract

Multiple item scales have long been used to measure latent constructs on individual-level data. This is appropriate when an otherwise unobserved construct is indirectly measured by combining observable correlated characteristics that are thought to measure slightly different dimensions of that construct. Network data, which consist of observations on the relationships between a set of actors, however, are typically drawn from single-relation measurements. While this approach is sufficient for learning about discrete relations (communication, co-authorship, etc), multi-item measurement of extemporaneous valued relationships, such as cohesion and conflict, may be of common interest in psychology and related sciences. In this paper, we evaluate the use of multi-relational network measurement in inferring valued latent construct networks. In particular, we present a psychometric framework for developing multi-relational measures of latent construct networks, evaluating their reliability and construct validity, and identification of appropriate scaling approaches for these construct-level networks.

Introduction

The evidence base is building that social relationships are significant contributors to cognition and behavior and, as such, are important to consider in the social and behavioral sciences. Methods examining these relationships and their potential influences have been developed and applied across psychology and related disciplines. These methods consider the social relationships and their association with cognitions and behaviors at different levels, including the individual-, dyadic-, and network-levels. For example, research questions that aim to understand how social relationships might impact individual-level outcomes are often addressed using generalized linear models, with indices of interpersonal processes entered as predictor variables (Tay, Tan, Diener, & Gonzalez, 2013). Such models are appropriate when considering independent samples and the interpersonal environment surrounding each independently sampled respondent. In other contexts, the focus is on dyadic relationships, and the interplay of intrapersonal attributes with interpersonal processes (Lewis et al., 2006; Pietromonaco, Uchino, & Dunkel Schetter, 2013). Dyadic models, such as the Social Relations Model or Actor-Partner Interaction Model (Kenny, Kashy, & Cook, 2006) which assume independence across participant dyads, are often used to address such questions. More recently, the field has expanded inquiry to consider whole groups and broader social systems, requiring methods that consider both the function of resources exchanged among members of such groups and their relational structure (Koehly, Ashida, Schafer, & Ludden, 2015). Social network analysis is one set of tools and methods that meet such needs and its use is becoming more prominent in psychology and related fields.

Although not exhaustive, a review of the literature provides examples of integrating social network concepts and methods into a variety of social and behavioral science domains. For example, applications of social network analysis have considered network effects in the job search (van Hoye, van Hooft, & Lievens, 2009), on job performance and group effectiveness (Curseu, Raab, Han, & Loenen, 2012; Gulati, Lavie, & Madhavan, 2011; Jokisaari, 2013), and interpersonal citizenship and counterproductive behaviors within an organizational context (Chung, Park, Moon, & Oh, 2011; Lyons & Scott, 2012). Further work in social psychological applications considered network influences on social identity (Jones & Volpe, 2011), happiness (Fowler, Christakis, et al., 2008), as well as individual differences related to network position and structure (Casciaro, 1998). Research in attachment orientation, ambiguous relational ties, and social support can also be considered within the purview of network science (Uchino, 2006; Uchino, Holt-Lunstad, Smith, & Bloor, 2004). Moreover, recent applications in health contexts consider social influence processes related to health behaviors (de la Haye, Robins, Mohr, & Wilson, 2010; Parker, Vardavas, Marcum, & Gidengil, 2013; Valente, 2010) and the impact of disease diagnoses or genetic test results on family level coping (Ersig, Hadley, & Koehly, 2011).

With this burgeoning use of network methods in psychology and related fields comes a need to consider how best to measure relational processes within a particular group or social context. Currently, the use of a single item measure as a proxy for the relational construct of interest is the normative approach in network science. While this approach may be appropriate in some situations, multiple measures may be more appropriate when constructs of interest are not directly observable or are prone to measurement error. To this end, the current paper aims to present a methodological agenda for the psychometric evaluation of a set of relational items proposed to measure a unidimensional relational construct network. We first define social networks and the notation that we will use throughout the paper, as well as provide a brief review of the current state of the science in relational measurement from a social network perspective.

Social Networks

We recognize that not all readers may be familiar with social network analysis and now turn to a brief review. A social network is defined as a set of actors and the relationships between them (Wasserman & Faust, 1994). Actors can include, for example, people, organizations, animals or countries. Relationships capture the ties or connections between actors. Thus, it is assumed that actors are in some way clustered or bounded, be it within schools, families, or within some geographical region. Some of these boundaries may be “rigid” such as students assigned to a particular classroom or “fuzzy” such as families, where membership may be defined by the network members themselves.

To measure networks, one must define the actors of interest, the bounds that group them, and the types of relationships that will be measured. There are two common approaches to collecting network data. The first is ego-centered in that a sample of individual actors—or “egos”—enumerate their network members—or “alters”— and indicate for each pair of network members, including ego-alter and alter-alter pairs, how they are related on some relationship of interest. In this context, the egos are often assumed to be independent, yet the structure within each ego-centered network can reflect dependencies in perceived social exchanges among their alters. The second approach is to consider a bounded group, and obtain information regarding how each member interacts with each other member (i.e., a census). Often, group members are relied upon as informants of the network during the assessment process, in which case the data reflect the concatenation of ego-alter ties. In this context, the informant responses are assumed to be dependent. Relationships may be directed (stemming from one actor towards another as in John spoke to Maria) or undirected (the relationship exists, as in John and Maria spoke). It is worth noting that ego-centered networks can be extracted from whole-network data, by inducing the local neighborhood, or subnetwork, surrounding a particular focal actor. In which case, the set of ego-centered networks derived from the same whole-network data would not be independent of each other.

Social network observations can be represented graphically or as a square adjacency matrix as depicted in Figure 1. The graph is composed of vertices representing actors in the network that are likewise indexed along the rows and columns of the adjacency matrix. The edges, or ties, that connect vertices of the graph are represented in the cells of the adjacency matrix, indicating the value of the relation incident on each pair of vertices. The diagonal of this matrix may optionally store values of self-ties—or loops (Wasserman & Faust, 1994)—as in, John talks to himself.

Figure 1.

Graphical and Adjacency Matrix Representation of a Hypothetical Social Network with Five Actors. Examples are drawn to illustrate the difference between representations of directed vs. undirected and symmetric vs. asymmetric relationships in a network in both graphical and adjacency matrix forms.

Graphical representations provide a visualization of the network structure that enhances understanding of members’ social proximity and how they are clustered together. The matrix representation, however, is convenient analytically such that matrix operations (or vectorized equivalents) can be implemented to compute network metrics or indices that capture the structure of the resulting social system. If we consider the graphical or visual representation presented in Figure 1, we can see that individual actors are connected together into dyads, which are connected to form larger structures within the network system. For undirected relations, there either is or is not a relationship between two actors. However, for directed relations, there are three dyad types: null dyads, in which there is no established relationship between the two actors as in John and Maria did not speak to each other; asymmetric dyads, in which the relational exchange is not reciprocated between two actors as in John spoke to Maria, but Maria did not speak to John, and mutual dyads, in which both actors reciprocate in the relational exchange as in John spoke to Maria and Maria spoke to John. If these dyads are considered at the network level, when directed relations are not always reciprocated, the resulting adjacency matrix is “asymmetric,” and when all dyadic relations are either null or mutual, the resulting adjacency matrix is “symmetric”. Undirected relations always result in symmetric adjacency matrices.

In social network analysis, a common set of actors—say children attending the same school—may have multiple types of relationships between them (e.g., friendship, study-partnership, dominance, sexual). These networks are said to be multi-relational (Marsden & Friedkin, 1993; Mitchell, 1969) and the particular set of relations in a multi-relational network may or may not correlate or overlap. In the current paper, we are interested in the particular subset of relations in a multi-relational network that are correlated and the elements of which measure different aspects of some underlying relational construct (Hirsch, 1979). This paper advances early work that cast overlapping correlated relationships between actors as “multidimensional” networks (Hirsch, 1979) by demonstrating their utility in measuring latent construct networks.

The set of multiple relations to be aggregated into the latent construct network are assumed to be measured using a social network approach. This approach differs from standard measurement that might be used in psychology and related sciences with the primary difference being the unit of measurement and analytic approach. For example, with regards to the classical measurement approach, respondents are assumed to be independent - the focus is on the individual actor, rather than how individual actors are related to each other. In contrast, a social network perspective considers relational measurements between pairs of actors that are usually bounded within a particular group or context - the focus is on the dependence that arises from the context in which the units are bounded. Indeed, here, the network is the unit of analysis, such that individual actors comprise dyads which are connected to form broader network structures (e.g., triads, cliques). Thus, individual actors are necessarily interdependent, and the structure of the relationships that connect actors captures the nature of this interdependence.

While there is a rich literature on measurement of individual-level characteristics in the psychological literature, measurement of social networks is much less developed. In two reviews separated by more than a decade, Marsden discusses conceptual considerations and implementation approaches for collecting network data, with a review of the literature on measurement quality (Marsden, 1990, 2005). In the next section, we briefly summarize these and more recent findings on the measurement of social relations from a network perspective. In so doing, we highlight how the approach that we propose here is distinct from those more common approaches previously presented in the literature.

Network Measurement in Review

Evaluation of measurement quality of network assessments has focused on three primary approaches: test-retest reliability, multi-method measurement format, and congruence based on multiple informant accounts. In general, measurement quality in network science has centered on reproducibility and accuracy of the structural characteristics of a network. For example, test-retest approaches have been used to examine the stability of network composition and aggregate measures of network structures, such as size, in ego-centered networks (Adams, Madhavan, & Simon, 2006). While there are consistent results suggesting that aggregate structures tend to be stable over time (Costenbader & Valente, 2003), the specific composition of the network (i.e., specific alters) appears to exhibit some temporal instability. This instability may be due to network turnover (or “churn”) where actors enter and exit the network between observation periods (Adams et al., 2006; Morgan, Neal, & Carder, 1997). Since interpersonal processes are often dynamic social mechanisms, such shifts or changes in network membership over time are often expected, suggesting that test-retest approaches are not always optimal for evaluating reliability of network assessments.

Another approach to evaluating measurement quality considers the use of multiple measurement modes. Much of this research was conducted in the context of ego-centered network assessment (one actor reporting on their own ties), rather than considering relational measurements obtained from multiple respondents nested within a particular social group. One example is the work by Kogovšek, Ferligoj, Coenders, and Saris (2002) who used a multitrait-multimethod (MTMM) approach to assess the effect of the mode of administration - that is, telephone vs. face-to-face interviews - and question format on the quality of network assessments in aggregate. The question format varied such that characteristics of the ego-alter relationships were obtained “by question” - that is, a given question is asked for all enumerated alters, going question by question - or “by alter” - that is, for each alter, all questions are asked, going alter-by-alter. Their results suggest, for example, that telephone interviewing provided more valid measurements than face-to-face interviews, regardless of question format (Kogovšek, 2006). Further, the question format was important such that asking respondents a series of relational questions for each alter (i.e., “by alter”) yielded more reliable aggregate measures of network attributes than the “by question” format. This research provided valuable guidance on optimal approaches for collecting network data in the context of ego-centered networks. It is worth noting that Coromina, Coenders, and Kogovšek (2004) expanded the work of Kogovšek et al. (2002) by modeling the hierarchical nature of ego-centered network data directly using a multilevel MTMM approach. In so doing, the variance-covariance components are partitioned to obtain reliability and validity indices at two levels: within-ego and between-egos. Thus, the multilevel MTMM approach allows for a more comprehensive understanding of measurement quality. However, neither the aggregate nor the multilevel MTMM provide metrics for evaluating the psychometric properties of multi-relational assessments of construct networks. Rather, relational measures were considered as single item assessments of separate constructs mapped within a given network system.

Multi-rater networks also have been used to evaluate measurement quality. Such networks arise when a set of judges, informants, or raters provide their perceptions or observations of the interactions among a group of network members. Respondents may be raters outside of the group, or they may be members of the group or network under study, in which case the data represent a cognitive social structure (Koehly & Pattison, 2005; Krackhardt, 1987; Newcomb, 1961). A model-based multi-rater approach described by Iacobucci, Neelamegham, and Hopkins (1999) resulted in an inter-rater reliability index comparable to Cronbach’s α, indicating levels of agreement among multiple raters. Similarly, using exponential random graph models, consensus among raters of the network can be stochastically derived (Koehly & Pattison, 2005); however, such an approach provides a general statement about the propensity for agreement or disagreement about the social structure among raters, rather than a specific index of reliability in measurement. While variability in rater assessments of network structure may indicate lack of measurement quality, an alternative explanation is that this variability reflects cognitive biases in informants’ mental models of social structure. Such biases may indicate variability in how network members perceive the social structure within which they are embedded. Indeed, there is evidence to suggest significant “ego bias” based on informant degree, such that informants perceive themselves to occupy more central positions in the network than observed by the other informants (Johnson & Orbach, 2002; Kumbasar, Romney, & Batchelder, 1994). Bayesian methods for modeling such biases directly as Type-1 and Type-2 errors have been developed by Butts (2003), but these are most appropriate when the actors are informing on a single item relation, such as authority or communication (Marcum,Bevc, & Butts, 2012).

While not a traditional index of measurement quality, mutual acknowledgement of a relationship is often used as an index of measurement accuracy (adams & Moody, 2007; Calloway, Morrissey, & Paulson, 1993). Here, it is assumed that a relational tie exists between two network members if and only if both indicate that it exists. This approach can be considered a multi-rater approach, where each rater is involved in the measured social tie. When considering mutuality—or concordance—as a confirmation of a tie, there is significant variability seen within the literature, with percentages ranging from about 40% to 50% (Bernard, Killworth, & Sailer, 1982; Freeman, Romney, & Freeman, 1987) to 75% (Calloway et al., 1993; Marsden, 1990) to 85% (Adams et al., 2006) across contexts. However, for directed relationships, the relationship between a given pair of network members does not need to reflect the same intensity or quality from both members’ perspectives. For such relationships, measures of agreement or concordance would not be appropriate as an index of measurement accuracy. Indeed, asymmetry may be an important dyadic type in some research contexts, which would be lost to observation using the mutuality rule.

Three different approaches to evaluating the quality of network assessments were detailed above, leveraging multiple assessments across time, multiple measurement methods, and multiple raters. However, none of these approaches address the context of multi-relational networks, where these multiple items are posited to measure one underlying unidimensional construct network. In this paper, we aim to fill this gap in the literature by considering relational constructs, or construct networks, that reflect latent qualities of interpersonal relationships within the context of a network system. A framework that outlines steps towards developing such construct networks and evaluating their psychometric properties follows.

A Measurement Theory for Construct Networks

Measurement of “hypothetical constructs,” that is entities, processes, or events that are not directly observed (MacCorquodale & Meehl, 1948), has a long history of methods development in psychology (Strauss & Smith, 2009). While the goal of this paper is not to summarize the universe of measurement theory, there are key concepts that can represent guiding principles for developing multi-relational measures of construct networks. These include: 1) develop a construct map (i.e., a nomological network¹); 2) evaluate measurement quality; and, 3) identify appropriate scaling.

Construct Map

The construct map represents the conceptual scaffolding that guides measurement development, and is directly linked to the concept of construct validity—the extent to which the operationalization of a construct (i.e. methods and items proposed to measure a given hypothetical construct) actually measures what theory suggests it measures (Cronbach & Meehl, 1955; Strauss & Smith, 2009). The construct map details theoretical constructs and how they are related to each other, along with the individual items used to measure each construct. Thus, there are three “levels” represented in the map: 1) the “macro” level representing how theoretical constructs are hypothesized to relate to each other; 2) the “meso” level representing the empirical items hypothesized to measure each construct; and 3) the “micro” level representing those empirical items hypothesized to correlate with each other through the constructs being measured.

Conceptually, this map is no different when considering networks; rather than individual-level items hypothesized to measure a given construct, a set of single network relations are proposed to measure a given construct network. Figure 2 represents two hypothesized construct networks, family conflict and family cohesion, and the network items assumed to measure each of them. At the theoretical level (or macro-level), one might consider predictive pathways between constructs that constitute avenues for assessing predictive or concurrent validity, as well as constructs that are theoretically unrelated, representing the potential for discriminant validity (Loevinger, 1957). In this way, the construct map depicts a framework for evaluating and assessing validity of a construct network, as well as a framework that facilitates theory advancement (Landy, 1986; Messick, 1995). In Figure 2, for example, we hypothesize that family conflict and family cohesion are uncorrelated, which is represented by the absence of an “arc”—or curved, two-headed arrow—connecting the two at the construct level. The presence of an arc would suggest that these two construct networks are hypothesized to be correlated. Using this construct map, we can assess whether the associations presented in the construct map hold as expected at the macro-, meso-, and micro-levels. Approaches for evaluating such expected associations are detailed in the next section.

Figure 2.

Construct Map: Example of Multiple Relational Item Measurement in a Hypothetical Construct Networks from Conflict and Cohesion Relations. Nodes represent actors (n=10) and directed lines and arrows (i.e., edges) represent conflictual and cohesive relationships between actors. Note that the actor set is constant across all relational observations. The value of each construct relationship is indicated by the width of the respective edges in each construct network. Edges are directed (as indicated by the arrowheads that are also scaled by value), which demonstrate sender-receiver relational perspectives.

Measurement Quality

Since our approach to measuring latent construct networks involves multiple relations, we first formalize our definitions for clarity as we adapt classical approaches to the network case. Here, we follow Wasserman and Faust’s (1994) notation for representing multiple relations incident on a set of actors. Let r be the number of binary relations between η actors. We define a set of r η × η adjacency matrices as X = {X₁,X₂,…,X_r}. The value of a relation indexed by the cells in each matrix is x_ijk = 1 if a tie from actor i to actor j on relation k exists (and x_ijk = 0 otherwise), for all 1 ≤ i ≤ η, 1 ≤ j ≤ η, i ≠ j, and 1 ≤ k ≤ r. We define the construct network R as the sum of relations in X:

$$R=X_1+X_2+\cdots +X_r,$$ (1)

which is a valued η × η matrix.

Of note is the connection between the construct map, our definition of the construct network, and True Score Theory. One might posit that R can be partitioned into that part which is stable, which we will refer to as “true score,” and measurement error:

$$R=\mathbf{\text{T}}+\mathbf{\text{E}},$$

where R is the observed construct score as in Equation 1; T represents the stable component or true score of the network measurement R, and E is random error in measurement. Here, T is a network, with elements τ_ij which reflect the relational true score between actor i and actor j on the latent construct network. The unique component (E) is assumed to be independent of the stable component (T). Thus, total variance observed in the valued construct network (R) can be partitioned into independent true score variance and error variance components. This adaptation of classical test theory to multi-relational networks provides a useful model for developing reliability and construct validity indices for those construct networks represented in the construct map.

Reliability and construct validity indices provide information ascertaining the potential utility of an assessment tool (Nunnally, Bernstein, & Berge, 1967). Reliability is defined as consistency in measurement, reflecting reproducibility of the measured construct over time or across contexts; validity, on the other hand, captures the degree to which the items or scale accurately measure the construct that they are presumed to measure.

Previous work on the reliability of network measurement has been done by Ferligoj and Hlebec (1995). That work demonstrated the general applicability of measures of equivalence to evaluate the reliability of a composite of relational observations—including the well-known coefficients Cronbach’s α and Armor’s θ. However, those derivations were used to validate composites based on varying measurement modes (e.g., telephone and face-to-face interviews) of a single relation network. While related, our problem considers assessing the reliability and construct validity of latent constructs built from multiple dichotomous relational measurements. Thus, we extend the work of Ferligoj and Hlebec (1995) to the context of multi-relational networks.

Reliability.

Cronbach’s α is a measure of the internal consistency between a set of individual-level items (Cronbach, 1951), and is often used as a conservative estimate of the reliability of a scale, estimating the proportion of true score variance relative to observed score variance, and the amount of measurement error expected when using the instrument (Nunnally et al., 1967). Such measures of internal consistency operate within the empirical framework, or micro-level, of the construct map, evaluating how well the observable items that map to a given construct “hang together.” We can apply a similar approach to construct networks built from multiple dichotomous relational measurements such as those described in Equation 1.

The Kuder-Richardson Formula 20 (KR-20) is a natural choice for the internal consistency estimate of reliability across such a set of dichotomous relations.² The KR-20 statistic is the binomial analog to the well-known Cronbach’s α measure of internal consistency and its relationship with binomial error models resonates with error arising from Bernoulli graphs (Feldt, 1984), which underly many statistical models of networks. Both KR-20 and the Bernoulli graph distribution, assume that the probability of an observed edge is independent of other observed edges. We define the network analytic α, as:

$$\alpha =\frac{r}{r-1}\left[1-\frac{\sum _{k=1}^r\mu \left(X_k\right)\left(1-\mu \left(X_k\right)\right)}{{\sigma }^2}\right],$$ (2)

where, r is the number of dichotomous relations in X (as above, the set of adjacency matrices), μ(X_k) returns the network density of adjacency matrix X_k, and σ² is the non-redundant variance of the construct network R (see Eq. 4, below). Network density, of course, is the ratio of extant to possible ties. As in classical applications, interpreting α as a measure of reliability in a single construct assumes that the items are essentially tau-equivalent - that is, each item is an equally sensitive measure of the underlying unidimensional construct. When tau-equivalence does not hold but all other assumptions do, then α will underestimate the construct network reliability.

In classical approaches to reliability, coefficients are often estimated from an independent random sample of respondents providing a generalizable estimate of reliability for the resulting measurement scale. From a network analytic perspective, one can assess a reliability coefficient, such as α, for a construct network on a single network system (particularly when the number of actors is large). However, in order to advance the use of construct networks, reliability assessment across a set of networks can also be conducted. This is especially warranted when a construct is assumed to measure a latent construct at the population-level but any particular network sampled only contains a small number of actors (and, thus, a small number of potential edges). A meta-analytic approach that considers the distribution of reliability coefficients across networks can be used to advance this agenda. Indeed, hypotheses regarding the size of such coefficients can be evaluated using permutation tests (i.e., Quadratic Assignment Procedure, or conditional uniform graph tests) based on standard Bernoulli graph assumptions (Wasserman & Faust, 1994).

Construct Validity.

While the proposed approach for reliability assessment of construct networks focuses on associations among the multi-relational networks composing the micro-level of the construct map, construct validity considers the meso-level, or the hypothesized associations between the network items and their underlying construct network.

A factor analytic approach can be applied to assess construct validity of these multi-relational construct networks. This approach identifies the factor structure that underlies a set of observable items developed to measure a particular construct (Nunnally et al., 1967). Empirically, factor analysis yields factor structural coefficients, which represent correlations between the observed items and the latent construct being measured. These structural coefficients are essential for interpreting the resulting scale and assessing construct validity (Guilford, 1946; Thompson & Daniel, 1996). In addition, these structural coefficients can inform scale development by identifying items that should be dropped because they are not highly correlated with the latent construct being measured. Such factor analytic approaches can be applied to the inter-item graph correlation matrix obtained from the multi-relational networks, yielding necessary elements for computing construct validity indices.

For example, the first eigenvalue of the (unrotated) principal components analysis of the graph correlation matrix based on the set of items assumed to measure a given unidimensional construct can be used as an index of construct validity. This value represents the maximum amount of variance in the first component shared by the items—as such, it is an upper-bound on the reliability (Zinbarg, Revelle, Yovel, & Li, 2005). The product-moment graph correlation between two labeled graphs, X₁ and X₂, is given by:

$$cor\left(X_1,X_2\right)=\frac{cov\left(X_1,X_2\right)}{\sqrt{var\left(X_1\right)var\left(X_2\right)}},$$ (3)

where the covariance is:

$${\sigma }_{X_1{X}_2}=\frac{1}{\eta (\eta -1)}\sum _{ij}\left(x_{ij1}-\mu \left(X_1\right)\right)\left(x_{ij2}-\mu \left(X_2\right)\right),i\ne j.$$ (4)

Note that the non-redundant variance of a labeled graph, is the graph’s covariance with itself and thus: var(X) = σ² = cov(X,X) = σ_X,X as Butts and Carley (2001) point out.

In practice, as above, one can use vectorized forms of the adjacency matrices in the calculations as we do here. We then take the largest eigenvalue of the graph correlation matrix (λ₁). As a measure of construct validity, we—following Ferligoj and Hlebec (1995)—prefer the weighting that Armor’s θ applies to the eigenvalue. This relaxes the assumption that each item is parallel to the underlying construct. Thus, we have:

$$\theta =\frac{r}{r-1}\left(1-\frac{1}{{\lambda }_1}\right).$$ (5)

While not assessed here, other measures that incorporate the graph correlation matrices can be used under their appropriate measurement assumptions. For instance, McDonald’s (1970) total ω, which further retains the property of incorporating the proportion of variance in scale scores associated with a general factor (Zinbarg et al., 2005), is a measure of congeneric reliability (Yurdug ul, 2006). Armor’s θ is akin to ω, though from a principal components basis rather than from a factor analytic basis, and both relax the assumption of tau-equivalence. Note that Eq. 5 is proportional to Eq. 2 when items are parallel- that is they are equally sensitive in measuring the underlying construct network and have equal error variances. From a factor analytic perspective, parallel items will result in equal factor loadings and uniquenesses across items hypothesized to measure the same latent construct. As a result, when all items are parallel, α = θ = ω, but when that assumption cannot be met these correlation based metrics may be interpreted as a maximized (or upper-bound) α in most network cases (Ferligoj & Hlebec, 1995; Yurdug ul, 2006).

The construct map represents hypothesized relationships among multiple construct networks at the theoretical, or macro-level. Using the graph correlations, pooled across networks, factor analytic approaches can be used to evaluate the tenability of these hypothesized associations. For example, factor-level correlations following oblique rotation can provide evidence of convergent or discriminant validity depending on the nature of the hypothesized associations. Similarly, confirmatory approaches can be used based on the graph correlations to model the micro-level measurement model as well as the macro-level associations, providing opportunity to evaluate construct validity and convergent and discriminant validity simultaneously.

Multiplexity in Construct Networks

Multi-relational measurement for construct networks is related to the idea of multiplexity in social network analysis. As a measure, multiplexity captures the tendency for the same set of actors (say, family members) to share a variety of different types of relationships. These relationships may be correlated like those contributing to a construct network, such as providing emotional support and listening partnerships, or they may be orthogonal such as lending money and sharing the same taste for music. There are a number of ways to define and operationalize multiplexity (Smith et al., 2015), but the key feature of any definition is that overlap between the relationships is conserved. Like Smith et al. (2015), we define multiplexity in a conditionally edgewise fashion by evaluating the extent to which ties that are present between actors in one of the relations (that is, edgewise) are also present in the others. We additionally normalize this value by the number of remaining relations in the construct network. Formally, for one conditioning graph (a single relation chosen for edgewise comparisons to the other relations) in a set of relations incident on the same actors:

$$M=\frac{1}{\sum _{ij}{x}_{ij1}}\sum _{k=2}^r\sum _{ij}\frac{x_{ijk}\left[x_{ij1}=1\right]}{r-1}$$ (6)

This measure of multiplexity quantifies the extent of overlap between the ties of one of the relations (the conditioning graph X₁ in Equation 6) and the ties of the rest of the relations in the construct network. Because this measure is sensitive to the initial conditioning graph, we take the mean multiplexity score by iterating the conditioning graph over all r relations in X as our index. The multiplexity index of a particular construct network, then, represents the average fraction of ties shared between its items. This index is proportional to the average of the network densities. It is useful to consider the extremes of this statistic: the value is zero when there is no overlap in the relationships measured on each item and the statistic reaches unity when all ties overlap. One might contrast the network concept of multiplexity against the average normalized sum of the items (Likert, 1974; Spector, 1992). The extremes are the same, but the intervening (and more likely observed) values are more conservative from the network multiplexity perspective due to the edgewise conditioning.

While not a measure of reliability, multiplexity may be used as a contrast against α and θ when describing and evaluating the construct network. This is most useful in the case of construct networks measured by aggregating a set of sparse networks. These constructs may have relatively high reliability but may also have relatively low multiplexity. While multiplexity does not tap into the internal validity of the construct in the same way that the traditional measures based on covariance do, it does provide insight into whether the distribution of the raw scores falls closer to the lower end of the possible range for that score. The index is very useful as a multivariate descriptive statistic as it summarizes the joint network densities of all the relational items contributing to a construct network.

Measurement Scaling

A scale generally is defined as a multi-item instrument that measures the amount or “strength” of a specific construct for a particular individual. Relational data are typically measured using a single item; however, we argue here that construct network scales may offer improved precision in measurement of relational mechanisms. From a social network perspective, we might characterize these constructs as the underlying social context in which discrete social exchanges transpire in time. Moreover, relationships between individuals are often thought to be valued and it is common to treat them as having a “strength.” This concept is directly applicable to latent construct networks as this value will vary with some linear combination of correlated relational observations. There are several approaches for obtaining scale scores that are discussed in the literature. All represent some aggregation of the observed values on the item set. Common approaches are unit weighted, resulting in a sum scale or mean response scale, or weighted composites based on item-level factor loadings. Here we describe several scaling options for multi-relational networks.

In some research contexts, one may wish to interpret the values of edges in R as the strength of the construct relationship. When X contains only binary relations, this interpretation has the convenient meaning that a zero-valued cell indicates the absence of the latent relationship between actors i and j, a 1-valued cell means the relationship is the weakest, and a cell value equal to the total number of items in the construct (r), means that the relationship is the strongest. One may wish to weight the relational items by the associated eigenvector or principal component of the graph correlation or covariance matrix prior to summation. This treats the values of the edges (our observations) as component or factor scores and conveniently retains the reliability of the raw sum while relaxing the assumption of tau-equivalence in the items. The selection of an appropriate weighting and re-scaling of the network variables should be driven by the research question.

In other research contexts, it may be desirable to dichotomize the construct network. Some options for dichotomization include thresh-holding at some theoretically meaningful value. Thresh-holding at the minimum edge value is equivalent to a union-rule network measurement and thresh-holding at the maximum is equivalent to the intersection (product) of the binary component relations (Krackhardt, 1987). Some selected options for scaling the construct network are presented in Table 1 with their calculations and assumptions. Of course, such dichotomous operationalizations may lose all or part of the measurement quality of the construct network—such that the reliability and construct validity indices presented in the previous section no longer apply.

Table 1.

Selected Options for Scaling a Construct Network (R) With Formulae and Assumptions

Scaling	Formula	Assumption
Raw Sum	$r_{ij}=\sum _{k=1}^r{x}_{ijk}$	Each relation contributes equally to construct network from Equation 1
Weighted Sum	$r_{ij}=\sum _{k=1}^r{x}_{ijk}{w}_k$	Each relation contributes some weight to the construct network
Union	$r_{ij}=\{\begin{array}{ll}1,\hfill & \text{if }\sum _{k=1}^r{x}_{ijk}>0\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}$ ∀1 ≤ i ≤ η and 1 ≤ j ≤ η	Presence of any non-zero valued edge of the construct network is sufficient
Intersection	$r_{ij}=\prod _{k=1}^r{x}_{ijk}$ ∀1 ≤ i ≤ η and 1 ≤ j ≤ η	All relations are necessary

Having situated our approach for multi-relational measurement of latent construct networks in the literature, we now turn to an empirical example using a network analog to the Family-Environment Scale on real data from the genomic health literature.

Empirical Example

Our empirical example is drawn from the Outcomes of Education and Counseling for HNPCC/Lynch Syndrome Testing Study (Ersig et al., 2011) (HNPCC). These data were collected within the context of a larger longitudinal study that compared how individuals who obtain voluntary genetic testing for Lynch Syndrome adapt to their test results. To understand the communal aspects of that adaptation process, the authors collected information on the family environment of participating families using a social network approach. The bounds on these networks were broadly defined and contain a mix of kin, friends, and others. For this example, our unit of analysis is at the family network level. There were 157 members from 31 different families interviewed about the relations in their family network. Each family informant enumerated those persons who they considered family. Using this set of enumerated family members, participants then responded to a series of relational questions and indicated which members were incident with them on each of the relations. The sampling frame was previously described by (Ersig et al., 2011), and here we use families from that dataset with at least two informants.

Uniquely, these data contain relationalized versions of the standard relationship dimension of the family-environment scale (FES) items (Moos & Moos, 1994). By relationalized, we mean that a particular FES item was recast to measure its relational equivalent from the perspective of one actor to another—i.e., as opposed to measuring aspects of actors’ overall network perceptions as is typically done in the FES approach (Koehly et al., 2003). So, for example, one of the items on the original FES asks respondents to think about their family of origin and rate the extent of agreement with: “We really get along well with each other.” The relationalized version of this item asks respondents to indicate (from the roster of their family members): “With whom in your family do you get along well?”

We focus on two subscales of the relationalized FES that are theoretically orthogonal to one another: 1) a subset of the relations measured on the family cohesion subscale and 2) a subset of the relations measured on the family conflict subscale (Moos & Moos, 1994). Our methodological approach follows directly from the strategy outlined above: 1) we posit latent cohesion and conflict networks; 2) we propose construct maps that give rise to these networks, which are here an aggregation of seven partially correlated binary relations for each network; 3) we then calculate reliability measures of the cohesion and conflict networks and evaluate them against multiplexity among the component relations; and finally, 4) we discuss four options for scaling the construct network (the raw sum, the weighted sum, the union, and the intersection, of the latent network) and report network density on each option. We report network density because it is highly correlated with many other network centralization statistics (Valente, Coronges, Lakon, & Costenbader, 2008). Table 2 lists the relationalized items from the network survey.

Table 2.

Relationalized Items for Cohesion and Conflict Construct Networks

Cohesion items

“Whom could you really count on to help you out in a crisis situation…?” “Which members of your family support you emotionally?” “With whom in your family do you spend plenty of time?” “With whom in your family do you feel close?” “Whom can you count on to console you when you are very upset?” “Whom can you really count on to be dependable when you need help?” “Whom can you really count on to listen to you when you need to talk?”

Conflict items

“With whom in your family do you not get along well?” “Which family members do not appreciate you as a person?” “With whom in your family do you often become angry?” “With whom in your family do you often lose your temper?” “Which family members do you feel the need to raise your voice with…?” “Which members of your family are overly critical of you?” “Which members of your family are you overly critical of?”

Each interviewed family member was asked to indicate whether the other network members were incident with them on each of these relations (Ersig et al., 2011).

Results

First, we evaluate whether the two construct networks are orthogonal. We do this by pooling the vectorized network data of the component relations in the two constructs from each of the 31 networks and conducting a principal components analysis (PCA). The results are plotted in the left pane of Figure 3. Each point in this figure represents a single relation (i.e., from the set of coordinates associated with the relational items contributing to each construct network). Relations contributing to the cohesion construct network are shown in dark gray and relations contributing to the conflict construct network are show in light gray. The two sets of relations are approximately 90° from one another, demonstrating good face validity for the orthogonality of the two constructs. Within each set, it is also clear from the figure that the relational items are homogeneously clustered and positively correlated. To further investigate the orthogonality assumption, we repeat this analysis using an oblique rotation (via the “promax” method) on the two-factor solution and compared the results to the PCA. The results, illustrated in the right pane of Figure 3, show very minor anti-correlation between the two factors (r = −0.11, or about 1% of their shared variance) and the two factor scores correlate highly with the first two principal components (r = 0.98 for cohesion and r = 0.88 for conflict, respectively). These results suggest that the two construct networks are not absolutely orthogonal as measured, but that the correlation between them is not substantial, providing evidence for discriminant validity.

Figure 3.

Biplots of the loadings from Principal Components Analysis (left pane) and Oblique 2 Factor Analysis (right pane) of the Cohesion and Conflict Relational Items, Pooled Across all Family Networks (n=31). Cohesion relational items are shown in dark gray and conflict relational items are shown in light gray.

Second, we turn to the reliability and multiplexity of the relational constructs. Figure 4 plots the empirical distribution for α, θ, and multiplexity for cohesion and conflict from the 31 family networks. These values were calculated separately for each construct network, reflecting the unidimensional nature of each subscale. The means, approximate 95% confidence intervals, minimum, and maximum values are reported in the first three rows of Table 3. The cohesion construct network demonstrated high reliability and high multiplexity in virtually every family network, with mean α = 0.74, θ = 0.85, and multiplexity= 0.71. Conflict, however, demonstrated moderate reliability with mean α = 0.60, θ = 0.77, but low multiplexity with a mean of 0.36 (or less than 3 out of 7 edges overlapping, on average).

Figure 4.

Empirical Distribution of Reliability Coefficients and Multiplexity for Cohesion and Conflict Latent Construct Networks

Table 3.

Descriptive Statistics for Cohesion and Conflict Construct Networks (n=31)

	Cohesion				Conflict
	Mean	95% CI	Min	Max	Mean	95% CI	Min	Max
Reliability Statistics
α	0.736	(0.704, 0.767)	0.428	0.911	0.599	(0.538, 0.660)	0.049	0.829
θ	0.853	(0.834, 0.873)	0.712	0.932	0.766	(0.720, 0.811)	0.395	0.975
Network Statistics
Multiplexity	0.707	(0.670, 0.744)	0.315	0.864	0.358	(0.290, 0.427)	0.092	0.917
Relationship Value (raw)	3.429	(3.325, 3.533)	0.000	7.000	0.628	(0.576, 0.679)	0.000	7.000
Relationship Value (weighted)	1.554	(1.508, 1.601)	0.000	3.086	0.154	(0.141, 0.167)	0.000	1.866
Density (raw)	0.048	(0.039, 0.058)	0.013	0.104	0.015	(0.011, 0.019)	0.002	0.042
Density (weighted)	0.011	(0.009, 0.012)	0.003	0.023	0.004	(0.003, 0.006)	0.001	0.015
Density (intersection)	0.008	(0.006, 0.010)	0.002	0.023	0.000	(0.000, 0.001)	0.000	0.002
Density (union)	0.190	(0.158, 0.222)	0.055	0.379	0.083	(0.059, 0.106)	0.011	0.244

Finally, we illustrate different scaling scenarios for the construct networks. Summary statistics for sums (that is, the value of the relationships in the construct networks), and network densities (that is, the ratio of the sum of extant to possible relationship values—adjusted by what’s possible given the number of informants in each network—in the construct networks (Wasserman & Faust, 1994)) are reported in the last six rows of Table 3. Relationship sums are reported for raw and weight scaled networks (we project the vector of factor—or principal component—scores back onto their appropriate edges in the raw sum scaled network), and densities are reported for raw, weighted, intersection, and union scale networks (the former two being valued and the latter two being binary networks). The results show that network density varies substantially depending on the scaling of the construct network. For cohesion, despite having relatively high average non-zero edge values, raw and weighted scales exhibit low network density. For conflict, on the other hand, raw and weighted scales are low on both metrics. Union rule construct networks, on the least restrictive scale, realize the most connectivity (an average of 20% of edge values in cohesion and 8% in conflict) while intersection rule construct networks, on the most restrictive scale, realize the least connectivity (almost all zero-valued relationships on average).

It is also strikingly clear from the results where the discrepancy in multiplexity arises between cohesion and conflict, with respect to reliability: the latent cohesion networks are richly connected while the conflict networks are sparse, making it relatively easier for edges to overlap in the observed cohesion relations than in the conflict relations. This is perhaps best illustrated by the example in Figure 5, which is a plot of the raw score cohesion and conflict construct networks for one of the families. The lines and arrows in each network are colored on a gradient from red-to-blue, where red represents the minimum value of the factor weighted edge value and blue represents the maximum. The four informants in this family are represented by large black nodes and non-informants are represented by smaller gray nodes. An edge between two actors i and j, is present when i nominated j as a network member and reported information relational information about conflict and cohesion. The sparsity of the conflict network is highlighted by the fact that most of the edges are pure red, or have minimal conflict. All other family networks are depicted in two aggregate figures in the Appendix.

Figure 5.

Cohesion and Conflict Construct Networks from Family 3 in the HNPCC Network Dataset. Edges and arrowheads are colored by the weighting that the factor score solution applies to each construct network. For visualization, the edge colors have been rescaled to represent the range of the minimum to the maximum values of the weighted edge data, such that pure red represents the minimum and pure blue represents the maximum and any color on the red-to-blue gradient represents intermediate values. There are 24 actors in this family network, including 4 informants (large black nodes) and 20 non-informants (small gray nodes). The graph drawing layout was accomplished through a force-directed placement algorithm á la Fruchterman & Reingold (1991).

Discussion

There is a strong methodological literature that considers multi-relational networks with a focus on developing multivariate techniques to identifying structurally equivalent network positions (Brusco, Doreian, Steinley, & Satornino, 20133) or fitting multivariate network regression (Krackhardt, 1988) or random graph models (Koehly & Pattison, 2005; Pattison & Wasserman, 1999). However, none of these consider the development of construct networks derived from multi-relational networks. Guided by scale development procedures commonly applied in psychology, we provide a psychometric framework for evaluating the measurement quality of these construct networks. This makes a considerable step forward in the literature we reviewed as, to date, there is no standard approach for evaluating the reliability and construct validity of multi-relational unidimensional construct networks. Here, we provide steps to guide research to develop such construct networks. First, we propose the development of a construct map that details the item set hypothesized to measure the constructs of interest, as well as how proposed construct networks are associated with each other. Second, we propose indices of reliability, construct validity, and multiplexity for evaluating the quality of construct network measurement based on multi-relational network assessments. And, third, we discuss several approaches to scaling construct networks, two of which are composite or aggregate measures that map directly to the proposed reliablity and construct validity indices.

With respect to measurement quality, we show that standard measures such as α and Armor’s θ (or even McDonald’s ω) can be adapted for unidimensional construct networks using graph correlations based on the set of adjacency matrices representing the set of observed binary relational networks. These approaches are contrasted with an omnibus multiplexity index, which captures the tendency for edges to be shared across the set of observed networks. Using an empirical example, based on a relationalized version of the Family Environment Cohesion and Conflict Subscales, we demonstrated similar levels of reliability and construct validity as previously published for the original individual-level scale. The index of multiplexity, however, illustrates particular issues with using α and θ in sparse networks (as opposed to dense ones, where all indices will tend to rise). As a purely network-level edgewise statistic, we recommend that multiplexity be calculated as an offset to overly optimistic structural statistics when relations contributing to a construct network are rare, as was the case in our conflict example.

We suggested four scaling approaches, two yielding dichotomous construct networks and two yielding valued construct networks. One advantage of the dichotomous scaling is the vast array of network metrics and analysis tools that have been developed specifically for binary relations. Our empirical example showed that the two dichotomous scalings yielded construct networks that were most sparse (e.g., that derived from the intersection rule) and most dense (e.g., that derived from the union rule). The intersection rule is the most restrictive and yields a construct network that may not be useful in practice, particularly when there are few items underlying the construct of interest. Scales derived from the union rule, on the other hand, may maximize variability yet be prone to measurement error. In contrast, the sum (or average) and weighted composite scales are grounded in true score theory, and thus have the advantage of the indices of measurement quality developed in the current paper.

Here, we proposed a psychometric framework for developing multi-relational construct networks. The construct map provides the scaffolding for developing theoretically sound unidimensional construct networks based on correlated multi-relational networks. It is important to note that scale construction may be iterative and the construct map modified based on subsequent analyses. For example, the factor analysis may yield factor loadings indicating that some items do not correlate highly with the hypothesized factor; such items can be removed from the construct map as the scale is further developed. In addition, theoretical associations among multiple constructs can be hypothesized at the macro-level and explored through both exploratory and confirmatory factor analytic approaches. The graph correlations used to conduct these analyses retain the underlying dependencies within the network system. The resulting construct network scale yields a valued network for further analysis of network structure. The resulting construct network scales may also be signed, in which case methods for modeling signed networks would be appropriate (Doreian, 2008; Doreian & Mrvar, 2009). Finally, while our empirical example focused on family networks with multiple-informants, the framework we have established in this paper is general to a range of network sampling strategies, from entirely ego-centered to entirely socio-centric (whole networks) and everything in between. Researchers should take care to adjust the calculations outlaid here to ensure that statistics reflect what is actually possible under their own network design scheme.

Limitations and next steps

There are several limitations that should be noted. The proposed indices of reliability (α) and construct validity (Armor’s θ) may be over-estimated when many items are used to measure the construct network (Nunnally et al., 1967) or when optimized to use only those items that are highly correlated (Kopalle & Lehmann, 1997). Here, we propose the use of an omnibus multiplexity index in addition to these classic measures. However, there is a need to further explore the utility of the multiplexity index. Moreover, the presented KR-20 α coefficient is a function of network density, and as such, assumes dyadic and item independence. Armor’s θ, however, is derived from the graph correlations amongst items, preserving underlying dependence structures across the set of multi-relational networks.

While the reliability and construct validity indices presented in the current paper can be applied to a single network system, there is need to develop multi-relational scales that are appropriate for broad use across multiple samples of networks. Here, we presented the distribution of these indices in a sample of thirty-one family networks. We used the mean of the reliability statistic to assess the overall measurement quality of the construct network. However, there was significant variability in such indices across networks, suggesting the need for further work to establish guidelines for evaluating measurement quality and developing construct network scales that are generalizable across sampling contexts.

Finally, as mentioned, the value of relationships in construct networks based on a set of correlated multi-relational networks may take the form of a sum, mean scale, or a weighted composite scale. In each of these cases, the construct network will be valued. However, the majority of available network methods, whether for describing structural characteristics or modeling the structure as an outcome, have been derived based on binary network measurement. Thus, modeling these valued networks can be challenging. While not the purpose of this paper, we note that many classical tools for describing and analyzing binary networks are not appropriate for the valued network case, especially when the network is the outcome. We therefore presented several options from the literature for dichotomizing the valued construct network and warned that such transformation loses the reliability associated with the construct. However, some existing options are appropriate for valued networks, including blockmodels (Žiberna, 2007), core-periphery models (Borgatti & Everett, 2000), centralization (Peay, 1980), non-structural regression (Krackhardt, 1988), among others. Additionally, work by Krivitsky (2012), Desmarais and Cranmer (2012), and others that advances the exponential random graph family of network models (Robins, Pattison, & Wasserman, 1999; Wasserman & Pattison, 1996) for valued network outcomes has been developed—leading us to be confident that such a limitation may be overcome.

Appendix

Figure 6.

All 31 Cohesion Construct Networks in HNPCC Network Dataset. Edges and arrowheads are colored by the weighting that the factor score solution applies to each construct network. For visualization, the edge colors have been rescaled to represent the range of the minimum to the maximum values of the weighted edge data, such that pure red represents the minimum and pure blue represents the maximum and any color on the red-to-blue gradient represents intermediate values. Informants are represented by large black nodes and non-informants by small gray nodes. The graph drawing layout was accomplished through a force-directed placement algorithm á la Fruchterman & Reingold (1991).

Figure 7.

All 31 Conflict Construct Networks in HNPCC Network Dataset. Edges and arrowheads are colored by the weighting that the factor score solution applies to each construct network. For visualization, the edge colors have been rescaled to represent the range of the minimum to the maximum values of the weighted edge data, such that pure red represents the minimum and pure blue represents the maximum and any color on the red-to-blue gradient represents intermediate values. Informants are represented by large black nodes and non-informants by small gray nodes. The graph drawing layout was accomplished through a force-directed placement algorithm á la Fruchterman & Reingold (1991).