Novel Psychometric Indicator Assessments: The Relative Excess Correlation and Associated Matrices

Abstract

For a series of indicators used to assess psychosocial constructs, we propose reporting new types of correlation matrices to gain greater insight into the relation of the indicators with one another. What we define as the observed residual correlation (ORC) matrix can give insight as to whether, when a given indicator is above the indicator-average scores across all indicators for that individual, what other indicators might be anticipated to be above that individual’s average score as well. What we define as the relative excess correlation (REC) matrix can give insight, for each pair of indicators, whether the strength of that particular correlation is above or below what might have been anticipated based on the correlation of each of those two indicators with all of the others. The ORC and REC matrices will, generally, have numerous negative entries even if all of the raw correlations between each pair of indicators are positive. We discuss the properties of, and the relations between, these correlation matrices, and their analogues for covariances. The positive deviations of the REC matrix entries from zero also can help identify clusters of indicators that are more strongly related to one another, providing insights somewhat analogous to factor analysis, but without the need for decisions concerning rotations or the number of factors. However, the ORC and REC matrices can also be used purely descriptively to provide insights into understanding the relation of indicators with one another.

In describing the properties of a scale consisting of a set of indicators, it is common to present correlation or covariance matrices. Not infrequently, these matrices are then further employed in factor analytic models to attempt to reason about various aspects of dimensionality concerning the overall set of indicators.^1–4 These quantitative assessments are often oriented towards identifying the commonalities and distinctions across the indicators. Such scales are then employed in subsequent biomedical and social science research.

With a scale consisting of a number of related indicators, one typically anticipates all correlations between indicators to be positive (subject to reverse coding when appropriate). Factor analysis attempts to identify the common sources of variation across indicator sets. Such approaches are often also used in refining sets of items to identify relatively homogeneous sets,⁵ either for purposes of reliability or for more refined inferences.

In this article, we propose additional correlation and covariance matrices that, either in the process of measure development or evaluation, shed more light on differences and similarities across indicators, and thereby also allow one to see when the relations between particular indicators are especially strong. While such insights also emerge from factor analysis, factor analyses are model-dependent, often employing parametric and linearity assumptions concerning a latent variable that cannot easily be assessed. Here, we will propose an approach that is not model-dependent. Much work in epidemiology, especially in causal inference,^6–10 has sought to provide definitions of quantities that do not rely on models. Models will often eventually be employed in estimation, but the definitions themselves are not model-dependent. The approach proposed here is in some ways analogous.

We propose two related sets of matrices. We will assume each indicator is assessed on the same scale, but we will return to this matter in the discussion. The first set of matrices we will refer to as observed residual covariance and correlation matrices. These matrices will correspond to the respective covariance and correlation matrix for variables defined, for each respondent, as the difference between each indicator and the average score across all indicators for that respondent. The second set of matrices we will define as relative excess correlation (REC) matrix (and corresponding covariance matrix), wherein each entry corresponds to the extent to which the difference between the correlation between two indicators versus the overall average correlation across all pairs of distinct indicators exceeds the sum of the difference between average of the correlations between the first indicator and other indicators versus the overall average correlation, and the difference between average of the correlations between the second indicator and other indicators versus the overall average correlation. The entries of this second matrix are somewhat analogous to “relative excess risk due to interaction,” sometimes employed in the epidemiologic literature on interaction analysis.¹⁰

We will discuss the relations and properties of these observed residual and relative excess matrices and illustrate their use through a series of examples. We will also formulate a proposal for presenting REC matrices to provide insight both into clusters of indicators that are especially strongly correlated and into whether theoretically derived partitions of indicators have empirical support. We will compare and contrast such uses with what is often accomplished with factor analysis. We will argue that the approach of using REC matrices has certain advantages, including alleviating the necessity of making decisions on the number of factors or on factor rotations, greater freedom from parametric assumptions, and potentially greater capacity to assess multiple overlapping theoretical partitionings. However, one can also treat these matrices purely descriptively, and this too can give additional insight into the properties of scales and assessments, and the relations of the indicators with each other.

OBSERVED RESIDUAL CORRELATION AND RELATIVE EXCESS CORRELATION MATRICES

Suppose that in the assessment of some underlying construct, a series of indicators $(X_1,\dots ,X_d)$ is measured. The average of these indicators, $\overline{X}=\frac{1}{d}\sum _{k=1}^d{X}_k$, is frequently employed as an overall assessment for the construct.¹¹ For each respondent, define the observed residual for indicator i as $E_i=X_i-\overline{X}$, where $\overline{X}=\frac{1}{d}\sum _{k=1}^d{X}_k$ is the mean for the respondent. The observed residual correlation (ORC) matrix is defined as $Corr(E_1,\dots ,E_d)$ and the observed residual covariance matrix as $Cov(E_1,\dots ,E_d)$. Note that because each indicator $X_i$ is part of the average $\overline{X}$, it might be expected that many of the entries in this matrix will be negative. If, for a respondent, a particular indicator $X_i$ exceeds the average $\overline{X}$ of all of the indicators for that individual, then the average of the other indicators excluding $X_i$ must be below the overall average $\overline{X}$ for that respondent. However, it may be the case that for two particular indicators, $X_i$ and $X_j$, their observed residuals in fact have a positive correlation so that $Corr(E_i,E_j)$>0. If this were so, then, for a respondent for whom $X_i$ lies above the average-indicator score $\overline{X}$ for that respondent, it would be expected that $X_j$ would be above the respondent’s average $\overline{X}$ as well. While, in the context of assessment scales for various constructs, it would in general be expected that all indicators would be positively correlated, a positive correlation between the corresponding residuals, $E_i$ and $E_j$, will occur less frequently and thus constitutes a stronger form of relatedness between two indicators.

The observed residuals are defined as the within-person centered scores and are similar to group-mean centering in hierarchical linear models^12,13 or growth models.¹⁴ Using the person mean across all indicators can be seen as a first approximation of factor scores in a single-factor model.¹⁵ The observed residuals might likewise be conceived of as the residuals obtained after fitting a one-component principal components model to data with loadings constrained to equality, though the definition of the ORC matrix itself is not model-dependent, and can be employed even if linearity assumptions do not hold.

We will now define the REC and relative excess covariance matrices and consider their relation with the observed residual matrices above. Let ${\rho }_{ij}$ denote the i-j entry for $Corr(X_1,\dots ,X_d)$; let ${\rho }_i=\frac{1}{d}\sum _{k=1}^{d}Corr(X_k,X_i)$ denote the average correlation of $X_i$ with each indicator, including itself, and let ${\rho }_{–}=\frac{1}{d^2}\sum _{k=1\text{,}l=1}^{d\text{,}d}Corr(X_k,X_l)$ denote the overall average value of all entries in the correlation matrix $Corr(X_1,\dots ,X_d)$. A matrix $R$ whose entries are given by $R_{ij}=({\rho }_{ij}-{\rho }_{\cdot \cdot })-\{({\rho }_{\cdot i}-{\rho }_{\cdot \cdot })+({\rho }_{\cdot j}-{\rho }_{\cdot \cdot })\}$ we define as the uncorrected REC matrix. The term $({\rho }_{ij}-{\rho }_{\cdot \cdot })$ expresses the extent to which the correlation between $X_i$ and $X_j$ exceeds the average correlation across all indicators. The term $({\rho }_{\cdot i}-{\rho }_{\cdot \cdot })$ expresses the extent to which the average correlation between $X_i$ and each of the indicators exceeds the overall average correlation across all indicators, and analogously for $({\rho }_{\cdot j}-{\rho }_{\cdot \cdot })$. The quantity $({\rho }_{ij}-{\rho }_{\cdot \cdot })-\{({\rho }_{\cdot i}-{\rho }_{\cdot \cdot })+({\rho }_{\cdot j}-{\rho }_{\cdot \cdot })\}$ thus expresses the extent to which the “excess” correlation for indicators $X_i$ and $X_j$ above the overall average correlation exceeds the sum of the “excess” average correlation for indicator $X_i$ above the overall average plus the "excess" average correlation for indicator $X_j$ above the overall average. The average across all values of the REC matrix will be 0.

The interpretation of this matrix is potentially complicated by the fact that the quantity ${\rho }_{\cdot i}$ includes in its average the correlation of $X_i$ with itself, that is, 1. In trying to understand the relative magnitude of correlations to one another, it might be thought best to carry out a similar set of comparisons, but excluding the correlations of each indicator with itself. Thus, analogously, let ${\rho }_{\cdot i}^{\ast }=\frac{1}{d-1}\sum _{k:k\ne i}Corr(X_k,X_i)$ denote the average correlation of $X_i$ with each other indicator excluding itself, and let ${\rho }_{\cdot \cdot }^{\ast }=\frac{1}{d(d-1)}\sum _{k,l:k\ne l}Corr(X_k,X_l)$ denote the overall average value of all off-diagonal entries in the correlation matrix $Corr(X_1,\dots ,X_d)$. A matrix $R^{\ast }$ whose entries are given by $R_{ij}^{\ast }=({\rho }_{ij}-{\rho }_{\cdot \cdot }^{\ast })-\{({\rho }_{\cdot i}^{\ast }-{\rho }_{\cdot \cdot }^{\ast })+({\rho }_{\cdot j}^{\ast }-{\rho }_{\cdot \cdot }^{\ast })\}$ we will refer to as the REC matrix. It captures the extent to which the excess of the correlation for indicators $X_i$ and $X_j$ above the overall average correlation between all distinct indicators pairs exceeds the sum of (1) the excess of the average correlation of indicator $X_i$. with other indicators above the overall average correlation between all distinct indicator pairs and (2) the excess of the average correlation of indicator $X_j$ with other indicators above the overall average correlation between all distinct indicator pairs. The average across all values of the REC matrix will likewise be 0. We will abbreviate “relative excess correlation” by “REC.”

In some ways, the REC matrix entries capture a type of “interaction,” but for correlations. Each entry captures whether that particular correlation is higher or lower than might be anticipated given the average correlation of the first indicator with all others, and given the average correlation of the second indicator with all others. It is analogous in some sense to the “relative excess risk due to interaction” sometimes employed in the epidemiologic literature for interaction analysis.¹⁰ When a given entry in the REC matrix is positive, then the value of that correlation is higher than might have been anticipated based on the average correlation of each of the two indicators with all others; the “interaction,” with regard to the correlation between indicators $X_i$ and $X_j$, is positive. When a given entry in the REC matrix is negative, then the value of that correlation is less than might have been anticipated based on the average correlation of each of the two indicators with all others; the “interaction,” with regard to the correlation between indicators $X_i$ and $X_j$, is negative. Such “negative interaction”—a negative entry in the REC matrix—can arise even if all raw pairwise correlations are positive. Indeed, the average of all of the REC entries will be zero. The REC matrix is, in some sense, picking up “second-order” or “interactive” correlational relations.

Replacing correlations by covariances gives rise to the analogous definitions for covariances. For example, the "uncorrected" relative excess covariance matrix $Q^{\ast }$ is defined by a matrix with entries $Q_{ij}^{\ast }=(C_{ij}-C_{\cdot \cdot }^{\ast })-\{(C_{\cdot i}^{\ast }-C_{\cdot \cdot }^{\ast })+(C_{\cdot j}^{\ast }-C_{\cdot \cdot }^{\ast })\}$, where $C_i=\frac{1}{d}\sum _{k=1}^{d}Cov(X_k,X_i)$ and $C_{\cdot \cdot }=\frac{1}{d^2}\sum _{k=1,l=1}^{d,d}Cov(X_k,X_l)$; and the definition of the relative excess covariance matrix Q* is analogously obtained by replacing correlations with covariances in the REC matrix R*.

We will now consider properties and relations of the observed residual and relative excess matrices. Proofs of propositions are given in the Appendix.

Proposition 1. The observed residual covariance matrix is equal to the uncorrected relative excess covariance matrix, that is, $Cov(E_i,E_j)=(C_{ij}-C_{\cdot \cdot })-\{(C_{\cdot i}-C_{\cdot \cdot })+(C_{\cdot j}-C_{\cdot \cdot })\}.$

Proposition 1 states that what we have defined as the observed residual covariance matrix is identical to what we have defined as the uncorrected relative excess covariance matrix. The matrix itself has two different and interesting interpretations: one concerning the covariances of the observed residuals, and the other corresponding to the relative magnitude of the covariance to what might be anticipated given the covariances across all indicators. The presentation of the matrix, with its two different interpretations, can give additional insight on the relation of indicators with one another. Covariance magnitudes, however, are often difficult to interpret, and correlations are thus often presented instead. The equivalence for the covariance matrices does not hold for the corresponding ORC matrix and the uncorrected REC matrix for correlations, but these corresponding correlation matrices may also be of interest for each of the two respective interpretations they provide.

The equivalence above also holds only for the "uncorrected" relative excess covariance matrix. However, the “corrected” relative excess covariance has another attractive property: namely that under a “tau-equivalent” classical test theory (CTT) model⁴ (assumptions described in the proposition below), the expected value of its entries is zero.

Proposition 2. Suppose the indicators ($X_1,\dots ,X_d)$ follow a tau-equivalent classical test theory model such that for i=1,…,d we have that $X_i=T+{\epsilon }_i$ where $T$ denotes some true underlying value for each individual and ${\epsilon }_i$ denotes an individual’s random error term for indicator i, and with ${\epsilon }_i$ independent across indicators and individuals. Then $Q^{\ast }=0$.

A consequence of Proposition 2 is that if the standardized indicators followed a tau-equivalent CTT model, then the REC matrix would likewise have entries with zero expectation. When the relative excess covariance or correlation matrix entries deviate from 0, this implies an inconsistency with a tau-equivalent CTT model. Deviating from 0 implies certain indicators are more strongly correlated with one another than such models would allow. The correlation between “error terms” are effectively such as to give rise to “excess correlation” between certain indicators. In a later section, we will use this insight to motivate particular uses of the REC matrix to provide novel forms of evaluation of the empirical support for various partitionings of the indicators into domains or subsets. Note that the reporting of relative excess covariance or correlation matrix does not in any way presume tau-equivalence models; however, assuming such models does result in special properties that may be useful for insights into the structure of covariances.

So as to allow for easier interpretability, and so as to also allow for both sets of interpretations of the observed residual and relative excess matrices, we would in practice propose reporting: (1) the ORC matrix and (2) the REC matrix, using correlations. The use of correlations, instead of covariances, will often aid with the interpretation of the quantities. Because correlations are being used, and also because it is the “corrected” rather than the “uncorrected” REC matrix that is being reported, the two matrices will not be identical. The two matrices, using correlations, give related information but under two different interpretations. The ORC matrix will make manifest those cases in which, when a given indicator is above the indicator-average score across all indicators for that respondent, what other indicators might be anticipated to be above that respondent’s average score as well. The REC matrix will, for each pair of indicators, make clear whether the strength of that particular correlation is above or below what might have been anticipated based on the correlation of each of those two indicators with all of the others. Both interpretations are of interest; both could be reported. In practice, precisely what is reported will often depend on the context. However, in a later section, we will propose an approach with the REC matrix that may also help evaluate aspects of the empirical support of a proposed partitioning of indicators into subsets; and so when only one matrix can be reported, the REC matrix might be considered primary. First, however, we will illustrate the use of both matrices, and their interpretation, with a descriptive empirical example.

EXAMPLE 1—DESCRIPTIVE REPORTING: THE SATISFACTION WITH LIFE SCALE

We illustrate the proposed matrices using Diener et al.’s¹⁶ Satisfaction with Life Scale (SWLS). The SWLS is a well-established measure that has been used extensively: the paper introducing the scale¹⁵ has been cited over 50,000 times. The scale consists of five items (Table 1). The SWLS employs a 7-point Likert scale with responses from 1 = “strongly disagree” to 7 = “strongly agree.” The scale has good psychometric properties: Cronbach’s alpha is high and a single underlying factor seems to explain a considerable proportion of the variance across item responses.^16,17 Our prior research, however, has indicated different potential causal relations across the different indicator facets.^18,19 Here, we examine relations between the indicators themselves using data from a sample of 4,531 respondents in the 2020 wave of the Health and Retirement Study. The raw correlation, ORC, and REC matrices are reported in Table 1.

Table 1.

Raw Correlations, the Observed Residual Correlation Matrix, and the Relative Excess Correlation Matrix for the Satisfaction with Life Scale

Item	(1)	(2)	(3)	(4)	(5)
	Raw correlations					Mean correlations
Life is close to ideal (1)		0.75	0.69	0.56	0.46	0.62
Life conditions are excellent (2)	0.75		0.76	0.60	0.47	0.65
Satisfied with life (3)	0.69	0.76		0.70	0.49	0.66
Have important things in life (4)	0.56	0.60	0.70		0.53	0.60
Change nothing if lived life over (5)	0.46	0.47	0.49	0.53		0.49

						Mean = 0.60
	Observed residual correlation (ORC) matrix
Life is close to ideal (1)		0.10	−0.16	−0.38	−0.41	−0.21
Life conditions are excellent (2)	0.10		−0.01	−0.36	−0.48	−0.19
Satisfied with life (3)	−0.16	−0.01		−0.04	−0.46	−0.17
Have important things in life (4)	−0.38	−0.36	−0.04		−0.17	−0.24
Change nothing if lived life over (5)	−0.41	−0.48	−0.46	−0.17		−0.38

						Mean = −0.24
	Relative excess correlation (REC) matrix
Life is close to ideal (1)		0.09	0.02	−0.05	−0.04	0.00
Life conditions are excellent (2)	0.09		0.05	−0.04	−0.06	0.01
Satisfied with life (3)	0.02	0.05		0.05	−0.06	0.01
Have important things in life (4)	−0.05	−0.04	0.05		0.04	0.00
Change nothing if lived life over (5)	−0.04	−0.06	−0.06	0.04		−0.03
						Mean = 0.00

The raw correlations are all positive. However, only for indicators 1 and 2 is the ORC entry positive, so that when one of these two indicators is above the respondent-level mean, it would be expected that the other is also. For all other indicator pairs, when one indicator is above the respondent-level mean, it would be expected that the other would be below the mean. Fairly similar patterns emerge in the REC matrix. However, in the REC matrix, not only for indicators 1 and 2, but also for indicator pairs 1–3, 3–4, and 4–5, their correlations are somewhat higher than might be anticipated given the average correlation that each of the indicators has with the others, though these RECs (0.05, 0.05, and 0.04, respectively) are not as high as that observed between indicators 1 and 2 (0.09).

USING THE RELATIVE EXCESS CORRELATION MATRIX TO ASSESS EMPIRICAL SUPPORT FOR PROPOSED DOMAINS

For many scales, items are often divided into domains. Sometimes such domains are proposed on theoretical grounds. Other times, domains are empirically derived using techniques such as exploratory factor analysis. The REC matrix arguably provides an appealing approach for assessing empirical support for proposed domains based on which indicators have particularly stronger-than-anticipated correlations with others. Because the average value of the REC matrix entries will be 0, this provides a helpful threshold to identify groups of indicators that are especially strongly correlated with one another.

Suppose that we have a partition of the indicators, ($X_1,\dots ,X_d)$, into t distinct domains: D₁, D₂, …, D_t. As an assessment of the empirical support of the partition, one might report for each domain k, the mean REC entry within that domain, versus the mean REC entry relating the indicators within that domain to those outside of that domain. Formally, if $R_{ij}^{\ast }$ denotes the REC for indicators i and j, then we could report $\frac{1}{\left|D_k\right|\times (|D_k|-1)}\sum _{i\ne j:i,j\in D_k}{R}_{ij}^{\ast }$ and compare this to $\frac{1}{\left|D_k\right|\times (d-|D_k|)}\sum _{i\in D_k,j\notin D_k}{R}_{ij}^{\ast }$ where the first expression is the mean REC entry among indicators in domain k, and the second expression is the mean REC entry relating indicators in domain k with those outside of domain k, and $\left|D_k\right|$ denotes the size of domain k. Suppose the indicators within domain k are particularly strongly correlated with one another, given each of their average correlations with all other indicators, then the former quantity should be considerably greater than the latter. If all REC entries relating indicators within domain k were positive, and all REC entries relating indicators in domain k with those outside of domain k were negative, this would constitute particularly compelling evidence for the indicators within domain k being especially strongly correlated with one another. With a partitioning of the indicators into domains, such metrics could be reported for each domain. A summary assessment could also be given by reporting the average of the within-domain REC means versus the average of the cross-domain REC means. We illustrate these approaches and metrics below.

Further indicator-level insight might also be attained by reporting, for each indicator, its mean REC entry with all other indicators within its assigned domain versus its average REC entry with each of the other domains. This might allow assessing whether a particular indicator was in fact more strongly correlated with indicators in other domains, given their overall average correlations across all indicators. Thus, for each indicator i, with corresponding domain k_i, one might report $\frac{1}{|D_{k_i}|-1}\sum _{j\ne i:j\in D_{k_i}}{R}_{ij}^{\ast }$ and compare this to, for each domain $l\ne k_i$, $\frac{1}{\left|D_l\right|}\sum _{j\in D_l}{R}_{ij}^{\ast }$. The latter quantity would exceed the former if, for domain l, indicator i had a stronger mean REC with the indicators in domain l than within domain k.

In reporting the various metrics above, a helpful baseline assessment to assist in evaluating the relative magnitude of these quantities might include the average of the absolute values of all REC matrix entries. In principle, one might also report the correlations and the REC matrix for the mean domain scores to better understand also how the domain scores themselves are related to each other. We illustrate these various uses of the REC matrix in a second example below.

EXAMPLE 2—EVALUATING THEORETICAL DOMAINS: COMMUNITY SUBJECTIVE WELL-BEING

Our second example concerns an assessment of subjective community well-being (SCWB)²⁰ using a sample of 2,127 residents of Columbus, Ohio.²¹ The SCWB assessment²⁰ consists of four items for each of five domains: good relationships (D1), proficient leadership (D2), healthy practices (D3), satisfying community (D4), and strong mission (D5). The original items from the SCWB measure²⁰ are given in eAppendix A; https://links.lww.com/EDE/C314 and were adapted in data collection²⁰ with “[this/our] community” replaced by “Columbus.” Responses to each item were assessed on a 7-point scale, from 0 = “strongly disagree” to 6 = “strongly agree.” The domains and items were proposed based on theoretical considerations.²⁰ A fuller psychometric evaluation of the SCWB, including assessments of reliability, standard error of measurement, item characteristic curves, and exploratory factor analysis, is given in Padgett et al.²³ Here we illustrate uses of the REC matrix to evaluate empirical support for the proposed domains.

The REC matrix for the SCWB assessment is given in Table 2, with boxes around the indicators constituting distinct domains (see also the heat map in eAppendix B; https://links.lww.com/EDE/C314). The average absolute value of REC entries is 0.06. The REC entries within domains are almost all positive, with the only exception being that the REC relating indicators 17 and 19 is −0.01. Most, but not all, of the REC entries relating indicators of different domains are negative.

Table 2.

The Relative Excess Correlation Matrix for the Subjective Community Well-being Assessment

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)	(12)	(13)	(14)	(15)	(16)	(17)	(18)	(19)	(20)
Close relationships (D1)		0.16	0.14	0.12	−0.03	−0.05	−0.06	−0.06	−0.01	−0.04	−0.05	−0.07	0.03	0.01	0.04	0.01	−0.03	−0.07	−0.02	−0.06
Respect (D1)	0.16		0.22	0.17	−0.03	−0.04	−0.05	−0.07	−0.06	−0.02	−0.05	−0.05	0.06	0.00	0.03	0.05	−0.03	−0.07	−0.09	−0.14
Trust (D1)	0.14	0.22		0.20	−0.03	−0.04	−0.05	−0.05	−0.06	−0.01	−0.04	−0.05	0.09	−0.01	0.02	0.02	−0.03	−0.07	−0.10	−0.14
Mutuality (D1)	0.12	0.17	0.20		−0.01	-0.01	−0.05	−0.03	−0.06	−0.04	−0.04	−0.03	0.05	−0.01	0.00	−0.01	−0.02	−0.04	−0.07	−0.10
Beneficence (D2)	−0.03	−0.03	−0.03	−0.01		0.19	0.14	0.14	0.03	0.03	−0.01	0.01	−0.04	−0.06	−0.04	−0.06	−0.07	−0.03	−0.05	−0.04
Integrity (D2)	−0.05	−0.04	−0.04	−0.01	0.19		0.15	0.14	0.01	0.03	−0.00	0.01	−0.05	−0.06	−0.04	−0.06	−0.04	−0.03	−0.05	−0.04
Competence (D2)	−0.06	−0.05	−0.05	−0.05	0.14	0.15		0.19	0.02	0.01	0.01	0.03	−0.06	−0.06	−0.06	−0.05	−0.04	−0.01	−0.01	0.01
Vision (D2)	−0.06	−0.07	−0.05	−0.03	0.14	0.14	0.19		0.04	0.02	0.01	0.05	−0.07	−0.06	−0.05	−0.08	−0.05	−0.04	−0.00	0.00
Relational growth (D3)	−0.01	−0.06	−0.06	−0.06	0.03	0.01	0.02	0.04		0.09	0.09	0.08	−0.04	−0.03	−0.04	−0.04	−0.03	0.01	−0.01	0.02
Fairness (D3)	−0.04	−0.02	−0.01	−0.04	0.03	0.03	0.01	0.02	0.09		0.12	0.11	0.02	−0.02	−0.02	−0.01	−0.01	−0.04	-0.08	−0.10
Sustenance (D3)	−0.05	−0.05	−0.04	−0.04	−0.01	−0.00	0.01	0.01	0.09	0.12		0.19	0.01	−0.02	−0.05	−0.03	−0.00	0.01	−0.08	−0.04
Achievement (D3)	−0.07	−0.05	−0.05	−0.03	0.01	0.01	0.03	0.05	0.08	0.11	0.19		-0.01	−0.02	−0.05	−0.05	−0.02	−0.00	−0.06	−0.03
Satisfaction (D4)	0.03	0.06	0.09	0.05	−0.04	−0.05	−0.06	−0.07	-0.04	0.02	0.01	−0.01		0.12	0.07	0.07	0.03	−0.02	−0.13	−0.14
Value (D4)	0.01	0.00	−0.01	−0.01	−0.06	−0.06	−0.06	−0.06	−0.03	−0.02	−0.02	−0.02	0.12		0.02	0.05	0.06	0.04	0.00	0.04
Belonging (D4)	0.04	0.03	0.02	0.00	−0.04	−0.04	−0.06	−0.05	−0.04	−0.02	−0.05	−0.05	0.07	0.02		0.16	0.09	0.02	−0.02	−0.06
Welcome (D4)	0.01	0.05	0.02	−0.01	−0.06	−0.06	−0.05	−0.08	−0.04	−0.01	−0.03	−0.05	0.07	0.05	0.16		0.12	0.07	−0.06	−0.07
Purpose (D5)	−0.03	−0.03	−0.03	−0.02	−0.07	-0.04	−0.04	−0.05	−0.03	−0.01	−0.00	−0.02	0.03	0.06	0.09	0.12		0.10	−0.01	0.01
Contribution (D5)	−0.07	−0.07	−0.07	−0.04	−0.03	−0.03	−0.01	−0.04	0.01	−0.04	0.01	−0.00	−0.02	0.04	0.02	0.07	0.10		0.08	0.11
Interconnectedness (D5)	−0.02	−0.09	−0.10	−0.07	-0.05	−0.05	-0.01	−0.00	−0.01	−0.08	−0.08	−0.06	−0.13	0.00	−0.02	−0.06	−0.01	0.08		0.56
Synergy (D5)	−0.06	−0.14	−0.14	−0.10	−0.04	−0.04	0.01	0.00	0.02	−0.10	−0.04	−0.03	−0.14	0.04	−0.06	−0.07	0.01	0.11	0.56

For each domain, the mean REC within domain versus the mean REC relating indicators within that domain with those of other domains is given in Table 3. In each domain, the within-domain mean REC is positive and considerably greater than its corresponding cross-domain mean REC, which are all negative. The smallest difference in these averages is for domain 4 (satisfying community), for which the within-domain mean REC is 0.08 and its cross-domain mean REC is −0.01. The RECs are clearly considerably higher within domains (mean = 0.13) than across domains (mean = −0.02).

Table 3.

Within-domain and Cross-domain Mean Relative Excess Correlation for the Subjective Community Well-being Assessment

Domain	Mean REC Within-domain	Mean REC of Domain Indicators with Indicators of Other Domains
Good relationships (D1)	0.17	−0.03
Proficient leadership (D2)	0.16	−0.03
Healthy practices (D3)	0.11	−0.02
Satisfying community (D4)	0.08	−0.01
Strong mission (D5)	0.14	−0.03

REC, relative excess correlation.

The mean REC of each indicator, relating it to those in each of the distinct domains, is given in Table 4. The mean REC metrics reported in Table 4 induce a pattern matrix somewhat similar to the results in factor analysis, and we will discuss distinctions with factor analysis and the different insights they allow in the following section. For nearly every indicator, the mean REC for its assigned domain is considerably higher than the mean REC of that indicator within any other domain. Of the 20 indicators, the only exception is indicator 17 (purpose), for which the mean REC within Domain 5 (strong mission) to which it was assigned was 0.03, which is lower than its mean REC of 0.07 with the indicators in Domain 4 (satisfying community). Examining the REC entries themselves in Table 2, we see that, while indicator 17 has a relatively large REC of 0.10 with indicator 18 (in Domain 5), its REC with indicators 19 and 20 in the domain are close to 0 (−0.01 and 0.01, respectively). This low mean REC does not mean that the indicators are not positively correlated, only that their correlations are roughly what might be anticipated given the correlations of each of these indicators with all others. In Table 2, indicator 17 (purpose) has a relatively high REC with indicators 15 (belonging) and 16 (welcome) in Domain 4 (satisfying community), with values of 0.09 and 0.12, respectively, which gives rise to its higher Domain 4 mean REC.

Table 4.

Domain Mean Relative Excess Correlation for Each Indicator of the Subjective Community Well-being Assessment

Item	Good Relationships (D1)	Proficient Leadership (D2)	Healthy Practices (D3)	Satisfying Community (D4)	Strong Mission (D5)
Close relationships (D1)	0.14	−0.05	−0.04	0.02	−0.04
Respect (D1)	0.18	−0.05	−0.04	0.04	−0.08
Trust (D1)	0.19	−0.04	−0.04	0.03	−0.08
Mutuality (D1)	0.16	−0.03	−0.04	0.01	−0.06
Beneficence (D2)	−0.03	0.15	0.02	−0.05	−0.05
Integrity (D2)	−0.04	0.16	0.01	−0.05	−0.04
Competence (D2)	−0.05	0.16	0.02	−0.06	−0.02
Vision (D2)	−0.05	0.16	0.03	−0.06	−0.02
Relational growth (D3)	−0.05	0.02	0.09	−0.03	−0.00
Fairness (D3)	−0.03	0.02	0.11	−0.01	−0.06
Sustenance (D3)	−0.04	0.00	0.13	−0.02	−0.03
Achievement (D3)	−0.05	0.02	0.12	−0.03	−0.03
Satisfaction (D4)	0.06	−0.05	−0.00	0.09	−0.06
Value (D4)	−0.00	−0.06	−0.02	0.06	0.04
Belonging (D4)	0.02	−0.05	−0.04	0.08	0.01
Welcome (D4)	0.01	−0.06	−0.03	0.09	0.02
Purpose (D5)	−0.03	−0.05	−0.01	0.07	0.03
Contribution (D5)	−0.06	−0.03	−0.01	0.03	0.10
Interconnectedness (D5)	−0.07	−0.03	−0.06	−0.05	0.21
Synergy (D5)	−0.11	−0.02	−0.04	−0.06	0.23

The REC matrix for the domain scores themselves are given in Table 5. The mean absolute REC of this matrix is 0.03, which is considerably lower than the mean absolute REC concerning indicators of 0.06 for the entries in Table 2. Many of the domain REC entries are close to 0, but that relating D2 (proficient leadership) and D3 (healthy practices) is comparatively large in magnitude, 0.06, and positive; and that relating D1 (good relationships) and D5 (strong mission) is comparatively large in magnitude, but negative, −0.06. This gives additional insight into the relations of not only the indicators but of the domains themselves.

Table 5.

The Relative Excess Correlation Matrix for the Domain Scores of the Subjective Community Well-being Assessment

	(1)	(2)	(3)	(4)	(5)	Mean Correlation
Good relationships (D1)		−0.00	−0.01	0.06	−0.06	−0.00
Proficient leadership (D2)	−0.00		0.06	−0.04	−0.01	0.00
Healthy practices (D3)	−0.01	0.06		−0.01	−0.01	0.01
Satisfying community (D4)	0.06	−0.04	−0.01		0.03	0.01
Strong mission (D5)	−0.06	−0.01	−0.01	0.03		−0.01

In eAppendix C; https://links.lww.com/EDE/C314, we demonstrate how the output for various REC matrices and metrics can be obtained through our R package recmetrics. The R package will output the REC or ORC matrix for a set of indicators, and, with a proposed partition, output also within- versus cross-domain mean REC, such as with Table 3, and indicator-specific domain REC means, as with Table 4, along with the REC matrix for the domain scores themselves, as with Table 5, and the correspond mean absolute REC values.

A final example concerning the ORC and REC metrics for the Patient Health Questionnaire-4 (PHQ-4) scale²¹ for anxiety and depression using US data from the Global Flourishing Study^22,23 is given in eAppendix D; https://links.lww.com/EDE/C314. Factor analyses have sometimes not been able to distinguish two factors for combined anxiety and depression scales,^24–26 though with the PHQ-4, evidence is clearer.²¹ Both the ORC and the REC metrics in eAppendix D; https://links.lww.com/EDE/C314 clearly distinguish the anxiety items from the depression items.

CONCLUSIONS AND DISCUSSION

In this article, we have proposed reporting new ORC and REC matrices to identify stronger types of relations between specific pairs of indicators. Certain uses of these matrices bear some relation to factor analysis, which is often used to try to gain deeper insight into relations between indicators that extend beyond what is immediately transparent from the covariance or correlation matrices themselves. In factor analysis, when a single factor seems sufficient to explain most of the shared variation across indicators, factor analysis often draws attention to commonalities across indicators. When two or more factors seem necessary to adequately explain the sources of common variation across indicators, then factor analysis often highlights distinctions across indicators corresponding to the different factors. However, even when a single factor is thought sufficient, the matrices proposed here could still reveal yet further distinctions across indicators and instances wherein stronger relations between indicators are manifest. Such was the case in our first example concerning Diener’s SWLS, for which prior analyses suggested that a single factor was sufficient to explain most of the common variation across indicators,¹⁶ and yet both the ORC matrix and REC matrix nevertheless revealed especially strong relationships between indicators 1 and 2. The matrices proposed here might thus be seen as a complement to factor analytic methods. Indeed, the metrics and matrices proposed here are effectively agnostic to the supposedly underlying factor structure.

Nevertheless, as illustrated by our second example, the REC matrix can be summarized in ways that evaluate aspects of empirical support for a proposed partitioning of indicators into domains. Exploratory and confirmatory factor analyses are often also used for this purpose^1–4,27 and so it is worthwhile comparing such uses of the REC matrix with those of factor analysis. Several points merit attention.

First, the proposed REC approach, like confirmatory factor analysis, but unlike exploratory factor analysis, allows one to specify, and then evaluate, a proposed a priori partitioning of the indicators into domains. Exploratory factor analysis attempts to uncover such potential partitionings; confirmatory factor analysis and the REC approach evaluate a partitioning already specified.

Second, the proposed REC matrix does not require decisions or evaluations concerning factor rotations or concerning the number of factors. There are thus fewer “investigator degrees of freedom.”^28,29 Of course, with theoretically prespecified domains, one can use it to report the types of domain-specific metrics as given in Tables 3 and 4 above, but the REC matrix itself is agnostic to the domains themselves or to their number.

Third, in factor analysis, when a particular indicator has stronger common variation with other indicators within a different domain than the one to which it was assigned, factor analysis will tend to group this indicator with the domain concerning which it has such stronger common variation. This, in turn, may effectively alter the interpretation of this domain or factor, and also affect what other indicators might be grouped with it, thereby rendering it potentially more difficult to evaluate the original proposed partitioning, even if just one indicator manifests such patterns. For example, with the SCWB assessment above, for only one of the 20 indicators (indicator 17) was the mean REC within its assigned domain lower than that with a different domain. However, when exploratory factor analysis with five factors is employed,³⁰ the fourth factor has relatively large loadings on all of the indicators in the fourth domain, but, because these indicators are also relatively strongly associated with indicator 17, this indicator also loads most heavily on the fourth factor, and, given its relatively strong association with indicator 18, indicator 18 itself then also loads most heavily on the fourth factor. However, in the domain-specific REC analyses given in Table 4, one can see that indicator 18 has a higher mean REC for the indicators in domain 5 (its own theoretically assigned domain) than for those in domain 4. The REC approach allows for a more straightforward assessment of the proposed domains, even if one of the indicators is not more strongly associated with the indicators of its own domain. This may be important in deciding whether items should be altered or eliminated, or whether the domains should be reconceived or redefined. The REC approach may thus be useful not only for measure evaluation, but also for measure development.

Fourth, in certain cases, a set of indicators may admit more than one reasonable theoretical partitioning into domains. The REC approach can more easily be carried out with each separate partitioning, a point to which we will return below.

Finally, the REC approach circumvents difficult questions concerning the meaning and ontological status of “factors.”^{18,19,31–33} The REC approach is effectively agnostic as to whether such factors in any way exist, and whether they might correspond to real scientific entities. This might also be seen as advantageous. In principle, one might likewise also use factor analysis while withholding judgment on whether factors correspond to univariate quantities that are somehow real and scientifically meaningful. However, such withholding of judgment is often not done in practice, and a leap in reasoning is often made from the identification of statistical factors to the belief that they correspond to scientifically meaningful real entities.^18,19

This final point also brings us to some further important interpretative qualifications. Both factor analysis and the REC matrix approach pertain to associations, and such associations may arise from a variety of sources. Correlations may arise because the items corresponding to the indicators might be strongly conceptually related, but they might also arise because of causal effects over time concerning the different facets of the phenomena to which the indicators effectively make reference.^18,34 Association among indicators can arise not only from conceptual relations, but also from causal relations. These considerations concerning conceptual versus causal interpretation of associations again pertain both to factor analysis and to the REC matrix.^19,34

Even if one item is conceptually more closely related to other items within its proposed domain, its corresponding indicator may nevertheless have stronger associations with indicators in another domain because of causal relations between the relevant indicator phenomena. For example, it is possible that some of the association between the purpose indicator in the SCWB assessment (indicator 17 in Domain 5) and the indicators in the satisfying community domain (Domain 4) arise because a common purpose causally gives rise to satisfying community, even though a common purpose itself is more strongly conceptually related to other aspects of a community’s having a strong mission. Such causal caveats and considerations should be taken into account when using the REC approach, or factor analysis, to help guide decisions on whether to modify a partitioning of domains, or on whether to eliminate or replace a particular item. Empirical analyses, such as factor analysis or the REC matrix, can provide considerable insight, but associations must be interpreted appropriately, and decisions about domain partitions and item inclusion should arguably ultimately be principally made on conceptual grounds, informed by empirical analyses.^18,31,35 The psychometric literature has tended to ignore issues of causal relations; more distinctively epidemiologic insights on causation can help remedy this deficiency.

Several directions of potential future research merit attention. First, in certain assessments, indicators are grouped hierarchically, not just into domains, but also into subdomains, that is, further nested partitions within a particular domain.^36,37 The REC approach described above could in principle be applied both to subdomains and to domains, and the approach could be employed at either level, or in a nested manner, first grouping indicators into their subdomains, and then subdomains into their domains.

Second, and conversely, while we have proposed the ORC and REC matrices as tools to gain insight into a complete set of indicators for an assessment, the same approaches and matrices could be employed for a subset of the indicators, such as those within a domain. For instance, in the second example above concerning the SCWB assessment, one might report ORC and REC matrices for the indicators within the “proficient leadership” domain itself to discern whether additional relations might be manifest, which are potentially obscured when the full set of 20 indicators is used. The ORC and REC entries for a subset of indicators will not be identical to the corresponding entries in the matrix for the full set of indicators. The metrics and matrices proposed here, when reported within domains or by factor or facet, might give additional insight.

Relatedly, in certain instances, a set of items may admit more than one reasonable conceptual partitioning into domains (i.e., multiple sets of non-nested domains). The various REC metrics proposed above might then be reported separately for each proposed partition. However, in such cases, for a given partition, one might not necessarily expect the cross-domain mean REC values to be especially low, because there may be relations between the indicators corresponding to an alternative partition. In such cases, an alternative set of metrics to the domain mean REC values might be to compare domain mean REC values for a given domain to mean REC values for indicators that do not share a domain in any of the various proposed partitions. Further work could develop and explore such approaches to employing REC metrics to nested, subsetted, or overlapping partitions.

The ORC and REC metrics can be defined even if indicators are binary or have a small number of response categories, and in principle, even when items employ different scales but are standardized or when alternative types of correlation coefficients are used. Further work could be carried out on the interpretation and properties of these metrics in such settings. Further work could likewise be carried out on the interpretation and statistical properties of the ORC metrics when a weighted mean of the indicators is employed.

The ORC and REC matrices were introduced to help discern stronger types of relationships that indicators may have with one another. We hope that the proposals of reporting these matrices will ultimately aid in a deeper understanding of the properties of scales and assessments.

HUMAN SUBJECTS PROTECTION STATEMENTS

Example 1 used data from the Health and Retirement Study, which is sponsored by the National Institute on Aging and conducted by the University of Michigan.³⁸ Because the data are deidentified and publicly accessible, this research is not considered human subjects research and did not require institutional review board approval or informed consent.

Example 2 used data collected by the Center for Human Resource Research (CHRR) at The Ohio State University. The Ohio State University Institutional Review Board approved the study protocol. Informed consent was obtained from all participants.

The example on anxiety and depression with the PHQ-4 scale included in eAppendix D; https://links.lww.com/EDE/C314 used data from the Global Flourishing Study in which data collection was performed by Gallup, which approved the study protocol and obtained informed consent from participants.²²

Appendix. Proofs.

Proof of Proposition 1. Consider the covariance between two residuals, $Cov(E_i,E_j)$. We can express this as:

$${\displaystyle \begin{array}{l}{\displaystyle \begin{array}{l}{\displaystyle Cov(E_i,E_j)=Cov(X_i-{\displaystyle \overline{X}},X_j-{\displaystyle \overline{X}})=Cov(X_i-\frac{1}{d}{\displaystyle \sum _{k=1}^d}{X}_k,X_j-\frac{1}{d}{\displaystyle \sum _{l=1}^d}{X}_l)}\\ {\displaystyle =Cov(X_i,X_j)-\frac{1}{d}{\displaystyle \sum _{k=1}^d}Cov(X_i,X_k)-\frac{1}{d}{\displaystyle \sum _{l=1}^d}Cov(X_l,X_j)}\\ {\displaystyle +\frac{1}{d^2}{\displaystyle \sum _{k=1,l=1}^{d,d}}Cov(X_k,X_l)=C_{ij}-C_{\cdot i}-C_{\cdot j}+C_{\cdot \cdot }}\\ {\displaystyle =(C_{ij}-C_{\cdot \cdot })-\{(C_{\cdot i}-C_{\cdot \cdot })+(C_{\cdot j}-C_{\cdot \cdot })\}.}\end{array}}\end{array}}$$

This completes the proof.

Proof of Proposition 2. Under the tau-equivalent classical test theory model above, the entries of the relative excess covariance matrix are given by:

$${\displaystyle \begin{array}{l}{\displaystyle \begin{array}{l}{\displaystyle Q_{ij}^{\ast }=(C_{ij}-C_{\cdot \cdot }^{\ast })-\{(C_{\cdot i}^{\ast }-C_{\cdot \cdot }^{\ast })+(C_{\cdot j}^{\ast }-C_{\cdot \cdot }^{\ast })\}=C_{ij}-C_{\cdot i}^{\ast }-C_{\cdot j}^{\ast }+C_{\cdot \cdot }^{\ast }}\\ {\displaystyle =Cov(X_i,X_j)-\frac{1}{d-1}{\displaystyle \sum _{k:k\ne i}}Cov(X_k,X_i)-\frac{1}{d-1}{\displaystyle \sum _{k:k\ne j}}Cov(X_k,X_j)}\\ {\displaystyle +\frac{1}{d(d-1)}{\displaystyle \sum _{k,l:k\ne l}}Cov(X_k,X_l)}\\ {\displaystyle =Cov(T+{\epsilon }_i,T+{\epsilon }_j)-\frac{1}{d-1}{\displaystyle \sum _{k:k\ne i}}Cov(T+{\epsilon }_k,T+{\epsilon }_i)}\\ {\displaystyle -\frac{1}{d-1}{\displaystyle \sum _{k:k\ne j}}Cov(T+{\epsilon }_k,T+{\epsilon }_j)+\frac{1}{d(d-1)}{\displaystyle \sum _{k,l:k\ne l}}Cov(T+{\epsilon }_k,T+{\epsilon }_l)}\\ {\displaystyle =Var\left(T\right)-\frac{1}{d-1}{\displaystyle \sum _{k:k\ne i}}Var\left(T\right)-\frac{1}{d-1}{\displaystyle \sum _{k:k\ne j}}Var\left(T\right)}\\ {\displaystyle +\frac{1}{d(d-1)}{\displaystyle \sum _{k,l:k\ne l}}Var\left(T\right)=Var\left(T\right)-Var\left(T\right)-Var\left(T\right)+Var\left(T\right)=0.}\end{array}}\end{array}}$$

This completes the proof.