Estimating the Effect of a Predictor Measured by Two Informants on a Continuous Outcome: A Comparison of Methods

Abstract

Investigators sometimes use information about a given variable obtained from multiple informants. We focus on estimating the effect of a predictor on a continuous outcome, when that predictor cannot be observed directly but is measured by two informants. We describe various approaches to using information from two informants to estimate a regression or correlation coefficient for the effect of the (true) predictor on the outcome. These approaches include methods we refer to as single informant, simple average, optimal weighted average, principal components analysis, and classical measurement error. Each of these five methods effectively uses a weighted average of the informants' reports as a proxy for the true predictor in calculating the correlation or regression coefficient. We compare the performance of these methods in simulation experiments that assume a rounded congeneric measurement model for the relationship between the informants' reports and a true predictor that is a mixture of zeros and positively-distributed continuous values. We also compare the methods' performance in a real data example -the relationship between vigorous physical activity (the predictor) and body mass index (the continuous outcome). The results of the simulations and the example suggest that the simple average is a reasonable choice when there are only two informants.

Investigators interested in measuring behavioral or psychological constructs, especially in children or adolescents, often collect information from multiple informants or sources. For example, investigators studying children may obtain reports from parents, teachers, and the child himself if he is old enough. The resulting multiple informant data are generally discrepant. In meta-analyses examining cross-informant correlations, the mean correlation between child report and other report of child psychopathology was 0.22,¹ and the mean correlations for cross-informant reports of adult psychopathology based on the same instrument ranged between 0.40 and 0.70.² Discrepancies between informants are thought to reflect random measurement error, systematic measurement bias, and the different perspectives of the informants (e.g., home versus school). In fact, discrepancy is the very reason for seeking reports from multiple sources: when information from multiple informants is combined, it is believed that different contexts and perspectives will be represented.¹

A considerable amount of research has focused on using multiple informant data as outcomes^3–5, but we focus here on using them as predictors.⁶ More specifically, we focus on the situation where it is assumed that there is an unobserved variable (the “true predictor”) underlying the informants' reports (which themselves are not perfect or near perfect (“gold-standard”) measures of the underlying variable), and where interest lies in estimating the effect of the true predictor on a continuous outcome. Our focus on this situation is motivated by a real data example⁷ where mother and child reports of the child's daily vigorous physical activity are used to estimate the effect of such physical activity on the child's body mass index (BMI). In situations where interest lies in modeling the effect of the informants' reports themselves on the outcome (e.g., when informants' perceptions of the true predictor are thought to be informative about the outcome⁸), it is common to perform separate regressions or include all the reports simultaneously as predictors in a regression model.⁶ In contrast, modeling the effect of the true predictor on the outcome is more complicated because the multiple informants' reports must be combined into a single measure of the true predictor. Further, the reports will ideally be combined in a way that corrects for measurement error. It is well known that failing to correct for measurement error in predictors can result in biased estimates and, when there are other predictors, invalid confidence intervals that do not have the nominal coverage properties.⁹

We compare five ways to combine multiple informant data to estimate a linear regression or correlation coefficient for the effect of the true predictor on the continuous outcome. First, we describe in detail how to implement these approaches. Then, we describe simulation experiments that compare the performance of these approaches in terms of bias and mean squared error in situations where the true predictor is a mixture of zeros and continuously-distributed positive values, as in the physical activity and BMI example. (This type of predictor is a commonly encountered example of a “semicontinuous” variable.) Finally, we compare the estimates produced by these approaches for the aforementioned physical activity and BMI example.

Overview of Approaches

We assume the following model for the relationship of interest:

$$Y_i=\alpha +\beta T_i+{\epsilon }_i\text{for}i=1,\dots ,n,$$ (1)

where T_i is the true predictor and Y_i is the continuous outcome for the i^th individual, and the ε_is are independently and identically distributed with mean 0 and constant variance ${\sigma }_{\epsilon }^2$. In some instances, the parameter of interest will be β. In the example of vigorous physical activity and BMI, we might be interested in the expected change in BMI for one extra hour of vigorous physical activity. In other instances, the parameter of interest will be ρ, the (Pearson) correlation coefficient between Y and T. For example, if Y and T are not measured in meaningful units, ρ might be preferred because it is scale-invariant.

However, because T cannot be observed directly, information about β or ρ must be inferred from the relationship between Y and the informants' reports on T. The true predictor T_i is measured with error by each of J (here, 2) informants, and the resulting measurements of (or reports on) the true predictor are referred to as X_i1,…, X_iJ. Figure 1 illustrates the relationship between the outcome, the true predictor, and the multiple informants' reports, and also introduces the concept of a gold standard measurement (a measurement of the true predictor with minimal measurement error, which we refer to as G_i) because it is relevant to the physical activity and BMI example.

Figure 1.

Illustration of Key Terminology. Variables that appear in black are observed, and variables that appear in gray are unobserved. The effect of interest is the effect of T on Y. This effect can be described by β or ρ, which are the parameters of interest.

We now describe some of the approaches commonly used to estimate β or ρ from the Xs and Ys. (Estimates of ρ and β will be referred to as $\widehat{\rho }$ and $\widehat{\beta }$, respectively.) We refer to each approach either by its popular name or by a name that we have chosen to be reasonably descriptive. The approaches all amount to using a weighted average of the Xs (with weights that may not necessarily sum to one) as a proxy for T in calculating β or ρ. For method M, $\widehat{\beta }$ thus takes the form

$${\widehat{\beta }}_M=\frac{{\Sigma }_{i=1}^n(Y_i-\stackrel{‒}{Y})(X_i^w-{\stackrel{‒}{X}}^w)}{{\Sigma }_{i=1}^n{(X_i^w-{\stackrel{‒}{X}}^w)}^2}$$ (2)

and $\widehat{\rho }$ takes the form

$${\widehat{\rho }}_M=\frac{{\Sigma }_{i=1}^n(Y_i-\stackrel{‒}{Y})(X_i^w-{\stackrel{‒}{X}}^w)}{\sqrt{{\Sigma }_{i=1}^n{(Y_i-\stackrel{‒}{Y})}^2}\cdot \sqrt{{\Sigma }_{i=1}^n{(X_i^w-{\stackrel{‒}{X}}^w)}^2}}$$ (3)

where $X_i^w=w_1^M{X}_{i1}+w_2^M{X}_{i2}$, with $w_1^M$ and $w_2^M$ being method-specific weights. There are a number of ways to determine the “best” weights for the two informants,^10,11 as illustrated by the following methods.

In the single informant (SI) method, information from only one informant is used, typically because that informant is thought to provide information about T that is more valid or less often missing. Supposing without loss of generality that Informant 1 is the preferred informant, then ${\widehat{\beta }}_{SI}$ and ${\widehat{\rho }}_{SI}$ can be estimated by using Equations (2) and (3), respectively, with $w_1^{SI}=1$ and $w_2^{SI}=0$.

In the simple average (SA) method, information from both informants is averaged using equal weights: ${\widehat{\beta }}_{SA}$ and ${\widehat{\rho }}_{SA}$ are estimated by using Equations (2) and (3), respectively, with $w_1^{SA}=w_2^{SA}=1∕2$. The simple average method is easy to implement and has been used in a variety of applications, e.g., reference 12.

In what we refer to as the optimal weighted average (OWA) method, the weights used to calculate ${\widehat{\beta }}_{\mathit{OWA}}$ and ${\widehat{\rho }}_{\mathit{OWA}}$ minimize the sum of squared errors from the linear regression (or identically, maximize the absolute value of the correlation coefficient) for Y and a weighted average of the Xs:

$$w^{OWA}=\underset{w_1,w_2}{\text{arg max}}\sum _{i=1}^n-{[Y_i-{\widehat{\alpha }}^w-{\widehat{\beta }}^w(w_1{X}_{i1}+w_2{X}_{i2})]}^2$$ (4)

where ${\widehat{\alpha }}^w$ and ${\widehat{\beta }}^w$ are estimates of the intercept and slope from the linear regression of Y_i on (w₁X_i1 + w₂X_i2) and where w₂ = 1 − w₁.

In the principal components analysis (PCA) method, the first principal component from the multiple informant data is used as the predictor. This method, along with the factor analysis method (see the Conclusions section below), is advocated by Kraemer and colleagues for use with three or more multiple informants carefully selected to cover multiple perspectives and contexts.¹³ In practice, however, principal components analysis is often used with any two or more informants. Principal components analysis is a eigen-decomposition of the covariance (or correlation) matrix for the data, in this case the n × 2 matrix containing the information from the multiple informants. The first principal component (or “score”) is calculated by multiplying the centered data matrix by the “loadings,” the scaled eigenvector corresponding to the largest eigenvalue from the eigen-decomposition. The eigenvector is usually scaled so that its squared elements sum to one, but here we re-scale it so that its (un-squared) elements sum to one in order to preserve the scale of the predictor construct. Because applying the principal components analysis method to the correlation matrix is identical to the simple average method when there are only two informants, we focus on the principal components analysis method as applied to the covariance matrix. In that case, ${\widehat{\beta }}_{\mathit{PCA}}$ and ${\widehat{\rho }}_{\mathit{PCA}}$ are estimated by using Equations (2) and (3), respectively, with $w_1^{\mathit{PCA}}=1∕(1+v)$ and $w_2^{\mathit{PCA}}=v∕(1+v)$, where

$$v=\frac{{\mathrm{SS}}_{X2}-{\mathrm{SS}}_{X1}+\sqrt{4{\left[\sum _{i=1}^n(X_{i1}-{\stackrel{‒}{X}}_{\cdot 1})(X_{i2}-{\stackrel{‒}{X}}_{\cdot 2})\right]}^2+{({\mathrm{SS}}_{X2}-{\mathrm{SS}}_{X1})}^2}}{2\sum _{i=1}^n(X_{i1}-{\stackrel{‒}{X}}_{\cdot 1})(X_{i2}-{\stackrel{‒}{X}}_{\cdot 2})},$$ (5)

with ${\mathrm{SS}}_{X1}={\Sigma }_{i=1}^n{(X_{i1}-{\stackrel{‒}{X}}_{\cdot 1})}^2$ and ${\mathrm{SS}}_{X2}={\Sigma }_{i=1}^n{(X_{i2}-{\stackrel{‒}{X}}_{\cdot 2})}^2$. (Note that in Equations (2) and (3), we do not center X_i1 and X_i2 before multiplying them by their respective weights, as is usually done in principal components analysis, because doing so has no effect on ${\widehat{\beta }}_{\mathit{PCA}}$ and ${\widehat{\rho }}_{\mathit{PCA}}$.)

The classical measurement error (CME) method is used to handle errors in predictors in environmental and nutritional epidemiology and in econometrics, but also has roots in classical test theory for psychometric tests (see Bollen¹⁴). The method is the same as the simple average method, except that the resulting slope is then divided by an “attenuation factor” (here, λ^CME) to correct for attenuation due to random measurement error. The attenuation factor is derived under the “classical measurement model” for the relationship between the informants' reports and the true predictor, which assumes that the informants' reports equal the true predictor plus an additive random measurement error that has constant variance, σ², across informants (see next section for more on the classical measurement model). The formula for the attenuation factor is:

$${\lambda }^{CME}=\frac{\text{Var}\left(T_i\right)}{\text{Var}\left(T_i\right)+{\sigma }^2∕2}$$

(see Carroll et al.⁹ for the derivation). The parameters Var(T_i) and σ² are unknown and, if only one attenuated measure of T_i were available, would have to be determined based on external information from theory or previous studies. However, because we have more than one measurement of T_i, we can estimate λ^CME using:

$${\widehat{\lambda }}^{CME}=\frac{{\Sigma }_{i=1}^n{({\stackrel{‒}{X}}_{i\cdot }-{\stackrel{‒}{X}}_{\cdot \cdot })}^2-\frac{{\Sigma }_{j=1}^2{\Sigma }_{i=1}^n{(X_{ij}-{\stackrel{‒}{X}}_{i\cdot })}^2}{2}}{{\Sigma }_{i=1}^n{({\stackrel{‒}{X}}_{i\cdot }-{\stackrel{‒}{X}}_{\cdot \cdot })}^2},$$

where $\stackrel{‒}{X}..=\frac{1}{2n}{\Sigma }_{j=1}^2{\Sigma }_{i=1}^n{X}_{ij}$ and ${\stackrel{‒}{X}}_i.=\frac{1}{2}{\Sigma }_{j=1}^2{X}_{ij}$.¹⁵ Then, ${\widehat{\beta }}_{\mathit{CME}}$ can be estimated by using Equation (2) with $w_1^{\mathit{CME}}=w_2^{\mathit{CME}}=\frac{1}{2{\widehat{\lambda }}^{\mathit{CME}}}$, and

$${\widehat{\rho }}_{CME}={\widehat{\beta }}_{CME}\cdot \sqrt{\frac{{\Sigma }_{i=1}^n{({\stackrel{‒}{X}}_{i\cdot }-{\stackrel{‒}{X}}_{\cdot \cdot })}^2-\frac{{\Sigma }_{j=1}^2{\Sigma }_{i=1}^n{(X_{ij}-{\stackrel{‒}{X}}_{i\cdot })}^2}{2}}{{\Sigma }_{i=1}^n{(Y_i-\stackrel{‒}{Y})}^2}}.$$

In situations that violate the assumptions of the classical measurement model, ${\widehat{\lambda }}_{\mathit{CME}}$ can be very small (in which case ${\widehat{\rho }}_{\mathit{CME}}$ may be greater than 1 in magnitude) or can even be negative (in which case the classical measurement error method fails because ${\widehat{\beta }}_{\mathit{CME}}$ and ${\widehat{\rho }}_{\mathit{CME}}$ should not and cannot, respectively, be calculated).

It is possible to analytically compare some, but not all, of the aforementioned estimators for β. For example, it is easy to show that

$$\mid {\widehat{\beta }}_{SA}\mid \le \mid {\widehat{\beta }}_{CME}\mid$$ (6)

(see eAppendix 1A [http://links.lww.com] for a proof).

Simulation Experiments

Setup

We conducted simulation experiments to compare how the five approaches perform in terms of bias and mean squared error (MSE) for multiple informant data generated from a rounded version of the congeneric measurement model first described by Joreskog.¹⁶ This model for the relationship between the multiple informants' reports and the true predictor takes the form:

$$X_{ij}=\{\begin{array}{cc}{\delta }_j+\mathrm{E}{T}_i+{\lambda }_j(T_i-\mathrm{E}{T}_i)+{∊}_{ij}\hfill & \mathrm{if}{\delta }_j+\mathrm{E}{T}_i+{\lambda }_j(T_i-\mathrm{E}{T}_i)+{∊}_{ij}>0\hfill \\ 0\hfill & \mathrm{if}{\delta }_j+\mathrm{E}{T}_i+{\lambda }_j(T_i-\mathrm{E}{T}_i)+{∊}_{ij}\le 0\hfill \end{array}\phantom{\}}$$ (7)

where i indexes the individuals being studied, j indexes the informant types (e.g., mother = 1, child = 2), and ∈_ij is a random measurement error that comes from a distribution with mean 0 and variance ${\sigma }_j^2$. The parameter ${\sigma }_j^2$ reflects the variation in measurement reliability across informant types. In contrast, the parameters δ_j and λ_j reflect the variation in measurement validity across informant types, with δ_j and λ_j representing, respectively, additive and multiplicative systematic measurement bias. (Note that λ_j reflects only multiplicative bias because T_i is centered by its mean, ET_i, before multiplying by λ_j.) The model in (7) is similar to other models that have been proposed for multiple informant data in the context of psychiatric research¹³ and marketing research,^17,18 except that the above model takes the form of a unidimensional rather than multidimensional factor analysis model and also involves rounding to reflect our focus on a true predictor that is a mixture of zeros and continuously-distributed positive values. Finally, note that the “classical measurement model” is a variant of the model in Equation (7) in which there is no rounding and where δ₁ = δ₂ = 0, λ₁ = λ₂ = 1, and ${\sigma }_1^2={\sigma }_2^2={\sigma }^2$.

In our simulation experiments, we investigated how the bias and MSE of the approaches described above vary with different values of the parameters in the model in (7): −0.50, −0.25, 0, 0.25 or 0.50, for δ₁ and δ₂; 0.40, 0.75, 1, 1.10, and 1.33 for λ₁ and λ₂; and 0.0, 0.1, or 0.5 for ${\sigma }_1^2$ and ${\sigma }_2^2$. We chose these values either because they were realistic based on actual applications¹⁹ of previously-proposed models^17,18 for multiple informant data or because they could aid us in understanding how the other parameters affected results (e.g., ${\sigma }_1^2=0$). We also made the following assumptions about the variables in the measurement model (Equation 7) and regression model (Equation 1): Cor(T_i, ∈_ij) = 0; Cor(∈_i1, ∈_i2) = 0; and Cor(∈_ij, ε_i) = 0 (referred to as non-differential measurement error). Further, we assumed that α = 1, β = 1, and ${\sigma }_{\epsilon }^2=1$, with the value of β based on the physical activity and BMI example. We generated 1000 datasets of size n = 250 for every combination of the above parameter values; then used the single informant method (arbitrarily based on informant 1) and the simple average, optimal weighted average, principal components analysis, and classical measurement error methods (by definition, based on both informants) to calculate $\widehat{\beta }$ and $\widehat{\rho }$ for each dataset; and finally used the resulting estimates and true values of β (= 1) or ρ (= 0.5) to calculate the average bias and average MSE. Readers interested in the details of the simulation experiments can consult eAppendix 1B (http://links.lww.com), which describes the simulation experiments more fully, and can download the code for the simulation experiments (eAppendix 2, http://links.lww.com).

Results

Tables 1 and 2 summarize how changes in the measurement model parameters affect the average percent bias in $\widehat{\rho }$ and $\widehat{\beta }$, respectively, for each method. In addition, readers interested in seeing the method-specific average bias and MSE for each combinations of measurement model parameters can view the figures in eAppendix 1B (http://links.lww.com) or download the results from the simulation experiment (eAppendix 3, http://links.lww.com).

Table 1.

Effect of Rounded Congeneric Measurement Model Parameters on Average Percent Bias in $\widehat{\rho }$

Parameter Valuesa		Range of Average Percent Bias in $\widehat{\rho }$
	Single Informantb	Simple Average	Optimal Weighted Average	Principal Components Analysis	Classical Measurement Error
Defaulta parameter values	0%	0%	0%	0%	0%
δ1 < 0, δ2 < 0	−4% to −2%	−4% to −2%	−4% to −2%	−4% to −2%	−4% to −2%
δ1 > 0, δ2 < 0	0%	0%	0%	0%	6% to 54%d
δ1 > 0, δ2 > 0	0%	0%	0%	0%	0% to 2%
λ1 < 1, λ2 < 1	0%	0%	0%	0%	0% to 6%
λ1 > 1, λ2 < 1	0%	0%	0%	0%	2% to 18%
λ1 > 1, λ2 > 1	0%	0%	0%	0%	0%
${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−32% to −10%	−20% to −4%	−20% to −4%	−20% to −4%	0%
δ1 < 0, δ2 < 0 and λ1 < 1, λ2 < 1	−8% to 0%	−8% to 0%	−8% to 0%	−8% to 0%	−8% to 20%
δ1 > 0, δ2 < 0 and λ1 < 1, λ2 < 1	0%	−2% to 0%	0%	−2% to 0%	18% to >100%c,d
δ1 > 0, δ2 > 0 and λ1 < 1, λ2 < 1	0%	0%	0%	0%	0% to 18%
δ1 < 0, δ2 < 0 and λ1 > 1, λ2 < 1	−4% to −2%	−6% to 0%	−4% to 0%	−4% to −2%	−2% to 48%
δ1 > 0, δ2 < 0 and λ1 > 1, λ2 < 1	0%	0%	0%	0%	14% to >100%c,d
δ1 > 0, δ2 > 0 and λ1 > 1, λ2 < 1	0%	0%	0%	0%	2% to 24%
δ1 < 0, δ2 < 0 and λ1 > 1, λ2 > 1	−4% to −2%	−4% to −2%	−4% to −2%	−4% to −2%	−4% to 0%
δ1 > 0, δ2 < 0 and λ1 > 1, λ2 > 1	0%	0%	0%	0%	2% to 36%
δ1 > 0, δ2 > 0 and λ1 > 1, λ2 > 1	0%	0%	0%	0%	0% to 2%
δ1 < 0, δ2 < 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−32% to −8%	−22% to −6%	−22% to −4%	−22% to −6%	−4% to 2%
δ1 > 0, δ2 < 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−34% to −10%	−22% to −6%	−20% to −4%	−26% to −6%	4% to 86%c,d
δ1 > 0, δ2 > 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−34% to −10%	−22% to −6%	−20% to to −4%	−22% to −6%	0% to 4%
λ1 < 1, λ2 < 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−68% to −16%	−56% to −10%	−54% to −8%	−62% to −10%	0% to 22%c,d
λ1 > 1, λ2 < 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−28% to >%	−32% to −4%	−26% to −4%	−26% to −4%	2% to 26%d
λ1 > 1, λ2 > 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−28% to −4%	−18% to −2%	−16% to −2%	−18% to −2%	8% to 2%

^aParameters δ₁ and δ₂ (additive bias) can equal: −0.5 or −0.25 (< 0), 0 (default value), 0.25 or 0.5 (> 0).

Parameters λ₁ and λ₂ (multiplicative bias) can equal: 0.4 or 0.75 (< 1), 1 (default value), 1.1 or 1.33 (> 1).

Parameters ${\sigma }_1^2$ and ${\sigma }_2^2$ (random error) can equal: 0 (default value), 0.1 or 0.5 (> 0).

Non-specified parameters are equal to default values, which correspond to no measurement bias/error.

^bInformants 1 and 2 are interchangeable for all methods except SI, which uses Informant 1 only.

^cThe classical measurement error method failed for some datasets.

^dThe classical measurement error method produced correlations greater than 1 in magnitude for some datasets.

Table 2.

Effect of Rounded Congeneric Measurement Model Parameters on Average Percent Bias in $\widehat{\beta }$

Parameter Valuesa		Range of Average Percent Bias in $\widehat{\beta }$
	Single Informantb	Simple Average	Optimal Weight Average	Principal Components Analysis	Classical Measurement Error
Defaulta parameter values	0%	0%	0%	0%	0%
δ1 < 0, δ2 < 0	17% to 35%	17% to 35%	17% to 35%	17% to 35%	17% to 35%
δ1 > 0, δ2 < 0	0%	8% to 18%	5% to 7%	7% to 14%	24% to >100%
δ1 > 0, δ2 > 0	0%	0%	−2% to 0%	0%	0% to 5%
λ1 < 1, λ2 < 1	33% to >100%	33% to >100%	33% to >100%	33% to >100%	33% to >100%
λ1 > 1, λ2 < 1	−20% to −7%	0% to 36%	13% to 90%	−6% to 12%	8% to 72%
λ1 > 1, λ2 > 1	−20% to −7%	−20% to −7%	−20% to −7%	−20% to −7%	−20% to −7%
${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−44% to −10%	−24% to −2%	−24% to −2%	−24% to −2%	8% to 22%
δ1 < 0, δ2 < 0 and λ1 < 1, λ2 < 1	50% to >100%	51% to >100%	51% to >100%	51% to >100%	51% to >100%
δ1 > 0, δ2 < 0 and λ1 < 1, λ2 < 1	33% to >100%	42% to >100%	39% to >100%	41% to >100%	96% to >100%c
δ1 > 0, δ2 > 0 and λ1 < 1, λ2 < 1	33% to >100%	33% to >100%	30% to >100%	33% to >100%	34% to >100%
δ1 < 0, δ2 < 0 and λ1 > 1, λ2 < 1	−12% to 23%	11% to 83%	−8% to >100%	4% to 49%	21% to >100%
δ1 > 0, δ2 < 0 and λ1 > 1, λ2 < 1	−25% to −8%	0% to 50%	−7% to 71%	−14% to 10%	33% to >100%c
δ1 > 0, δ2 > 0 and λ1 > 1, λ2 < 1	−26% to −9%	−4% to 34%	−6% to 45%	−11% to 10%	4% to 91%
δ1 < 0, δ2 < 0 and λ1 > 1, λ2 > 1	−11% to 22%	−10% to 22%	−11% to 22%	−10% to 22%	−10% to 22%
δ1 > 0, δ2 < 0 and λ1 > 1, λ2 > 1	−25% to −9%	−18% to 7%	−21% to −3%	−19% to 4%	−13% to >100%
δ1 > 0, δ2 > 0 and λ1 > 1, λ2 > 1	−25% to -−9%	−25% to −9%	−26% to −9%	−25% to −9%	−25% to −5%
δ1 < 0, δ2 < 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−37% to 19%	−15% to 28%	−15 to 28%	−15 to 28%	20% to 52%
δ1 > 0, δ2 < 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−53% to −17%	−26% to 7%	−23% to 18%	−36% to 1%	25% to >100%c
δ1 > 0, δ2 > 0 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−54% to −18%	−35% to −9%	−36% to −9%	−36% to −9%	1% to 16%
λ1 < 1, λ2 < 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−68% to 2%	−39% to 47%	−39% to 47%	−54% to 47%	43% to >100%c
λ1 > 1, λ2 < 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−44% to −15	−24% to 19%	−32% to 6%	−33% to 4%	14% to >100%
λ1 > 1, λ2 > 1 and ${\sigma }_1^2>0$, ${\sigma }_1^2>0$	−45% to −15	−31% to −9%	−31% to −9%	−31% to −9%	−18% to 10%

^aParameters δ₁ and δ₂ (additive bias) can equal: −0.5 or −0.25 (< 0), 0 (default value), 0.25 or 0.5 (> 0).

Parameters λ₁ and λ₂ (multiplicative bias) can equal: 0.4 or 0.75 (< 1), 1 (default value), 1.1 or 1.33 (> 1).

Parameters ${\sigma }_1^2$ and ${\sigma }_2^2$ (random error) can equal: 0 (default value), 0.1 or 0.5 (> 0).

Non-specified parameters are equal to default values, which correspond to no measurement bias/error.

^bInformants 1 and 2 are interchangeable for all methods except SI, which uses Informant 1 only.

^cThe classical measurement error method failed for some datasets.

Regarding ρ, the single informant, simple average, optimal weighted average, and principal components analysis estimates follow similar patterns and are often very similar in value. They are never exaggerated (biased away from 0), but in some situations they are attenuated (biased toward 0). This attenuation bias becomes worse as ${\sigma }_1^2$ (and ${\sigma }_2^2$, for the simple average, optimal weighted average, and principal components analysis methods) increases, and as λ₁ (and λ₂, for the simple average, optimal weighted average, and principal components analysis methods) becomes increasingly less than 1. In contrast, the classical measurement error estimates have at most small (< 10%) attenuation bias, but can become very exaggerated (in some cases, to the point where they are greater than 1) and can even fail. The exaggeration bias becomes worse as δ₁ and δ₂ become increasingly discrepant, and as λ₁ and λ₂ become increasingly discrepant or become increasingly less than 1. As for comparisons between methods, the optimal weighted average method performs better than (in terms of a smaller magnitude of bias of $\widehat{\rho }$) or at least as well as the single informant, simple average, and principal components analysis methods for all combinations of parameters. Optimal weighted average also performs better than classical measurement error for some combinations of parameters, especially when δ₁ and δ₂ differ considerably or λ₁ and λ₂ differ considerably. However, classical measurement error performs considerably better than optimal weighted average (and, by extension, the single informant, simple average, and principal components analysis methods) for other combinations of parameters, especially when the difference between δ₁ and δ₂ is small and both ${\sigma }_1^2$ and ${\sigma }_2^2$ are large.

Turning to β, the estimates from all five methods generally have a greater magnitude of bias than the corresponding estimates of ρ. The single informant, simple average, optimal weighted average, and principal components analysis estimates of β again follow a similar pattern but, unlike the corresponding estimates of ρ, can be either exaggerated or attenuated. The exaggeration bias becomes worse as δ₁ (and δ₂, for the simple average, optimal weighted average, and principal components analysis methods) becomes increasingly less than 0, and as λ₁ (and λ₂, for the simple average, optimal weighted average, and principal components analysis methods) becomes increasingly less than 1. The attenuation bias becomes worse as λ₁ (and λ₂, for the simple average, optimal weighted average, and principal components analysis methods) becomes increasingly more than 1, and as ${\sigma }_1^2$ (and ${\sigma }_2^2$, for the simple average, optimal weighted average, and principal components analysis methods) increases. The classical measurement error estimates of β, like the corresponding estimates of ρ, have at most moderate attenuation bias (< 25%), but can become very exaggerated or even fail. The exaggeration bias becomes worse as δ₁ and δ₂ become increasingly discrepant or increasingly less than 0, as λ₁ and λ₂ become increasingly less than 1, and as ${\sigma }_1^2$ and ${\sigma }_2^2$ increase. The attenuation bias becomes worse as λ₁ and λ₂ become increasingly more than 1. As for comparisons between methods, each method outperforms all of the other methods for some combinations of parameters, as can be seen in Table 3.

Table 3.

Best-performing^a Method(s) for Various Combinations of Rounded Congeneric Measurement Model Parameters

Parameter Values	Single Informantb	Simple Average	Optimal Weight Average	Principal Components Analysis	Classical Measurement Error
δ1 = −0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.0	X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.0			X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$ σ2 = 0.0	X	X	X	X	X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0				X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$ σ2 = 0.0				X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = 0; δ2 = 0; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.0			X
δ1 = 0; δ2 = 0; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.5	X
δ1 = 0; δ2 = 0; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.5					X
δ1 = −0.50; δ2 = −0.50; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = 0.50; δ2 = −0.50; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0	X
δ1 = −0.50; δ2 = 0.50; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0			X
δ1 = 0.50; δ2 = 0.50; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = −0.50; δ2 = −0.50; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0	X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0			X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0	X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.0				X
δ1 = −0.50; δ2 = −0.50; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0			X
δ1 = 0.50; δ2 = −0.50; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$ σ2 = 0.0				X
δ1 = −0.50; δ2 = 0.50; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$ = 0.0; σ2 = 0.0			X
δ1 = 0.50; δ2 = 0.50; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0				X
δ1 = −0.50; δ2 = −0.50; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0		X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0	X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.0	X	X	X	X	X
δ1 = −0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.0				X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$ σ2 = 0.0		X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.0			X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.0			X
δ1 = −0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.5				X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.5	X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.5		X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.0$; σ2 = 0.5	X		X
δ1 = −0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.5		X	X
δ1 = 0.50; δ2 = −0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.5			X
δ1 = −0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.5			X
δ1 = 0.50; δ2 = 0.50; λ1 = 1.00; λ2 = 1.00; ${\sigma }_1^2=0.5$; σ2 = 0.5					X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.5$; σ2 = 0.0		X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.5$; σ2 = 0.0		X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.5$; σ2 = 0.0		X		X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.5$ σ2 = 0.0					X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.5		X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.0$; σ2 = 0.5		X		X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.5		X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.0$; σ2 = 0.5					X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 0.40; ${\sigma }_1^2=0.5$ σ2 = 0.5		X	X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 0.40; ${\sigma }_1^2=0.5$; σ2 = 0.5		X
δ1 = 0; δ2 = 0; λ1 = 0.40; λ2 = 1.33; ${\sigma }_1^2=0.5$; σ2 = 0.5		X
δ1 = 0; δ2 = 0; λ1 = 1.33; λ2 = 1.33; ${\sigma }_1^2=0.5$; σ2 = 0.5					X

^aX indicates method with smallest magnitude of bias for given parameters. Multiple Xs per row indicate ties.

^bInformants 1 and 2 are interchangeable for all methods except single informant, which uses Informant 1 only.

Vigorous Physical Activity and Body Mass Index Example

To illustrate the five approaches, we consider a validation study²⁰ used to design a larger study⁷ of the relationship between physical activity and obesity in children in two towns of Mexico City in 1996. In the validation study, which included 114 ten- to fourteen-year-old students, two informants (child and mother) completed a questionnaire designed to assess the child's physical activity and inactivity. Physical activity was also assessed by 24-hour recall, which was provided by the child in an interview with a trained nutritionist on three separate days; the three measurements of physical activity were averaged, with different weights for weekend days versus weekdays. Here, we focus on hours of daily vigorous physical activity (T_i) as a predictor for body mass index (Y_i). We treat the weighted average of the 24-hour recall measurements of vigorous physical activity as the gold standard measure of vigorous physical activity (G_i). We treat the mother reports of vigorous physical activity as X_i1 and the child reports of vigorous physical activity as X_i2, and we use them to estimate β and ρ. For the sake of simplicity, we do not include any of the available covariates (e.g., child sex, age, etc.) in our analyses.

We focus on cases with complete data (n = 81). Figure 2 displays frequency plots for 24-hour recall and mother and child report of vigorous physical activity, as well as scatterplots of BMI versus those measures. For 24-hour recall, mother report, and child report, the means (variances) are 0.48 (0.40), 0.80 (0.46), and 1.00 (0.91), respectively, and the percentage of values that are 0 are 43%, 9%, and 1%, respectively. Both mother and child, and especially the child, overestimate vigorous physical activity relative to 24-hour recall. The Pearson correlations of the mother and child report with 24-hour recall are 0.41 and 0.24, respectively, and the Pearson correlation of the mother and child report with each other is 0.23. These numbers suggest that the mother report may better capture the relationship between vigorous physical activity and BMI.

Figure 2.

Frequency and Scatterplots (Versus Body Mass Index (BMI)) for the 24-Hour Recall Gold Standard Measurement and Mother and Child Reports of Vigorous Physical Activity.

Table 1 displays estimates of ρ and β from all five methods applied to the mother and child reports of vigorous physical activity, as well as estimates of ρ and β based on 24-hour recall of vigorous physical activity, the gold standard measurement. The estimates reveal that the optimal weighted average method (which gives weights of approximately 3/4 and 1/4 to the mother report and child report, respectively) produces estimates identical to the gold standard estimates of both ρ and β. The single informant, simple average, and principal components analysis methods all underestimate both ρ and β; the next-closest estimates to the gold standard estimates are given by the simple average method (which, of course, weights both reports equally), followed by the single informant method based on mother report only, then the principal components analysis method (which gives weights of approximately 1/4 and 3/4 to the mother report and child report, respectively), and then the single informant method based on child report only. Last, the classical measurement error method overestimates both ρ and β by a considerable amount. These results show that the mother report is a better proxy for 24-hour recall of vigorous physical activity in terms of ability to predict BMI, but that combining mother report with child report (via the optimal weighted average or simple average methods) produces better estimates than using mother report alone.

Conclusions

We have described five methods of estimating the regression or correlation coefficient for the effect of a predictor on a continuous outcome, when that predictor cannot be observed directly but is reported on by two informants. We then compared the performance of these methods in situations where the true predictor is a mixture of zeros and continuously-distributed positive values.

Regarding the correlation coefficient, the simulation experiments suggest that estimates obtained by the single informant, simple average, optimal weighted average, and principal components analysis methods become attenuated when random error is present – unsurprisingly given that these methods do not correct for attenuation due to random measurement error. However, these estimates are relatively unaffected by additive or multiplicative measurement bias on their own. (Without the rounding in the congeneric measurement model, the estimates would be completely unaffected by additive or multiplicative bias.) In contrast, the classical measurement error method assumes that the measurements contain random error, but not additive or multiplicative bias. Unsurprisingly, then, the classical measurement error estimates become exaggerated (or even fail) when the additive or multiple bias differs between informants, because the second term in the numerator of ${\lambda }^{\widehat{C}ME}$ becomes too large. Also unsurprisingly, then, the classical measurement error estimates are unaffected by random error on its own since the method corrects for attenuation due to random error. In terms of comparisons between methods, the optimal weighted average estimates are never more biased than the single informant, simple average, and principal components analysis estimates, which is to be expected given that the optimal weighted average estimates weight the two informants optimally. For instance, in the physical activity and BMI example, the optimal weighted average estimates give more weight to the better informant (the mother), in contrast to the simple average estimates, which weights both informants equally, and the principal components analysis estimates, which give more weight to the worse informant (the child) because it has more variance. The single informant estimate (based on mother) gives all the weight to the better informant, but suffers more from attenuation due to random measurement error because it is based on only one informant. Finally, the optimal weighted average estimates can be more biased than the classical measurement error estimates when the random error is large and the additive and multiplicative bias do not differ greatly between informants.

Estimates of the regression coefficients are typically more biased than corresponding estimates of the correlation coefficients. The effects of random measurement error and additive and multiplicative measurement bias on the estimates are similar to those described in the preceding paragraph, with two exceptions. First, for all the methods, upward or downward multiplicative bias results in attenuated or exaggerated estimates, respectively, because multiplicative bias changes the scale of the predictor. Second, for the classical measurement error method, random error on its own results in small-to-moderate exaggeration in the estimates of the regression coefficients, which is due to the rounding in the congeneric measurement error model. (Without the rounding, the classical measurement error estimates would be unbiased.) In terms of comparisons among methods, each method performs best in some situations.

Overall, when there are only two informants, the simple average method is a reasonable choice. The simple average method, although not always optimal, rarely performs much worse than the single informant, optimal weighted average, and principal components analysis methods, and often performs similarly to the most optimal of those methods, as in the physical activity and BMI example and as found in previous studies.²¹ Further, with the simple average method, it is straightforward to compare results across samples because this method uses the same (equal) weights for informants in every sample. Although the classical measurement error method performs better than the other methods (including the simple average method) when there is a large amount of random error and little difference in additive or multiplicative bias between informants, it should be avoided in other situations because the estimate can be very exaggerated (as in the physical activity and BMI example).

Of course, it is preferable to have more than two informants, or validation data, because this allows the use of more sophisticated methods such as latent variable models (e.g., factor analysis) or extensions of the classical measurement error method that allow for measurement bias or correlated measurement errors (see Cheng and Van Ness¹⁵).

Table 4.

Results for Vigorous Physical Activity as a Predictor of Body Mass Index

	$\widehat{\rho }$	$\widehat{\beta }$
Estimate based on 24-hour recall of vigorous physical activity:	−0.18	−1.0
Estimates based on mother and child report of vigorous physical activity:
Single informant method (mother report only)	−0.17	−0.86
Single informant method (child report only)	−0.11	−0.39
Simple average method (both reports)	−0.17	−0.90
Optimal weighted average method (both reports)	−0.18	−1.0
Principal components analysis method (both reports)	−0.13	−0.59
Classical measurement error method (both reports)	−0.28	−2.8