A New Functional Representation of Broad Sense Agreement

Abstract

We derive a new functional representation of Broad Sense Agreement (BSA) index that evaluates the agreement/alignment between a continuous measurement and an ordinary measurement. Using this result, we develop an alternative BSA estimator, which can offer significant numerical advantages.

1. Introduction

In Biomedical research, investigators often encounter the problem of evaluating the agreement between different instruments or the measurements from different raters. Various methods have been developed for assessing the agreement between categorical measurements (Cohen, 1960; Fleiss, 1971; Williamson et al., 2000; Guo and Manatunga, 2009, for example) or between continuous measurements (Lawrence and Lin, 1989; Lin et al., 2002; Koo and Li, 2016). Extensions to different data structures, such as replicated samples, repeated measures, survival outcomes and multivariate observations have also been studied (Lin et al., 2007; Quiroz, 2005; Guo and Manatunga, 2007, for example). However, these methods cannot be used to measure the alignment between a continuous measurement and an ordinal measurement.

To fill this gap, Peng et al. (2011) proposed the concept of broad sense agreement (BSA), which was designed to characterize the capability of identifying a set of cut-points of the continuous measurements that produces an exact one-to-one correspondence with the ordinal measurement. A BSA index was defined accordingly and a nonparametric estimator was derived based on the idea of stratified sampling. However, the calculation of Peng et al. (2011)’s BSA estimator involves all possible sampling of the continuous measurement stratified by the ordinal measurement. As a result, this procedure can be computationally intensive when the sample size is moderately large, for example 100 or 200, or the ordinal measurement has more than a few levels, for example 4 or more.

In this article, we propose a new representation of the BSA index as a functional of the survival functions of the continuous measurement stratified by the ordinal measurement. This representation allows us to derive an alternative estimator of the BSA index through plugging in the empirical estimator of the conditional survival function of the continuous measure given the ordinal measurement. Furthermore, we reveal that this new estimator coincides with the original nonparametric estimator proposed by Peng et al. (2011). As shown in our simulation studies, Peng et al. (2011)’s estimator and the new estimator are identical based on all simulated datasets.

More importantly, the new estimator is simple and quick to compute, even for large sample sizes. As shown by our numerical studies in Section 3, Peng et al. (2011)’s estimator, which involves exhausted stratified sampling, can be computationally prohibitive with moderately large sample sizes, such as 240. In that case, calculating the new estimator only takes a few seconds, which is about 9,000 times faster than the original BSA estimator. Given such a dramatic computational gain, our new estimator should replace Peng et al. (2011)’s estimator in real applications. The associated numerical benefit is expected to significantly enhance the practical utility of the BSA methodology.

2. The Proposed Method

2.1. A review of BSA framework

The concept of BSA was first introduced by Peng et al. (2011). Let X and Y denote a continuous measurement and an ordinal measurement of an outcome from the same object respectively. Let ${\mathcal{D}}_X$ and ${\mathcal{D}}_Y$ denote the domains of X and Y respectively. By the definition of Peng et al. (2011), X and Y are in perfect broad sense agreement (or disagreement) if and only if there exists an increasing (or decreasing) step function Ψ from ${\mathcal{D}}_X$ to ${\mathcal{D}}_Y$ such that Y = Ψ (X) with probability 1. By this definition, perfect broad sense agreement (or disagreement) can be interpreted as the existence of a set of cut-off points in the continuous X that produces an exact one-to-one concordance (or discordance) between the discretized X and Y.

Taking ${\mathcal{D}}_Y$ = {1, …, L}, Peng et al. (2011) proposed a BSA index,

$${\rho }_{bsa}=1-\frac{E\left\{{\sum }_{l=1}^L{\left(l-R_l\right)}^2\right\}}{E\left\{{\sum }_{l=1}^L{\left(l-R_l\right)}^2|X⫫Y\right\}},$$ (1)

where (R₁, ⋯, R_L) denote the ranks of {X_(*1), ⋯, X_(*L)}, with X_(*l) denoting a randomly selected X with Y = l, l = 1, …, L. The notation E(·|X ⫫ Y) represents expectation under the assumption that X and Y are independent. The value of ρ_bsa is between −1 and 1 with ρ_bsa = 1 (or −1) corresponding to perfect agreement (or disagreement). A larger magnitude of ρ_bsa indicates better interpretability of the continuous X according to the ordinal Y.

Denote the observed data, n replicates of (X, Y), by ${\left\{\left(X_i,Y_i\right)\right\}}_{i=1}^n$. Arrange the observed data in the sequence started with (X_i, Y_i = 1)’s, followed by (X_i, Y_i = 2)’s, …, and ended with (X_i, Y_i = L)’s. Peng et al. (2011) proposed a nonparametric estimator of ρ_bsa,

$${\widehat{\rho }}_{bsa}=1-\frac{{\left({\Pi }_{l=1}^L{n}_l\right)}^{-1}{\sum }_{j_1=1}^{n_1}\cdots {\sum }_{j_{L=1}}^{n_L}{{\sum }_{l=1}^L\left\{l-{\sum }_{r=1}^{L}I({\check{X}}_{l,j_i}\ge {\check{X}}_{r,j_r})\right\}}^2}{\left(L^3-L\right)/6}.$$ (2)

where ${\check{X}}_{s,t_s}=X_{{\sum }_{l=1}^{s-1}{n}_l+t_s}$ for 1 ≤ t_s ≤ n_s with $n_l={\sum }_{i=1}^{n}I\left(Y_i=l\right)$. It is easy to see that calculating ${\widehat{\rho }}_{bsa}$ involves exhaustive stratified sampling of {X_(*1), ⋯, X_(*L)}, and thus can be computationally intensive.

2.2. A functional representation of BSA index and the proposed estimator

Let S_l(x) = Pr(X > x|Y = l) denote the conditional survival function of X given Y = l. In the following theorem, we show that the ρ_bsa in equation (1) can be equivalently expressed as a functional of {S₁(·), ⋯, S_L(·)}.

Theorem 1. Suppose S_l(x) is continuously differentiable and strictly decreasing in x for l = 1, …, L. We have

$${\rho }_{bsa}=\frac{2{\sum }_{r=1}^{L-1}{\sum }_{l=r+1}^L\{-(l-r){\int }_{-\infty }^{\infty }{S}_l(x)d{S}_r(x)\}}{\left(L^3-L\right)/6}-1$$ (3)

The detailed proof of Theorem 1 is relegated to the Supplementary Materials.

2.3. Estimation

Based on Theorem 1, we propose a nonparametric estimator of BSA by plugging in the empirical estimators of the conditional survival functions, S₁(·), ⋯, S_l(·). The proposed estimator the form,

$${\check{\rho }}_{bsa}=\frac{2{\sum }_{r=1}^{L-1}{\sum }_{l=r+1}^L\{-(l-r){\int }_{-\infty }^{\infty }{\widehat{S}}_l(x)d{\widehat{S}}_r(x)\}}{\left(L^3-L\right)/6}-1,$$ (4)

where ${\widehat{S}}_l(x)\doteq n_l^{-1}{\sum }_{i=1}^{n}I\left(X_i>x,Y_i=l\right)$ is the empirical conditional survival function of X given Y = l, l = 1, ⋯, L.

In the following theorem, we establish the equivalence between ${\check{\rho }}_{bsa}$ and Peng et al. (2011)’s estimator ${\widehat{\rho }}_{bsa}$.

Theorem 2. Suppose $\text{Pr}\left({\cup }_{0\le r<l\le L}\left\{X_{(*r)}=X_{(*l)}\right\}\right)=0$. It holds that

$${\widehat{\rho }}_{bsa}={\check{\rho }}_{bsa}$$

with probability one.

The proof of Theorem 2 is provided in the Supplementary Materials.

Given the equivalence between ${\check{\rho }}_{bsa}$ and ${\widehat{\rho }}_{bsa}$, it immediately follows from the theoretical properties of ${\widehat{\rho }}_{bsa}$ (Peng et al., 2011) that the proposed estimator ${\check{\rho }}_{bsa}$ is consistent and asymptotic normal. To estimate the variance of ${\check{\rho }}_{bsa}$, we can similarly use the jackknife method adopted by Peng et al. (2011). A variance estimator takes the form,

$$\widehat{V}\left({\check{\rho }}_{bsa}\right)=\frac{n-1}{n}\sum _{i=1}^n{\left({\check{\rho }}_{bsa}^{-i}-n^{-1}\sum _{k=1}^n{\check{\rho }}_{bsa}^{-k}\right)}^2,$$

where ${\check{\rho }}_{bsa}^{-i}$ denotes the proposed estimator ${\check{\rho }}_{bsa}$ obtained from the data excluding (X_i, Y_i). To construct confidence intervals of ρ_bsa, we impose Fisher-z transformation (Fisher, 1921), ϕ (·), to address the distributional skewness when ${\check{\rho }}_{bsa}$ is close to the boundary −1 or 1. That is, a confidence interval of level 1 – α for ρ_bsa may be given by

$$\left({\varphi }^{-1}\left(\varphi \left({\check{\rho }}_{bsa}\right)-z_{1-\alpha }\dot{\varphi }\left({\check{\rho }}_{bsa}\right){\left\{\widehat{V}\left({\check{\rho }}_{bsa}\right)\right\}}^{1/2}\right),{\varphi }^{-1}\left(\varphi \left({\check{\rho }}_{bsa}\right)+z_{1-\alpha }\dot{\varphi }\left({\check{\rho }}_{bsa}\right){\left\{\widehat{V}\left({\check{\rho }}_{bsa}\right)\right\}}^{1/2}\right)\right),$$

where z_1−α denotes the 100(1 − α)th percentile of N(0, 1) and ϕ⁻¹(·) denotes the inverse function of ϕ(·).

In the presence of ties, suppose ${\check{X}}_{i_1,j_{i_1}}={\check{X}}_{i_2,j_{i_2}}=\cdots ={\check{X}}_{i_m,j_{i_m}}$ are tied observations with i₁ < … < i_m. If we rank them in the order of ${\check{X}}_{i_1,j_{i_1}},{\check{X}}_{i_2,j_{i_2}},\cdots ,{\check{X}}_{i_m,j_{i_m}}$, by the proof of Theorem 2, we can show that ${\widehat{\rho }}_{bsa}={\check{\rho }}_{bsa}$.

It is remarkable that the proposed estimator ${\check{\rho }}_{bsa}$ is easy to compute, and does not involve the exhausted stratified sampling that is required by ${\widehat{\rho }}_{bsa}$. Thus using ${\check{\rho }}_{bsa}$ can dramatically relieve the computation burden that may be encountered when calculating ${\widehat{\rho }}_{bsa}$ with moderately large n or L greater than 3.

3. Numerical studies

We conducted Monte Carlo simulation studies with n = 80, 120 or 240 and L = 3 or 4. The ordinal Y was generated from {1, ⋯, L} with equal probability, and the continuous X was generated from Y + Normal (0, 0.4²). For each simulation setting, we simulated 400 datasets.

Based on all simulated datasets, ${\check{\rho }}_{bsa}$ calculated by equation (4) coincides with ${\widehat{\rho }}_{bsa}$ calculated by equation (2). In Table 1, we present the empirical biases (EmpBias) and average estimated standard deviations (EstSD) of ${\check{\rho }}_{bsa}$, along with the empirical coverage probabilities (Cov95) of the 95% confidence intervals based on ${\check{\rho }}_{bsa}$. The results in Table 1 show that ${\check{\rho }}_{bsa}$ is virtually unbiased. The empirical coverage probabilities are close to the nominal level 95%, indicating satisfactory finite sample performance of the jackknife-based inferences.

Table 1:

Simulation results: Empirical biases (EmpBias), average estimated standard error (EstSD), and empirical coverage probabilities of the proposed estimator ${\check{\rho }}_{bsa}$ as well as the computation time (in seconds) of ${\check{\rho }}_{bsa}$ and that of Peng et al. (2011)’s estimator ${\widehat{\rho }}_{bsa}$.

L*	n†	EmpBias	EstSD	Cov95	${\widehat{\rho }}_{bsa}\left(\text{seconds}\right)$	${\check{\rho }}_{bsa}\left(\text{seconds}\right)$
3	80	0.001	0.016	95.8	23.1	1.1
3	120	−0.001	0.014	95.3	113.3	1.8
3	240	−0.0003	0.009	94.3	2643.0	4.7
4	80	−0.0002	0.009	96.5	168.6	2.0
4	120	−0.001	0.007	96.5	2557.1	2.8
4	240	−0.001	0.005	94.5	63072.0	7.2

^*:number of categories of Y;

^†:sample size.

We also compared the average computation time (in seconds) between ${\check{\rho }}_{bsa}$ and ${\widehat{\rho }}_{bsa}$ under each simulation set-up. The saving in computation time associated with the new estimator ${\check{\rho }}_{bsa}$ is substantial; the computation time of the original BSA estimator ${\widehat{\rho }}_{bsa}$ is at least 23 times that of ${\check{\rho }}_{bsa}$. It is apparent that the reduction in computation time increases dramatically as n or L increases. For example, when L = 4 and n = 240, the computation time of ${\widehat{\rho }}_{bsa}$ is about 9,000 times that of ${\check{\rho }}_{bsa}$. As explained in Section 2, the dramatic computational saving is mainly because of the functional representation derived for ρ_bsa.

The computational advantage would be further heightened for the jackknife variance estimation, which requires calculating the BSA estimators n times. For example, when L = 4 and n = 240, the computation of one ${\widehat{\rho }}_{bsa}$ takes about 17.5 hours (see Table 1), and this leads to an estimate of 75,584 hours for computing the variance estimate for ${\widehat{\rho }}_{bsa}$, which is computationally prohibitive. In contrast, we expect to obtain the variance estimate for ${\check{\rho }}_{bsa}$ within half an hour. This example clearly indicates the significant numerical value of the new functional representation of BSA index derived in this work.

We analyzed a dataset from the Diabetes and Depression Study (Moussavi et al., 2007) to evaluate the BSA between the graded depression severity by the Mini International Neuropsychiatric Interview (MINI) (no depression: n₁ = 737; mild depression: n₂ = 150; severe depression: n₃ = 126) and the self-reported score from Zung depression scale (mean= 45.6, median= 43.8). The detailed data background can be found in Qiu et al. (2019). Our analysis yielded an estimate of ${\widehat{\rho }}_{bsa}={\check{\rho }}_{bsa}=0.663$ with the standard error, 0.022, and a 95% confidence interval, (0.617, 0.704). These results suggest a moderate capacity of interpreting the continuous self-reported Zung score in terms of no depression, mild depression, and severe depression. For this dataset, the computation time for the new estimator ${\check{\rho }}_{bsa}$ and the original estimator ${\widehat{\rho }}_{bsa}$ (without jackknife variance estimation) are 0.022 and 1154.773 seconds respectively. This again demonstrates the significant computational benefit of the proposed BSA method.