Intraclass Correlation Coefficients for Planning Cluster Randomized Trials in Community-Dwelling Older Adults

Abstract

Objectives:

With the emerging trends, more cluster randomized trials will be conducted in older adults, where facilities are randomized rather than individuals. Similarity of individuals from a facility (intraclass correlation coefficient/ICC) plays a critical role, but not readily available. We document ICCs for measures commonly used in community-dwelling older adults and discuss implications.

Methods:

Secondary analysis of a range of baseline measures from the On the Move cluster randomized trial, whose ICCs were computed using a linear mixed model.

Results:

Self-reported disability measures related to facility characteristics and sense of community had the greatest ICCs (>0.10), while mobility performance measures had 0.05–0.10, and cognitive measure 0.11.

Discussion:

The ICCs for measures commonly used in older adults are of a sufficient magnitude to have a substantial impact on planned sample size of a study and credibility of results, and should be taken into consideration in study planning and data analysis.

Background

As the segment of older adults, which represents the largest consumer of healthcare, increases with the emerging demographic trends, more randomized trials are likely to be planned and performed in such populations. As its subpopulation living in organized settings also increases, many of trials will be cluster randomized trials, where groups of participants (as defined by affiliation with a facility) are randomized rather than individuals. Therefore, methodological groundwork for such trials to be planned and conducted is urgently needed. One such methodological aspect is the need to rigorously account for clustering of individual participants by residential facility.

Clustering by residential facility is the general tendency for individuals from the same facility to be more similar or have a greater correlation among them with respect to a certain outcome, compared to individuals from different facilities. The phenomenon may be driven by facility characteristics, and plays a critical role in planning a cluster randomized trial. The most common reasons for cluster randomization are to prevent cross-contamination between individuals in different intervention arms due to increased interaction from being affiliated with the same facility, and ease of intervention delivery. The intra-cluster correlation (ICC), which quantifies the said dependence among individuals, causes the credibility afforded by N number of individuals in a trial to be lessened by a factor called the design effect. Therefore, the required sample size in an individually randomized trial must be inflated by the design effect, and it represents a critical component in cluster randomized trial planning. The design effect is commonly estimated as $1+\left(m-1\right)\rho ,$ where m is the cluster size and $\rho$ the intra-cluster correlation (Kerry & Bland, 1998). With unequal cluster sizes, the design effect generalizes to $1+\left[\left\{1+{CV}^2(k-1)/k\right\}\stackrel{-}{m}-1\right]\rho ,$ where it additionally depends on the number of clusters k, mean cluster size $\stackrel{-}{m}$ and coefficient of variation of cluster size CV², and is approximated by $1+\left\{\left(1+{CV}^2\right)\stackrel{-}{m}-1\right\}\rho$ when number of clusters is sufficiently large (Eldridge, Ashby & Kerry, 2006).

In planning a randomized trial, typically a smaller pilot trial is conducted under the same protocol, and descriptive statistics such as standard deviations from the pilot trial and other sources provide reasonable variability estimates for the computation of sample size requirements for the planned trial. However, when planning a cluster randomized trial, the critically needed intra-cluster correlation $\rho$ cannot be estimated from a small pilot trial. For reliable estimation of $\rho$, individual data from a larger trial in a similar population and with multiple clusters are needed, which may not be readily available. Therefore, many cluster randomized trials are being planned simply assuming or without a documented rigorous justification for the value for $\rho ,$ which may not be reasonably accurate leading to inaccurate a priori sample size and power computations. In reviews of primary prevention and non-therapeutic intervention trials, only 19% of the trials considered clustering in sample size planning and only 50–57% in data analysis (Simpson, Klar & Donnor, 1995; Donner, Brown & Brasher, 1990). A more recent review of orthopedic surgery trials did not show a substantial improvement in the state of affairs, and suggested not including a statistical specialist in the study team may also play a role (Oltean & Gagnier, 2015). Precisely for the above reasons, in certain disciplines such as education where cluster randomized trials are common, values of $\rho$ can be easily found in literature for study planning (Hedges & Hedberg, 2007; Shackleton, Hale & Viner, 2016). Even in certain areas of health sciences such as primary care, in-patient heart failure and diabetes populations, published values of $\rho$ can be found (Parker, Evangeloua & Eaton, 2005; Kul, Vanhaecht & Panella, 2014; Littenberg & MacLean, 2006). Research funding agencies such as the Patient-Centered Outcomes Research Institute even mandates that $\rho$ be incorporated in the sample size justification and data analysis (Patient Centered Outcomes Research Institute, 2017). However, values of $\rho$ in older adult populations are scarce in the literature except for one report for binary outcomes (Smeeth & Ng, 2002), a gap we sought to address.

In the present report, to facilitate planning and interpreting cluster randomized trials in community-dwelling older populations, we seek to document the observed intra-cluster correlation coefficients of commonly used continuous measures of mobility, disability, function, and cognition in older adults affiliated with organized settings such as residing in independent living communities and senior apartment complexes, and attending senior community centers. We also discuss their implications for planning and interpreting results of cluster randomized trials using similar measures in community-dwelling older adults.

Methods

Setting and Participants-

We conducted a secondary analysis of baseline data from the On the Move trial (Brach, Perera, Gillmore, Vanswearingen, Brodine, Nadkarni & Ricci, 2017; Wert, Perera, Nutini, Ricci, Coffman, Turnquist & Brach, 2017). Detailed study methods appear elsewhere (Brach, Perera, Gillmore, VanSwearingen, Brodine, Wert, Nadkarni & Ricci, 2016). Briefly, the parent study was a cluster randomized trial of On the Move timing and coordination group exercise program to improve mobility in community-dwelling older adults. Thirty-two independent living facilities, senior apartment complexes and senior community centers in the greater Pittsburgh metropolitan area participated in the study (2014–2016). Inclusion criteria were age 65 or older, a resident or a member of a participating facility, and ability to ambulate independently (with or without a straight cane) for household distances with a gait speed of at least 0.60 m/s. Those who were non-English speaking, had impaired cognition (unable to follow a 2-step command or understand the informed consent process), an acute or unstable medical condition, an inappropriate physiologic response (based on heart rate and blood pressure) to the 6-minute walk test, or planned to leave the area for an extended period of time were excluded. Four-hundred-twenty-four individuals across the three organized settings participated in the parent trial. The study was approved by the University of Pittsburgh Institutional Review Board and registered in clinicaltrials.gov (NCT01986647).

Measures-

The study included a range of health status, mobility and physical performance, disability, function, cognitive and mood-related measures commonly treated as continuous variables. Health status measures included body mass index and the Duke Comorbidity Index (range 0–8) (Rigler, Studenski, Wallace, Reker & Duncan, 2002). Walking performance measures included usual pace gait speed, and stance time, step length and step width variability (defined as the between-step standard deviation) collected via 6 passes on a 4-meter instrumented walkway (Zeno Walkway, Zenometrics, Peekskill, New York); six-minute walk distance (including time for rest as needed) (Butland, Pang, Gross, Woodcock & Geddes, 1982); and times for competing challenging walks on narrow, obstacle and figure-of-8 paths (Shumway-Cook, Guralnik, Phillips et al., 2007; Hess, Brach, Piva & VanSwearingen, 2010). Disability measures included the domain scores of the self-reported Late Life Function and Disability Index: overall function, upper extremity function, basic lower extremity function, advanced lower extremity function, disability frequency, social role, personal role, instrumental role, management role and disability limitations (Jette, Haley, Coster et al., 2002; Haley, Jette, Coster et al., 2002), all with a range of 0–100. Cognition was assessed using the Digit-Symbol Substitution Test (range 0–90) (Wechsler, 1981). Other measures included the modified Gait Efficacy Scale (range 10–100) (Newell, VanSwearingen, Hile & Brach, 2012) and Patient Health Questionnaire (PHQ-9; range 0–27) (Kroenke, 2002).

Statistical Analysis-

We used appropriate summary statistics (means, standard deviations, frequencies and percentages) to describe the sample characteristics. We fitted a series of intercept-only linear mixed models with each measure of interest as the dependent variable, no fixed effects, and a compound symmetric working correlation structure within each facility. Between- (${\sigma }_B^2$) and within-facility (${\sigma }_W^2$) variance components were estimated using restricted maximum likelihood method, and intra-cluster correlation estimated as $\widehat{\rho }={\widehat{\sigma }}_B^2/({\widehat{\sigma }}_B^2+{\widehat{\sigma }}_W^2)$. We used SAS^® version 9.3 for all analyses with MIXED procedure for the estimation of $\rho .$

Results

Participant characteristics appear in Table 1. Seven senior community centers, 10 independent living facilities and 15 senior apartments complexes participated in the parent study for a total of 32 facilities. From the 32 facilities, 424 individuals (92 from community centers, 176 from independent living facilities and 156 from senior apartment complexes) were randomized. Median number of participants per facility (cluster size) was 12, with a range of 4–29. Participants generally represented the very old, were predominantly white females, and the majority lived alone. Participants had mean a body mass index in the pre-obese range and comorbidities in about 3 organ systems. Over half of the participants reported having excellent or very good mobility and health, but only about a third reported the same with respect to balance. Only a minority of participants met the mobility performance criteria adequate for community ambulation with respect to gait speed and 6-minute walk distance (Lerner-Frankiel, Vargas, Brown, Krusel & Schoneberger, 1986; Robinett & Vondran, 1988). Approximately a third of participants reported fear of falling and a history of falls during the prior year.

Table 1:

Participant characteristics and measures

Characteristic (Possible Range)	Mean ± Standard Deviation or N (%) N=424
Intervention setting
Community senior center	92 (21.7)
Independent living facility	176 (41.5)
Senior apartment complex	156 (36.8)
Age (years)	80.7±7.8
Female gender	349 (82.3)
White race	352 (83.0)
Live alone	307 (72.4)
Married	99 (23.4)
College education	215 (51.0)
Comorbidities
Cardiovascular	76 (17.9)
Neurological	33 (7.8)
Musculoskeletal	347 (81.8)
General	171 (40.3)
Visual/hearing	316 (74.5)
Diabetes	84 (19.8)
Cancer	84 (19.8)
Lung	93 (21.9)
Duke comorbidity index (0–8)	2.84±1.40
Six-minute walk distance (m)	276.8±89.6
<300	239 (56.4)
≥300	185 (43.6)
Narrow walk time (s)	6.45±2.72
Obstacle walk time (s)	9.19±2.60
Digit symbol substitution test (DSST)	36.3±10.9
Figure of 8 walk test
Walk time (s)	10.45±3.29
Number of steps	17.9±4.2
Modified Gait efficacy scale (mGES) (10–100)	75.2±14.3
Short physical performance battery (SPPB) (0–12)	9.40±1.81
Patient health questionnaire (PHQ-9) (0–27)	2.43±2.78
Fear of falling	148 (34.9)
Fall prior year	128 (30.2)
Self-reported excellent/very good mobility	247 (58.3)
Self-reported excellent/very good health	225 (53.1)
Self-reported excellent/very good balance	131 (30.9)
Height (m)	1.61±0.12
Weight (kg)	73.8±20.5
Body mass index (kg/m2)	28.9±13.4
Late Life Function and Disability Index
Overall function (0–100)	59.5±9.4
Upper extremity function (0–100)	77.2±12.0
Basic lower extremity function (0–100)	73.0±14.1
Advanced lower extremity function (0–100)	47.9±15.0
Disability frequency (0–100)	52.7±6.4
Social role (0–100)	47.9±8.2
Personal role (0–100)	64.6±16.0
Instrumental role (0–100)	78.7±15.3
Management role (0–100)	91.2±12.5
Disability limitations (0–100)	78.8±14.5
Instrumented walkway gait speed (m/s)	0.91±0.20
<0.8	123 (29.0)
0.8–1.0	152 (35.8)
>1.0	126 (29.7)
Stance time variability (s)	0.044±0.022
Step length variability (cm)	3.49±1.05

Between- and within-facility variances, and ICCs appear in Table 2. For most commonly used physical performance measures of mobility, ICCs were generally in the 0.05–0.10 range. ICC was $\widehat{\rho }$=0.09 for 6-minute walk distance, $\widehat{\rho }$=0.05 for SPPB, $\widehat{\rho }$=0.10 for gait speed, and $\widehat{\rho }$=0.06 for stance time variability. Of the considered measures, ICCs were relatively higher in measures of self-reported LLFDI domains. Although overall function, and basic and advanced lower extremity scores had relatively very low coefficients of $\widehat{\rho }$=0.01, disability frequency ($\widehat{\rho }$=0.20), social role ($\widehat{\rho }$=0.14), personal role ($\widehat{\rho }$=0.21), instrumental role ($\widehat{\rho }$=0.09), management role ($\widehat{\rho }$=0.31) and disability limitations had relatively higher ICCs. Similar to the walking performance measures, ICC for the cognitive functioning speed, DSST had $\widehat{\rho }$=0.11. The LLFDI upper extremity function domain had relatively negligible ICCs.

Table 2:

Between- and within-facility variabilities and intraclass correlation coefficients

Outcome	Between-Facility Variance (${\widehat{\sigma }}_B^2$)	Within-Facility Variance (${\widehat{\sigma }}_W^2$)	Intraclass Correlation Coefficient ($\widehat{\rho }$)
Walking & Physical Performance:
Instrumented walkway gait speed (m/s)	0.00	0.04	0.096
Stance time variability (s)	0.00	0.00	0.058
Step length variability (cm)	0.07	1.03	0.060
Six-minute walk distance (m)	687.8	7407.9	0.085
Narrow walk time (s)	0.07	7.34	0.009
Obstacle walk time (s)	0.37	6.41	0.054
Figure of 8 walk test walk time (s)	1.56	9.76	0.138
Figure of 8 walk test number of steps	0.91	16.82	0.052
Short physical performance battery (SPPB)	0.16	3.14	0.049
Disability Measures:
Late Life Function and Disability Index
Overall function	0.88	87.59	0.010
Upper extremity function	−1.94 [0.00]	145.87	−0.013 [0.000]
Basic lower extremity function	1.97	196.14	0.010
Advanced lower extremity function	2.57	221.17	0.011
Disability frequency	8.28	32.71	0.202
Social role	9.57	58.80	0.140
Personal role	53.81	208.11	0.205
Instrumental role	20.12	215.85	0.085
Management role	54.22	120.11	0.311
Disability limitations	20.62	190.20	0.098
Cognitive Measure:
Digit symbol substitution test (DSST)	12.69	106.31	0.107

Discussion

We have obtained estimates of ICCs for a range of mobility performance, disability and cognitive measures in community-dwelling older adults affiliated with organized settings. We are aware of only one prior study by Smeeth et al. reporting ICCs for community-dwelling older adults (Smeeth & Ng, 2002). The study focused on older adults over the age of 75 from a general practice setting in the United Kingdom, and mostly binary outcomes of morbidity, functioning, social variables, daily activities, alcohol use and smoking were reported. To our knowledge, ours is the first report documenting intraclass correlations in continuous interval and ratio scale measures commonly used as outcomes in community-dwelling older adults, clustered in their residential or community rather than healthcare setting, and complements the work of Smeeth et al (Smeeth & Ng, 2002). Many have called for the publication of ICCs from cluster randomized trials, including the CONSORT guidelines extension to cluster randomized trials (Murray, McKinlay, Martin, Donner, Dwyer, Raudenbush & Graubard, 1994; Campbell, Piaggio, Elbourne & Altman, 2012). The findings reported are a response to the calls above with respect to community-dwelling older adults.

In planning cluster randomized trials, the reported ICCs and anticipated cluster sizes can be used to compute a design effect or the factor by which to inflate the unclustered sample size required to achieve a given level of acceptable statistical power. For example, in planning a hypothetical 1:1 two-arm randomized trial with gait speed with ρ=0.096 as the primary outcome, if the required sample size to achieve 90% statistical power with an unclustered design is 500 participants, and anticipated cluster size is 10 participants per facility, the design effect is $1+\left(m-1\right)\rho$ =1.864. The sample size required is 500×1.864=932 for a cluster randomized design. If the cluster size cannot be fixed, and anticipated to vary with a range of 4–16 with a mean of 10 for example, then an approximation for the coefficient of variation can be estimated as range/(4×mean) or 0.3. Then the design effect can be approximated as $1+\left\{\left(1+{CV}^2\right)m-1\right\}\rho$=1.950 and the required sample size as 976. The sample size requirement increases with respect to ICC, average cluster size and relative variability in cluster size. Under the abovementioned fixed cluster size of m=10 scenario, if the effect of clustering had not been considered and only 500 participants were randomized in a simple two-group means comparison with equal sample sizes and variances, the study would have had an effective sample sizes of only 500/1.864=268, and statistical power of only 66% compared to the anticipated 90%. The same observation can be used in re-interpreting findings from a published older study which failed to account for clustering in analysis. For example, if the hypothetical study reported a test statistic of t=1.99, degrees of freedom df=498 and a p-value of 0.0471 as its main finding, one can interpret it with a lesser level of credibility approximated as $t=1.99\sqrt{268/500}=$1.46 with df=266 and p=0.1455. While ICCs smaller than 0.40 are considered poor in other common settings such as test-retest reliability, the magnitude of the ICC should be interpreted based on its impact on design effect in the current setting. For instance, in the above example with a fixed cluster size m=10, an outcome with an ICC of seemingly very small magnitude of 0.02 would yield a design effect of $1+\left(m-1\right)\rho$ =1.18 and require inflation of sample size by a non-trivial factor of 18%. Finally, while sample sizes are commonly estimated using only primary outcomes and ICCs of secondary outcomes may be less relevant in study planning, all analytic strategies (including those for secondary outcomes) should account for clustering effects to prevent conclusions biased towards the alternative hypothesis.

Although perhaps less appreciated, the clustering phenomenon can also reduce the required sample size in certain multi-center study designs. For example, when individuals are randomized to different interventions within the same facility, such as in a drug trial with little opportunity for cross-contamination due to similarly packaged active and placebo medications, the role of between-facility variability is largely eliminated from the between-treatment comparisons and the study can be completed with a lesser number of participants than when the clustering effect is ignored, thereby potentially reducing study cost and duration, and risks to participants. The design effect representing the factor by which the sample size is reduced is given by $1-\rho$ (Vierron & Giraudeau, 2007). The design effects in the above hypothetical scenario would be $1-\rho$=0.904, and the study could be conducted with only 500×0.904=452 participants.

Of the considered measures, we observed relatively greater magnitudes of ICCs in LLFDI domains for disability frequency, social role, personal role, instrumental role and management role. We speculate the greater intra-facility correlations are likely due to the nature of questionnaire items that constitute the domain scores. Responses to items such as keeping in touch with others through letters, phone or email; visiting friends and family in their homes; providing care or assistance to others; inviting people into your home for a meal or entertainment; going out with others to public places such as restaurants and movies; and taking part in organized social activities are used in computation of the above domain scores (Jette, Haley, Coster et al., 2002; Haley, Jette, Coster et al., 2002). Responses to these items are likely to be influenced to a greater degree by the sense of community in the facility environment than responses to other measures, thereby increasing the ICC.

The between-facility variance estimates and ICCs presented for LLFDI upper extremity function is negative, and deserves comment as variances, by definition, are non-negative. Although we operationally computed ICC using the numerical value of between-facility variance, the negative variance estimate is an artifact of the negative ICC. Although uncommon, there is no mathematical or conceptual reason that an ICC cannot be negative (Littell, Milliken, Stroup, Wolfinger & Schabenberger, 2006). A lower bound for ICC is −1/(m-1), a negative value that depends on cluster size m (Bartko, 1976). Examples might be in situations where intra-cluster interference or competition is a factor, such as with measures of attention or ones that involve a limited quantity (Littell, Milliken, Stroup, Wolfinger & Schabenberger, 2006). The greater the value of the measure for some individuals in a cluster, the lesser the value for remaining individuals thus causing a negative ICC. However, the measure in question does not plausibly involve competition or interference. We feel it is likely a simple numerical artifact, and within-facility variance is much larger compared to between-facility variance. An essentially zero ICC being estimated by iterative computational algorithms converging to a near zero negative numbers such as −0.013. Therefore, we interpret the said ICC as 0, thus resulting in a design effect of 1 and no adjustment to sample size.

Some limitations should be considered when interpreting our findings. First, stable estimation of ICCs ideally requires sufficiently large numbers of both clusters and individuals per cluster. Although we had a considerable number of 32 clusters, some clusters had as little as 4 individuals. It may have affected our estimates of within-cluster variability and ICC. Second, our findings are based on individuals meeting eligibility criteria for a randomized trial of an exercise intervention, and may not readily generalize to the wider community-dwelling older population. However, any planned cluster randomized trial will have inclusion/exclusion criteria, and findings from a similar group are arguably more relevant. Third, it is important to note that one could reasonably propose an analytic strategy for a certain cluster randomized trial where data is first averaged by cluster and then the cluster level averages are analyzed without further consideration of clustering. Such strategies do not allow for individual-level covariate adjustment (Simpson, Klar & Donnor, 1995) and may lead to some information loss due to averaging by cluster. It may have been a reasonable alternative in some older studies conducted before the appropriate statistical methods and computing tools were widely available. In such cases, the clusters tend to be independent and uncorrelated, thus making the ICC irrelevant. Finally, our study was conducted in a single metropolitan area, and may be subject to other concerns of generalizability due to an under-representation of non-White participants and over-representation of women compared to the wider population.

Conclusion

We conclude that most of the measures of mobility performance, cognition and disability considered have ICCs of sufficient magnitudes to make a substantial impact on the sample size of a planned study and credibility of its results. They should be taken into consideration in study planning, and appropriate statistical techniques that consider the impact of high ICCs on standard errors and significance should be used for analysis of such trials.