Cluster randomized trials of cancer screening interventions: are appropriate statistical methods being used?

Abstract

The design and analysis of cluster randomized trials can require more sophistication than individually randomized trials. However, the need for statistical methods that account for the clustered design has not always been appreciated, and past reviews have found widespread deficiencies in methodology and reporting. We reviewed cluster randomized trials of cancer screening interventions published in 1995–2010 to determine whether the use of appropriate statistical methods had increased over time. Literature searches yielded 50 articles reporting outcome analyses of cluster randomized trials of breast, cervix and colorectal cancer screening interventions. Of studies published in 1995–1999, 2000–2002, 2003–2006 and 2007–2010, 55% (6/11), 82% (9/11), 92% (12/13) and 60% (9/15) used appropriate analytic methods, respectively. Results were suggestive of a peak in 2003–2006 (p=.06) followed by a decline in 2007–2010 (p=.08). While the sample of studies was small, these results indicate that many cluster randomized trials of cancer screening interventions have had deficiencies in the application of correct statistical procedures for the outcome analysis, and that increased adoption of appropriate methods in the early and mid-2000’s may not have been sustained.

Introduction

In cluster randomized trials, natural groups or “clusters” of individuals such as families, health care providers or communities are randomized to intervention and control conditions. A cluster randomized design can have a number of advantages, such as reduced potential for contamination compared to an individually randomized trial and increased efficiency of intervention delivery [1–3]. In addition, many interventions are naturally applied at a group level (e.g., interventions directed at health care providers to promote cancer screening among patients) [2]. For these reasons, cluster randomized trials have become an increasingly important study design in intervention and clinical research [1, 3–5].

The design and analysis of cluster randomized trials can require more sophisticated approaches than individually randomized trials. Individuals who receive care from the same provider or live in the same community tend to be more alike than individuals who have different providers or live in different communities. As a result, in cluster randomized trials, the outcomes of individuals within the same cluster are statistically dependent rather than independent, a phenomenon that must be accounted for in the design and analysis of a trial [1–3, 6]. The degree of correlation among participants within the same cluster is typically quantified by the intraclass correlation coefficient (ICC). Several references provide comprehensive expositions of appropriate design and analysis methods for cluster randomized trials [1, 2, 7].

The special features of cluster randomized trials have not always been appreciated by investigators. Reviews of cluster randomized trials published in the 1990’s and early 2000’s found widespread deficiencies in conduct and reporting, with many studies not accounting for clustering in the planning stage, the data analysis or both, and not providing adequate reporting of study features [8–12]. In response to these deficiencies, the Consolidated Standards of Reporting Trials (CONSORT) statement extension to cluster randomized trials was published in March 2004 [13]. This statement extended the CONSORT statement [14, 15], which provides a framework for reporting parallel group individually randomized trials, to include reporting of the special features of cluster randomized trials, and provides a checklist of specific items to report.

While there are many features of a cluster randomized trial that should be accounted for and reported, perhaps of most importance is that the outcome analysis use appropriate statistical methodology that accounts for the clustered design. Failure to account for clustering generally results in an inflated Type I error rate (the probability of incorrectly rejecting the null hypothesis of no effect), leading to artificially small p-values and erroneous conclusions that a treatment is effective when it is not [2, 5, 16].

We identified cluster randomized trials of cancer screening interventions published from 1995 to 2010 and examined the methodology they used for the outcome analysis as a key quality indicator. Our research question was whether the use and reporting of appropriate statistical methods had increased over time, and in particular after the publication of the CONSORT extension to cluster randomized trials. We also examined whether the use of appropriate methods differed by journal impact, and whether studies reported ICCs for outcomes, which are useful for investigators planning future trials and are included in the CONSORT checklist of items to be reported.

Methods

We conducted a literature search to identify articles that reported outcome analyses of cluster randomized trials of cancer screening interventions. Our inclusion criteria for articles were: (1) receipt of breast, cervical or colon cancer screening as an outcome; (2) use of a cluster randomized design; (3) in English; and (4) published between January 1, 1995 and December 31, 2010. We used PubMed and Web of Science to identify papers that met our inclusion criteria. Due to the lack of standardized nomenclature for reporting the use of a cluster-randomized design, we searched broadly for papers reporting randomized trials of breast, cervical or colorectal cancer screening interventions, then screened the resulting collection of papers for those reporting cluster-randomized trials with a cancer screening outcome. In PubMed, searches were conducted utilizing the PubMed automatic MeSH translation capability, with limits specified as (1) clinical trials, (2) English language, and (3) publication dates January 1, 1995 through December 31, 2010. Three searches were conducted using the terms “randomized trial [site] cancer screening,” where site was “colorectal,” “breast” or “cervical.” The searches were conducted on February 1, 2011. We supplemented these searches with citation searches of articles reporting study design, baseline data or preliminary results, in an attempt to find the associated outcome paper. We also presented our initial list of articles to cancer screening experts familiar with the literature to identify omitted studies.

Each article was independently reviewed by two statisticians to ascertain the statistical methodology used for the analysis of the cancer screening outcome variable. Authoritative texts [1, 2, 7] were used as a guide for classifying methods as appropriate or not. Since the style of reporting methods varied widely, identification of the statistical methods used usually necessitated an in-depth review of the Methods and Results sections of each article. Articles that made reference to an appropriate method were classified as using an appropriate method even if the article did not provide enough detail to conclusively determine that the method was applied appropriately. Initial disagreements between raters were settled by article re-review, discussion and consensus.

To assess whether rates of the use of appropriate methods varied by journal impact, we classified each article as “high influence” or “low influence” using Article Influence scores obtained from www.eigenfactor.org. This measure uses citation data to assess the influence of a journal in relation to other journals and is roughly analogous to the 5-Year Journal Impact Factor in that it is a ratio of a journal’s citation influence to the size of the journal’s article contribution over a period of five years. The mean score is 1.00. Articles with Article Influence scores greater than 1.00 were categorized as high influence; articles with scores less than 1.00 were categorized as low influence. Four articles did not have Article Influence scores; these articles were in publications with little or no coverage in citation databases and were classified as low influence.

We used symmetric nearest neighbor smoothing [17], implemented with the Stata user-contributed “running” command, to obtain smoothed estimates of the running proportion of studies using appropriate methods over time (i.e., proportions averaged over nearest neighbors in time). The binary indicator 0/1 for use of appropriate methods was logit transformed and a span of 0.4 was used. Logistic regression with linear and quadratic terms for year of publication was used to infer the statistical significance of overall time trends. Due to the relatively small sample sizes involved, we used exact methods to obtain estimates of the proportion of studies using appropriate methods by time interval and to test for differences in proportions between time intervals. For these analyses, we divided our time periods into four intervals, January 1, 1995–December 31, 1999, January 1, 2000–December 31, 2002, January 1, 2003–December 31, 2006 and January 1, 2007–December 31, 2010. The four intervals were chosen so as to yield approximately equal sample sizes across the intervals and so that the last interval would consist of studies published a sufficiently long time after publication of the CONSORT statement extension in March 2004 to allow time to incorporate the guidelines into planned statistical analyses as well as time for analysis and publication. 95% confidence intervals were obtained as Clopper-Pearson binomial exact confidence intervals. Fisher’s exact test was used to test for differences in proportions between time intervals.

To explore whether failure to use appropriate methods was associated with a higher probability of a finding of statistical significance, we classified each article as having one or more statistically significant results for cancer screening outcomes, and compared these proportions between the “appropriate” and “not appropriate” method groups using Fisher’s exact test. One study was omitted from this analysis because the authors considered their results ambiguous.

All analyses were conducted using Stata 11 [18].

Results

Table 1 summarizes the articles reporting outcome analyses of cluster randomized trials of cancer screening interventions that we identified through our search. We identified 50 articles meeting our inclusion criteria, including 14 articles reporting the outcome of a breast screening intervention trial, 6 reporting a cervical cancer screening intervention trial, 20 reporting colorectal cancer screening interventions, 4 reporting joint interventions for both breast and cervical cancer screening and 6 reporting interventions for screening all three cancer sites.

Table 1.

Studies reporting outcome analyses of cluster randomized trials of breast, cervical or colorectal cancer screening interventions

First author	Year	Randomization units	Cancer site	Reference
Urban	1995	Communities	BR	[40]
Clover	1996	Communities	BR	[41]
Dietrich	1998	Health centers	BR,CV,CR	[42]
Hodge	1998	Clinics	CV	[43]
Kinsinger	1998	Medical practices	BR	[44]
Manfredi	1998	Medical practices	BR,CV,CR	[45]
Slater	1998	Apartment buildings	BR	[20]
Bastani	1999	Families	BR	[46]
Bonevski	1999	Physicians	CV	[47]
Powe	1999	Senior centers	CR	[48]
Tilley	1999	Worksites	CR	[49]
Andersen	2000	Communities	BR	[50]
Thompson	2000	Clinical care teams	CR	[51]
Allen	2001	Worksites	BR,CV	[52]
Hancock	2001	Communities	CV	[53]
Richards	2001	Medical practices	BR	[21]
Segura	2001	Apartment buildings	BR	[22]
Powe	2002	Senior centers	CR	[54]
Reuben	2002	Various community-based sites	BR	[55]
Taylor	2002	Neighborhoods	CV	[56]
Vinker	2002	Physicians	CR	[57]
Zhu	2002	Housing complexes	BR	[58]
Holloway	2003	Medical practices	CV	[26]
Luckmann	2003	Medical practices	BR	[59]
Campbell	2004	Churches	CR	[24]
Roetzheim	2004	Clinics	BR,CV,CR	[60]
Ruffin	2004	Medical practices	BR,CV,CR	[61]
Braun	2005	Civic clubs	CR	[62]
Federici	2005	Physicians	CR	[25]
Ferreira	2005	Clinics	CR	[28]
Hughes	2005	Medical practices	CR	[63]
Roetzheim	2005	Clinics	BR,CV,CR	[64]
Walsh	2005	Physicians	CR	[65]
Federici	2006	Medical practices	CR	[27]
Thompson	2006	Communities	BR,CV,CR	[66]
Dinshaw	2007	Communities	BR,CV	[67]
Mishra	2007	Churches	BR	[23]
Stephens	2007	Families	CR	[68]
Hiatt	2008	Clinics	BR,CV	[69]
Jandorf	2008	Various community-based sites	BR,CV	[70]
Lane	2008	Health centers	CR	[71]
Jensen	2009	Physicians	CV	[29]
Ling	2009	Physicians	CR	[72]
Ma	2009	Churches	CR	[73]
Sequist	2009	Physicians	CR	[74]
Atlas	2010	Medical practices	BR	[75]
Blumenthal	2010	Various community-based sites	CR	[76]
Liu	2010	Communities	BR	[77]
Morgan	2010	Churches, community-based organizations	CR	[78]
Ornstein	2010	Medical practices	CR	[79]

BR, breast; CR, colorectal; CV, cervix

In 25 of the studies, the randomization units were health care providers or facilities, most commonly medical practices (n=10) or physicians (n=9). There were 21 community-based trials, in which the units of randomization were communities or neighborhoods (n=8), housing units (n=3) or sites where community members congregate such as churches, clubs or senior centers (n=10). Two studies randomized by worksite, and two by family.

Review of the methodology used for analyzing the cancer screening outcome in each study indicated that 72% (36/50, exact 95% confidence interval 58%–84%) of the studies used methods appropriate for cluster randomized trials. Among high influence (n=22) versus low influence (n=28) articles, 77% (55%–92%) versus 68% (48%–84%) used appropriate methods; the difference was not significant (p=.54, Fisher’s exact test).

Figure 1 presents smoothed estimates of the running proportions of articles reporting appropriate methods over the study period of 1995 to 2010, for all articles (Figure 1(a)), high influence articles (Figure 1(b)), and low influence articles (Figure 1(c)). All three figures suggest a rising trend followed by a plateau then a decline. Logistic regression confirmed a significant quadratic trend in the overall sample (p=.03) and yielded results consistent with quadratic trends in both the high and low influence articles (both p=.08). For articles in high influence journals, the proportion appeared to reach a higher plateau than for articles in low influence journals. During this plateau, which ran from 2000 to 2007, all articles in high influence journals used appropriate methods. The smoothed estimates suggested that the proportion for low influence articles fell to a lower level by 2010 than that for high influence articles.

Figure 1. Smoothed running proportions of articles reporting appropriate methods, 1995 to 2010.

Symmetric nearest neighbor smoothing was used to obtain smoothed estimates of the running proportion of studies using appropriate methods. The binary indicator for use of appropriate methods was logit transformed and a span of 0.4 was used. Values of 0 and 1 are jittered and plotted below and above the smoothed curves, respectively.

Table 2 provides the proportions of studies that used appropriate analytic methods by publication time interval and results of tests for differences in proportions over time. The proportions in the intervals 1995–1999, 2000–2002, 2003–2006 and 2007–2010 were 0.55, 0.82, 0.92 and 0.60, respectively. No differences in proportions were significant at the 0.05 level; however, the results were consistent with a peak in the 2003–2006 time frame (p=.06) and drop from 2003–2006 to 2007–2010 (p=.08).

Table 2.

Proportions of articles using appropriate analytic methods for cancer screening outcomes by publication time interval

Publication time interval	Proportion of studies using appropriate methods for cancer screening outcome	Exact 95% confidence interval	P-value, difference of proportions compared to previous interval	P-value, difference of proportions compared to 1995–1999
By interval:
1995–1999	0.55 (6/11)	0.23–0.83
2000–2002	0.82 (9/11)	0.48–0.98	.36	.36
2003–2006	0.92 (12/13)	0.64–0.99	.58	.06
2007–2010	0.60 (9/15)	0.32–0.84	.08	.99
Overall:
1995–2010	0.72 (36/50)	0.58–0.84

P-values for differences of proportions were obtained using Fisher’s exact test.

Of the articles using appropriate methods, the proportion reporting a statistically significant finding 0.72 (95% exact confidence interval 0.53–0.86). For studies not using appropriate methods, the proportion was 0.76 (0.50–0.93). The difference between the two groups was not statistically significant (p = .99 by Fisher’s exact test).

Studies not using appropriate analytic methods most often erred by treating participant outcomes as independent rather than correlated. Of the 36 studies using analytic methods that accounted for clustering, the most commonly used method was mixed models (31%, 11/35), followed by generalized estimating equations (26%, 9/33). Four studies stated that they used a robust variance estimator, four used clusters as the unit of analysis and three used survey data methods. Other methods included the Cochran-Mantel-Haenzel chi-square test, adjusted z-test [19], and permutation tests. In several cases, the papers included statements to the effect that the standard errors were adjusted for clustering or a hierarchical model was used to account for the design, but the particular method used was not stated explicitly.

Only seven studies reported ICC values for the primary outcome [20–26]; one of these studies reported the ICC for the control group only [21]. An additional two studies [27, 28] reported the design effect, which is equal to 1 + ICC × (average cluster size − 1). One study [29] reported ICCs for baseline data but not outcome data. Table 3 provides a compilation of ICC values for outcome data from the trials. The percentages of studies reporting ICCs for outcomes by time interval were 9% (1/11) in 1995–1999, 18% (2/11) in 2000–2002, 38% (5/13) in 2003–2006 and 7% (1/15) in 2007–2010. Fisher’s exact test gave a p-value of .07 for a difference between the last two time intervals. While these numbers are small, the pattern of proportions is consistent with the time trend for use of appropriate methods.

Table 3.

Intraclass correlation coefficients reported in cancer screening intervention trials

Reference	Year	Cluster type	Outcome	ICC
Slater [20]	1998	Apartment buildings	Self-reported receipt of mammogram	−0.0015
Segura [22]	2001	Apartment buildings	Attendance for mammogram	0.065
Campbell [24]	2004	Churches	Fecal occult blood test screening	<0.01
Mishra [23]	2007	Churches	Self-reported receipt of mammogram	0.19
Richards [21]	2001	Medical practices	Attendance for breast screening	0.023 in control group
Holloway [26]	2003	Medical practices	Receipt of Pap smear	0.25 in intervention group, 0.0018 in control group
Federici [27]	2006	Medical practices	Receipt of FOBT or flexible sigmoidoscopy	0.008 in FOBT arm, <0.001 in flexible sigmoidoscopy arma
Federici [25]	2005	Physicians	Receipt of FOBT	0.056
Ferreira [28]	2005	Physicians	Receipt of colorectal cancer screening tests	0.077 in intervention group, 0.033 in control groupa

FOBT, fecal occult blood test; ICC, intraclass correlation coefficient

^aDesign effect provided in original article; ICC was computed as (design effect – 1)/(average cluster size – 1)

Discussion

Our investigation suggests that many cluster randomized trials of cancer screening interventions have had deficiencies in the application of correct statistical procedures for the outcome analysis, and that a trend towards increased adoption of appropriate methods in the late 1990s and early 2000s may not have continued after the mid-2000s. In particular, we did not find evidence to support an increase in the use of appropriate methods or reporting of the ICC after publication of the CONSORT statement extension to cluster randomized trials in March 2004. A particularly striking finding is that only a single study in the 2007–2010 time interval reported an ICC. Overall, differences in the rates of use of appropriate methods between articles in high influence versus low influence journals were not remarkable. However, there was a period in the mid-2000s when all articles in high influence journals used appropriate methods.

It is instructive to examine our findings in the context of earlier reviews of the methodological quality of cluster randomized trials. Past reviews of cluster randomized trial methodology for health-related studies are summarized in Table 4. The studies that examined time trends had findings consistent with increased adoption of appropriate methods over periods ranging to the mid-2000s, which is consistent with our findings. We speculate that ongoing methodological reviews highlighting deficiencies and increased dissemination of proper methods in seminal books, e.g., [2, 7] were responsible for the increasing trend through the mid-2000’s. None of these reviews included articles beyond 2008 and thus these studies would not have captured the downward trend in recent years that was suggested by our review. Our review adds to the literature because it includes more recent publications through 2010, many of them published well after the 2004 CONSORT extension for cluster randomized trials.

Table 4.

Summary of previous published reviews of methodology used in health-related cluster randomized trials

Reference	Type of studies reviewed	Publication years included	Percent of studies accounting for clustering in analysis	Time trends (when reported)
Donner [8]	Non-therapeutic intervention trials	1979–1989	50% (8/16)	1979–1985: 17% (1/6) 1986–1989: 70% (7/10)
Divine [9]	Physician patient-care behavior studies	1980–1990	30% (16/54)	No time trend
Simpson [10]	Primary prevention trials published in American Journal of Public Health and Preventive Medicine	1990–1993	57% (12/21)	NA
Chuang [11]	Clinical decision support system studies	1975–1998	58% (14/24)	Odds ratio 1.12 (95% CI 0.97–1.30) for one-year increase in year of publication
Isaakidis [12]	Sub-Saharan Africa intervention trials	1973–2001	37% (19/51)	1973–1995: 24% (6/25) 1996–2001: 50% (13/26)
MacLennan [80]	Guideline implementation studies	1966–2000	58% (87/149)	Odds ratio 1.08 (p=.025) for one-year increase in year of publication
Varnell [81]	Trials published in American Journal of Public Health and Preventive Medicine	1998–2002	54% (32/59)	NA
Murray [82]	Cancer prevention and control trials	2002–2006	45% (34/75)	NA
Bowater [83]	Tropical parasitic disease intervention trials	1998–2007	43% (15/35)	1998–2002: 33% (6/18) 2003–2007: 53% (9/17)
Handlos [84]	Trials of maternal and child health care interventions in developing countries	1998–2008	80% (28/35)	1998–2002: 38% (3/8) 2003–2008: 67% (18/27)

Our findings have several implications. First, since failure to account for clustering generally inflates the Type I error rate, it is likely that some non-equivalence trials of cancer screening interventions have overstated the statistical significance of the intervention effect. In particular, there may be instances in which an intervention was declared effective when a careful analysis adjusting for clustering would have failed to find an effect. Thus the evidence base for some cancer screening interventions may be overstated. In our review, the rates of any finding of statistically significance for cancer screening outcomes were similar between studies using appropriate methods and those that did not; however, our measure may have been insufficiently sensitive to detect a difference. We found that it was difficult to extract from the articles important information that would have permitted a more sensitive measure to be used (e.g., exact number of tests conducted, p-values for each test).

Our findings also have implications for the design of future trials of cancer screening interventions. Determination of the sample size needed for a planned cluster randomized trial to achieve the desired level of power generally requires an estimate of the ICC (see, e.g., [1, 2, 30]). An underestimated ICC will result in an underpowered study, and an overestimated ICC will cause a study to be overly large and wasteful of resources. Published literature is an important source of plausible ICC values for the planning of future studies [3, 6]. Omission of ICC values from published studies represents a lost opportunity to help in the efficient and effective design of future studies. A recent Journal of the National Cancer Institute monograph providing ICC estimates for cancer screening outcomes highlights the importance of published ICCs as a resource for other investigators who are planning studies [31]. Given this paucity of reported ICCs in the trial literature, investigators may wish to consider using such methods as sample size adaptive designs [32] as a way of dealing with uncertainty in the magnitude of the ICC that will be realized in a trial.

Three articles reported ICCs or design effects separately for each arm of the trial [26–28], and the arm-specific ICCs were quite different. A reanalysis of the trial reported by Mishra et al. [23] also found differences in the ICCs by study condition, with an ICC of 0.06 in the control condition and 0.34 in the intervention condition [33]. Thus in addition to accounting for correlation among participants, investigators planning or analyzing data from cluster randomized trials may also wish to consider the possibility of differences in the magnitude of the ICC between study conditions.

We found that it is difficult to systematically identify cluster randomized trials in the published literature. There is no standard nomenclature for describing a cluster randomized design; searches on terms such as “cluster randomized” or “group randomized” miss the many studies that do not use these terms anywhere in their abstracts, titles, keywords or text; this lead to our decision to broadly capture articles reporting randomized trials then scrutinize each such article to determine whether the trial was cluster randomized. Identification of a trial as cluster randomized often entailed a careful reading of the methods section of a paper, and even then, in some cases the randomization scheme was still unclear. A recent article by Taljaard et al [34] describes the challenges of identifying reports of cluster randomized trials in the literature in detail. Conformance with the CONSORT statement extension, which calls for identifying a trial as cluster randomized in the title and abstract, would remedy this problem.

The categorization of a trial as cluster randomized can be ambiguous. We excluded two studies that were time-randomized [35, 36] and thus might have involved some clustering effects attributable to time period. Some studies involved assigning individuals to small groups and randomizing these groups (e.g., [37, 38]). While it is debatable whether such studies should be termed “cluster randomized,” such designs do involve correlation among participants within the same group, a feature that should be accounted for in study design and analysis. Since we were not able to identify such studies systematically and because the necessity of accounting for correlation among participants in such trials has not been widely discussed or disseminated, we did not include such studies in our review.

We restricted our literature search to studies that had receipt of breast, cervical or colorectal cancer screening (yes/no) as an outcome. Thus our pool of studies did not include cancer screening interventions that did not have receipt of screening as an outcome, although cluster randomization is also used in many such interventions (e.g., [39]).

Conclusions

In conclusion, our work shows that there is a need for renewed attention to the methodological quality of cluster randomized trials. While we confined our investigation to breast, cervix and colorectal cancer screening intervention trials, it is likely that other intervention and clinical studies share similar methodological and reporting deficiencies. Future reports of cancer screening intervention trials should adhere to the CONSORT statement and its extension to cluster randomized trials and include clear descriptions of the randomization scheme, unit of analysis and the statistical methods used to account for correlated data, and report the ICC to allow for better planning of future studies.

Abbreviations

BR

breast

CE

cervical

CR

colorectal

CONSORT

Consolidated Standards of Reporting Trials

FOBT

fecal occult blood test

ICC

intraclass correlation coefficient