Sample size estimation and re-estimation of cluster randomized controlled trials for real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation for out-of-hospital cardiac arrests

Abstract

Background

In cluster randomized controlled trials (RCTs) for emergency medical services (EMS) system, we encounter the situation that the actual cluster size and ratio of allocated patients between two groups eventually differ from those used for sample size estimation because of the nature of patient enrollment. In such trials, estimations of effect size of test intervention and intra-cluster correlation coefficient (ICC) used for sample size estimation are also difficult. To improve efficient management on clinical cluster RCTs, we need to understand the effect of such inconsistencies of the design parameters on the type I error rate and statistical power of testing.

Methods

We planned the trial which evaluated the 1-month favorable neurological survival of out-of-hospital cardiac arrest patients with or without real-time feedback, debriefing, and retraining system by EMS personnel. Under the conditions that we possibly encountered in this trial, we examined the effect of inconsistencies in the actual ICC, cluster size, and ratio of patient allocation with those expected for sample size estimation on the type I error rate and power, using simulation studies. We further investigated the contribution of incorporating sample size re-estimation, based on the results of interim analysis of the trial, on the power increase.

Results

This simulation study showed that the inconsistencies of cluster size and patient allocation ratio decreased the power by 5–10% in some cases. In addition, the power decreased by 3–4% when the actual ICC was larger than that expected for sample size estimation. Furthermore, the use of a generalized estimating equation method to evaluate the difference in the 1-month favorable neurological survival between two groups caused inflation of type I error rate. Finally, the increase in power by incorporating sample size re-estimation was limited.

Conclusions

We identified remarkable effects of sample size estimation and re-estimations in a cluster RCT for real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation for out-of-hospital cardiac arrests. The estimation of design parameters for sample size estimation is generally challenging in cluster RCTs for EMS system; therefore, it is important to conduct a trial simulation that assesses the statistical performances under sample sizes based on the various expected values of the design parameters before beginning the trial.

1. Introduction

A cluster randomized controlled trial (RCT) is performed as an intervention in a group of subjects (collectively labeled as a ‘cluster’), rather than for an individual. This design is used in emergency medicine when researchers would like to conduct clinical trials focusing on “cluster” such as fire department and emergency medical services (EMS) station [[1], [2], [3], [4], [5], [6]]. Sample size estimation in cluster RCTs requires the expected effect size of the test intervention, the intra-cluster correlation coefficient (ICC), the cluster size (i.e., the number of treated patients per EMS station), and the ratio of patient allocation between two groups, in addition to the type I error rate and the statistical power of testing [[7], [8], [9]]. However, we often encounter the situation that the actual values of the cluster size and the ratio of allocated patients between two groups eventually differ from those used for sample size estimation because of the nature of patient enrollment [6]. Furthermore, in cluster RCTs for EMS system, the difficulty lies in estimating the effect size of the test intervention and ICC used for sample size estimation, because few estimates of the effect size and ICC from existing research have been published, leaving investigators with little data on which to base their expectations in some cases [10]. To improve efficient management on cluster RCTs, we need to understand the effect of the uncertainty of these design parameters on the type I error rate and power.

Recently, we started an open-label cluster RCT that enrolled 3000 to 5000 patients with out-of-hospital cardiac arrest (OHCA) and included 120 EMS stations from February 1, 2017 in Japan (UMIN000021431) to examine the clinical benefit of real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation (CPR) for OHCAs [11]. The intervention group (high performance CPR group) in this trial refers to CPR that makes use of a real-time feedback device, which quantifies and monitors the quality of CPR (real-time feedback), combined with post-event debriefing and physical retraining. The control group (standard CPR group) is the standard CPR without real-time feedback, debriefing, and retraining using the use of a real-time feedback device. The 120 EMS stations were randomly allocated to high performance CPR group or standard CPR group on a 1:1 ratio. Therefore, the EMS personnel belonging to the same EMS station treat cardiac arrest patients using the allocated procedure. The primary endpoint was the proportion of patients who survive with 1-month favorable neurological survival, defined as cerebral performance category score 1 or 2, for one month after the event.

Before beginning the trial, under the conditions that we might encounter, we comprehensively examined the effect of inconsistencies in the actual ICC, cluster size, and ratio of patient allocation with those expected for sample size estimation on the type I error rate and power using simulation studies. With respect to the uncertainty of the effect size of the test intervention, we investigated the contribution of incorporating sample size re-estimation based on the observed difference in the 1-month favorable neurological survival between the two groups at an interim analysis of the trial for the power increase.

In this study, we share and discuss the remarkable effects of sample size estimation and re-estimations based on our cluster RCT for EMS system, with and without sample size re-estimation. In the following section, we briefly summarize the sample size estimation for cluster RCT, the definition of ICC, and the sample size re-estimation we used in our trial, respectively. Subsequently, we conducted comprehensive simulation study and discussed our findings.

2. Materials and methods

2.1. Sample size calculation for a cluster RCT

In the standard cluster RCT, the sample size is often calculated by,

$$N=Jm=\frac{{\left(Z_{\alpha /2}+Z_{\beta }\right)}^2\left[p_t\left(1-p_t\right)+p_c\left(1-p_c\right)\right]\left[1+\left(m-1\right)\rho \right]}{{\left(p_t-p_C\right)}^2}$$

where p_t and p_c are the expected proportion of 1-month favorable neurological survival in high performance CPR group and standard CPR group, respectively, J is the number of EMS stations, m is the cluster size for each station, and $\rho$ is the ICC [9]. ${\text{Z}}_{\alpha \text{/}2}$ and ${\text{Z}}_{\text{β}}$ are the upper ($\alpha /2$)th and $\text{β}$th percentiles of the standard normal distribution, respectively.

In our trial, the p_t was set to 0.061914 based on data from Osaka prefecture of past 5 years and p_c was set to 0.038490 using odds ratio = 1.65, which was expected by previous research of real-time feedback or debriefing [11]. To calculate the sample size required to achieve an 80% power ($\text{β}$= 20%) at 5% level of significance ($\alpha$= 5%) under this assumption, we further needed to specify the values of J and $\rho$, respectively. The specification of ICC $\left(\text{ρ}\right)$ is described in the following section.

2.2. Intra-cluster correlation coefficient

In this section, we introduce the ICC definition used in our trial. We supposed that the clustered, binary outcome variable (e.g., 1-month survival) is denoted by Y_ij (= 0 or 1) for the jth patient (j = 1, 2, …, m_i) in the ith cluster (i = 1, 2, …, J), where m_i represents the cluster size. The intervention indicator is denoted by a dichotomous variable, X_ij (= 0 or 1) whose value depends only on the cluster; therefore, X_ij = X_i.

The mixed effect and marginal probability models based on the generalized estimating equation (GEE) developed by Liang and Zeger [12] take into account the correlation of patient survival outcome in the inferential processes. We considered the mixed effect model:

$$\text{logit}\left(p_{ij}\right)={\alpha }_1+{\beta }_1{X}_i+U_i$$

where p_ij = E(Y_ij|X_i,U_i), ${\alpha }_1$ is a constant representing the baseline log-odds, and ${\text{β}}_1$ is population-average log-odds. The random variable, U_i is a random effect and is distributed with a mean of 0 and unknown variance ${\sigma }^2$, U_i ∼ Normal (0, ${\sigma }^2$). In the mixed effect model, the ICC (${\rho }_{\text{lo}}$) is defined as ${\sigma }^2$/(${\sigma }^2$+${\text{π}}^2/3$).

In our trial, the ICC estimate was ${\rho }_{\text{lo}}$  = 0.003, which corresponded to the value of ${\sigma }^2$  = 0.01. ${\sigma }^2$ was based on assuming random effect of about ±0.3 relative to β₁. Thus, the required cluster size (or total sample size) when ${\rho }_{\text{lo}}$ = 0.003 was 25 (or approximately 3000) when the number of EMS stations were (J = ) 120.

In addition, we also considered the marginal probability model based on the GEE:

$$\text{logit}\left(p_{ij}\right)={\alpha }_2+{\beta }_2{X}_i$$

The parameter of ‘exchangeable’ compound symmetry correlation structure could be used as the ICC. Furthermore, it should be noted that in the case where the probability of success in a binary endpoint is low, the ICC tends to be low [13]. Eldridge et al. [8] have provided several definitions of ICC in cluster RCTs.

2.3. Sample size re-estimation

As we described earlier, the difficulty lies in the determining the expected difference in the 1-month favorable neurological survival between two groups in the sample size estimation. Although we assumed an expected odds ratio of 1.65, a lower odds ratio (e.g., 1.60 corresponding to proportion of 1-month favorable neurological survival of 3.8% and 5.95% in standard CPR and high performance CPR groups, respectively) would also be clinically significant. Therefore, we planned to conduct an interim analysis to re-estimate the required sample size during the trial. Among the several useful methods for sample size re-estimation, we used the approach proposed by Mehta and Pocock [14] because it does not need to adjust the overall type I error rate of the trials, although careful inspection of the operating characteristics of the Mehta and Pocock (MP) method is required [15,16]. Although we do not explain the MP method in more detail in this paper, the sample size increases when interim analysis results fall within a ‘promising zone’ (e.g., 50–80% of conditional power), where it is deemed worthwhile to increase the conditional power to detect the difference in the primary endpoint between the two groups by adding more patients.

2.4. Simulation studies

2.4.1. Simulation settings

The above-mentioned sample size estimation of 3000 patients (i.e., 25 of cluster size for each EMS station) assuming that p_t = 0.061914, p_c = 0.038490, ${\rho }_{\text{lo}}$  = 0.003, $\alpha$ = 0.05, and $\text{β}$  = 0.20 supposes an equal cluster size between the EMS stations, equal patient allocation between the groups, and a common expected ICC between the groups. These design parameters possibly differ from those observed in the actual trial; therefore, we investigated the type I error rate and statistical power of testing using the mixed effect model and GEE method under the following possible settings that we might encounter with respect to the cluster size, the ratio of patient allocation, and the ICC, with and without incorporating the sample size re-estimation through the simulation studies.

In the simulation studies, the 1-month survival probability was generated based on the mixed effect model such that p_t = exp(-3.231 + 0.497 + u_i)/{1 + exp(-3.231 + 0.497 + u_i)} (or p_c = exp(-3.231 + u_i)/{1 + exp(-3.231 + u_i)}), where u_i ∼ Normal(0, ${\sigma }^2$). The value of ${\sigma }^2$ was set to 0.01 (${\rho }_{\text{lo}}$  = 0.003) or 0.11 (${\rho }_{\text{lo}}$  = 0.032 as conservative setting) for the two groups. Five possible situations for cluster size were considered (Table 1). Situation 1 is ideal wherein the cluster size of all the EMS stations are the same; however, it is apparently impractical. In situations 2–5, the cluster size is varied depending on the EMS stations. Additionally, under situation 1, the power was also evaluated when the ratios of sample sizes for the high performance CPR group and standard CPR group were (1200:1800), (1000:2000), (1750:1275), (1800:1200), (2000:1000), and (1275:1750), respectively.

Table 1.

Five cluster-size situations for each group.

Cluster size	Number of EMS stations	Number of patients	Cluster size	Number of EMS stations	Number of patients	Cluster size	Number of EMS stations	Number of patients
Situation 1			Situation 2			Situation 3
5	0	0	5	6	30	5	2	10
10	0	0	10	6	60	10	4	40
15	0	0	15	7	105	15	8	120
20	0	0	20	7	140	20	10	200
25	60	1500	25	8	200	25	12	300
30	0	0	30	7	210	30	10	300
35	0	0	35	7	245	35	8	280
40	0	0	40	6	240	40	4	160
45	0	0	45	6	270	45	2	90
Situation 4			Situation 5
5	10	50	5	0	0
10	8	80	10	0	0
15	6	90	15	12	180
20	4	80	20	10	200
25	4	100	25	14	350
30	4	120	30	14	420
35	6	210	35	10	350
40	8	320	40	0	0
45	10	450	45	0	0

EMS: emergency medical services.

Using the above-mentioned probabilities and situations, the number of 1-month survivors in each group was generated based on the Binominal distribution (e.g., Binomial (1500, p_t) and Binominal (1500, p_c)) and subsequently, the mixed effect model and the GEE method introduced in the previous section were applied to estimate the odds ratio and its p-value. We performed 10,000 repetitions (i.e., 10,000 simulated trials) for each setting and reported the proportion of simulated trials in which the p-value was less than or equal to 0.05 as the statistical power of testing (or the type I error rate under the null hypothesis) and average of estimated odds ratio. In the case of incorporating sample size re-estimation, we also calculated the proportion of observing the ‘promising zone’ result among 10,000 simulated trials, as well as the average of the final sample size after sample size re-estimation. We used SAS version 9.4 software (SAS Institute Inc., Cary, NC, USA) to implement the simulation studies.

3. Results

3.1. Mixed effect model and GEE

We found that the parameter estimation of the random effect in the mixed effect model did not converge because of the extremely low rates of patient survival in each cluster. The successful parameter estimation of the mixed effect model was less than 50% of 10,000 simulated trials, while it was 100% for the GEE method; therefore, we present the results using the GEE in the following section. We set the main analysis as GEE, and the secondary analysis as mixed effect model.

3.2. Power

Table 2 shows the results of simulation studies with situations of varying ICC and cluster sizes. In the case where the value of ${\rho }_{\text{lo}}$ was identical to the expected value of 0.003 for both groups, and the cluster size was commonly 25 among the 120 EMS stations (i.e., Situation 1), the simulated power of 83% was almost same as the pre-specified target value of 80%. The average odds ratio of 1.68 was also close to the expected value of 1.65. The simulated power decreased by 3–4% when the ICC of the high performance and/or standard CPR groups increased to 0.032; however, those were between 76 and 85% in all the situations of cluster size we assumed in the simulation studies.

Table 2.

Results of simulation studies with situations of varying ICC and cluster sizes.

ICC of standard CPR group	ICC of high performance CPR group	Situation in standard CPR group	Situation in high performance CPR group	Power, %	Odds ratio
0.003	0.003	1	1	83	1.68
0.032	0.032	1	1	76	1.60
0.032	0.003	1	1	77	1.61
0.003	0.003	2	2	84	1.68
0.032	0.032	2	2	84	1.67
0.032	0.003	2	2	77	1.60
0.003	0.003	2	3	84	1.68
0.003	0.003	2	4	84	1.68
0.003	0.003	2	5	84	1.68
0.032	0.032	2	3	84	1.67
0.032	0.032	2	4	84	1.67
0.032	0.032	2	5	85	1.67
0.032	0.003	2	3	77	1.60
0.032	0.003	2	4	77	1.60
0.032	0.003	2	5	77	1.60
0.003	0.003	3	3	84	1.68
0.032	0.032	3	3	84	1.67
0.032	0.003	3	3	77	1.60
0.032	0.003	3	3	77	1.60
0.003	0.003	3	4	84	1.68
0.003	0.003	3	5	84	1.68
0.032	0.032	3	4	84	1.67
0.032	0.032	3	5	85	1.67
0.032	0.003	3	4	77	1.60
0.032	0.003	3	5	77	1.60
0.003	0.003	4	4	84	1.68
0.032	0.032	4	4	84	1.67
0.032	0.003	4	4	77	1.60
0.003	0.003	4	5	84	1.68
0.032	0.032	4	5	85	1.67
0.032	0.003	4	5	77	1.60
0.003	0.003	5	5	84	1.68
0.032	0.032	5	5	85	1.67
0.032	0.003	5	5	77	1.60

ICC: intra-cluster correlation coefficient, CPR: cardiopulmonary resuscitation.

Table 3 shows the results of the simulation studies under conditions of varying the ICC and ratios of patient allocation between the two groups under situation 1 shown in Table 1. When the ICC was different in the two groups, the simulated power decreased by up to 10% from the target value of 80%. Overall, the difference between the target and simulated powers was approximately ±5%.

Table 3.

Results of simulation studies under conditions of varying ICC and ratio of sample size between the two groups.

ICC of standard CPR group	ICC of high performance CPR group	Sample size in standard CPR group	Sample size in high performance CPR group	Power, %	Odds ratio
0.003	0.003	1800	1200	82	1.68
0.032	0.032	1800	1200	83	1.67
0.032	0.003	1800	1200	75	1.60
0.003	0.003	1200	1800	81	1.68
0.032	0.032	1200	1800	83	1.68
0.032	0.003	1200	1800	74	1.61
0.003	0.003	2000	1000	80	1.68
0.032	0.032	2000	1000	81	1.67
0.032	0.003	2000	1000	73	1.60
0.003	0.003	1000	2000	78	1.70
0.032	0.032	1000	2000	79	1.69
0.032	0.003	1000	2000	70	1.62
0.003	0.003	1750	1275	83	1.68
0.032	0.032	1750	1275	83	1.67
0.032	0.003	1750	1275	76	1.60
0.003	0.003	1275	1750	82	1.68
0.032	0.032	1275	1750	84	1.68
0.032	0.003	1275	1750	75	1.61

ICC: intra-cluster correlation coefficient, CPR: cardiopulmonary resuscitation.

3.3. Type I error rate and power of design with sample size re-estimation

In this section, we describe the evaluation of the type I error rate under the null hypothesis (i.e., odds ratio = 1.0) and the power when the odds ratio was 1.60 (i.e., the true odds ratio is lower than the expected odds ratio of 1.65) when using the GEE method in the 10,000 simulated trials, with and without sample size re-estimation. The method of data generation was the same as that described in the previous section. The interim analysis was conducted when the sample size reached 1800 (60%) or 2400 (80%). For simplicity's sake, the cluster size was set to 25 for 60 EMS stations with equal patient allocation. The pre-specified, permitted maximum sample size (n_max) that is often determined by budgetary limitations and feasibility of recruitment was 3500 or 4000.

Table 4, Table 5 show the type I error rates and powers of trials performed with and without sample size re-estimation, along with the average sample size of trials with sample size re-estimation. As shown in Table 4, the type I error rate exceeded the nominal value of 5% by approximately 1% when the ICC between the two groups was different. Furthermore, the inflation was not corrected even if the trial included sample size re-estimation. We also found that the power did not sufficiently increase when using the sample size re-estimation based on the MP method (Table 5). This was likely because the probability that the interim results falls in a ‘promising zone’ was only approximately 15%. Furthermore, even if the sample size re-estimation was performed, increasing the sample size by 500 (or 1000) patients did not result in a satisfactory increase in the power.

Table 4.

Type I error rates for the GEE method with and without sample size re-estimation.

ICC of standard CPR group	ICC of high performance CPR group	n1	nmax	Type I error rate without SSR, %	Type I error rate with SSR, %
0.003	0.003	1800	3500	5.41	5.25
0.032	0.032	1800	3500	5.09	5.33
0.032	0.003	1800	3500	6.04	5.98
0.003	0.003	1800	4000	5.41	5.48
0.032	0.032	1800	4000	5.09	5.57
0.032	0.003	1800	4000	6.04	6.38
0.003	0.003	2400	3500	5.41	5.19
0.032	0.032	2400	3500	5.09	5.26
0.032	0.003	2400	3500	6.04	5.90
0.003	0.003	2400	4000	5.41	5.54
0.032	0.032	2400	4000	5.09	5.67
0.032	0.003	2400	4000	6.04	6.26

GEE: generalized estimating equation, ICC: intra-cluster correlation coefficient, CPR: cardiopulmonary resuscitation, n₁: sample size at interim analysis, n_max: permitted maximum sample size, SSR: sample size re-estimation.

Table 5.

Powers and average sample sizes for the GEE method with and without sample size re-estimation.

ICC of standard CPR group	ICC of high performance CPR group	n1	nmax	Power without SSR, %	Power with SSR, %	Favorable, %	Promising, %	Unfavorable, %	Average of final sample size after SSR
0.003	0.003	1800	3500	80	79	58	16	27	3066
0.032	0.032	1800	3500	73	72	51	16	33	3066
0.032	0.003	1800	3500	72	72	51	15	33	3065
0.032	0.032	1800	4000	80	80	58	15	27	3101
0.032	0.003	1800	4000	73	73	50	17	33	3111
0.003	0.003	2400	3500	72	73	51	16	33	3108
0.032	0.003	2400	3500	80	79	64	11	25	3042
0.003	0.003	2400	4000	73	72	56	12	31	3047
0.032	0.032	2400	4000	72	72	56	12	31	3048
0.032	0.003	2400	4000	80	80	65	11	24	3068

GEE: generalized estimating equation, ICC: intra-cluster correlation coefficient, CPR: cardiopulmonary resuscitation, n₁: sample size at interim analysis, n_max: permitted maximum sample size, SSR: sample size re-estimation.

4. Discussion

The power decreased by 5–10% in some cases when the cluster size and patient allocation ratio observed in the actual trial were different from those expected in the sample size estimation. The power also decreased by 3–4% when the actual ICC was larger than that expected for sample size estimation. In addition, the use of the GEE method to evaluate the difference in the 1-month favorable neurological survival between two groups yielded inflation of the type I error rate by up to 1.0%. This tendency was also observed in other research studies [13]. According to the results of the simulation studies conducted by Heo and Leon [13], the GEE method resulted in the highest type I error rates when the ICC = 0. Finally, the increase in power produced by incorporating sample size re-estimation was limited.

5. Conclusions

We determined the remarkable effects of sample size estimation and re-estimations in a cluster RCT for real-time feedback, debriefing, and retraining system of cardiopulmonary resuscitation for out-of-hospital cardiac arrests. We used of the GEE method for cluster RCT, and the inflation in type I error rate was adjusted. The misspecifications of the design parameters for sample size estimation possibly decreased the statistical power of testing by 5% on average in our simulation studies; however, the effect may vary depending on the situations that the investigators encounter. The precise estimation of design parameters for sample size estimation is generally challenging in cluster RCT for EMS system; therefore, it is important to conduct a trial simulation that assesses the statistical performance under a sample size based on the various expected values of the design parameters before beginning the trial.