Cluster Effects in a National Dental PBRN Restorative Study

Abstract

Items in clusters, such as patients of the same clinician or teeth within the same patient, tend to be more similar than items from different groups. This within-group similarity, represented by the intraclass correlation coefficient (ICC), reduces precision, yielding less statistical power and wider confidence intervals, compared with non-clustered samples of the same size. This must be considered in the design of studies including clusters. We present ICC estimates from a study of 7,826 restorations placed in previously unrestored tooth surfaces of 4,672 patients by 222 clinicians in the National Dental Practice-Based Research Network, as a resource for sample size planning in restorative studies. Our findings suggest that magnitudes of ICCs in practice-based research can be substantial. These can have large effects on precision and the power to detect treatment effects. Generally, we found relatively large ICCs for characteristics that are influenced by clinician choice (e.g., 0.36 for rubber dam use). ICCs for outcomes within individual patients, such as tooth surfaces affected by a caries lesion, tended to be smaller (from 0.03 to 0.15), but were still sufficiently large to substantially affect statistical power. Clustering should be taken into account in the design of oral health studies and derivation of statistical power estimates for these studies (ClinicalTrials.gov, NCT00847470).

Introduction

Oral health practice-based research often includes multiple patients from the same dental practice and multiple teeth or restorations within the same patient, resulting in clusters of correlated observations. Clustering typically reduces precision of estimation, yielding lower statistical power and wider confidence intervals compared with studies of equal sample size but without clusters. This reduced power is termed the ‘design effect’ (DE). The DE must be considered in the planning of oral health studies to ensure adequate statistical power.

DE depends on the similarity among outcome observations within the same cluster, compared with that among observations in different clusters, and on the average cluster size. The intraclass correlation coefficient (ICC) represents the proportion of the total variability due to variation among clusters (Donner and Klar, 2000). Practically, DE is the ratio of the sample size required for a clustered design to that of a design with independent samples to achieve the same power. For example, to obtain equal statistical power, a clustered design with DE = 2 requires twice as many observations as a non-clustered design. To calculate the sample size needed for a planned study, the DE for the anticipated cluster sizes and ICC must be considered. In study designs with more than one level of clustering, separate estimates of ICC and DE can be calculated for each level of clustering, and used sequentially to adjust the required sample size. An example of this would be a study including multiple teeth per patient and multiple patients per practice.

Most classic statistical techniques are based on the assumption of independent sampling. Because observations within clusters are not independent, these methods may not apply. To obtain valid statistical tests, the analysis must account for clustering. The use of appropriate statistical methodology for clustered studies seems to be increasing in research publications, and standards for reporting results from clustered designs have been developed (Campbell et al., 2004, 2012). However, a methodological review (Simpson et al., 1995) found that only 12 of 21 cluster-randomized studies evaluated used analytic methods accounting for clustering. In a study of the quality of 23 cluster-randomized studies, only 15 (65%) studies had accounted for clustering in sample size calculations, 18 (78%) accounted for clustering in the analysis, and 8 reported estimates of ICC (Froud et al., 2012). A similar report including 559 published dental studies reported that only 39% appropriately accounted for clustering (Fleming et al., 2013).

Studies in healthcare settings often include hierarchical, clustered designs (Dickinson and Basu, 2005). Cluster randomization is particularly common in oral health research, where interventions may be allocated at the patient level or at the dental practice level, rather than at the level of individual observations, such as individual teeth or individual dental restorations (Froud et al., 2012). The National Dental Practice-Based Research Network (National Dental PBRN) is a consortium of dental practices and dental organizations focused on improving the scientific basis for clinical decision-making (Gilbert et al., 2013). The network was funded in 2012 and builds upon the former regional dental PBRNs, including the “Dental Practice-Based Research Network (DPBRN)” (Gilbert et al., 2008), which existed from 2003-2012. Several studies conducted in the DPBRN and the National Dental PBRN, while not randomized studies, include hierarchical sample designs with clustered observations, such as dental restorations or tooth surfaces clustered within patients, patients clustered within practices, and practices clustered within large dental practice groups or geographical regions.

Despite the apparent trend for increased publication of ICC estimates, such estimates for particular research areas may not be available. Because this information is needed for accurate sample size calculation, published information on ICC values from clustered designs is important when such studies are designed. Therefore, the aim of this study was to provide estimates of ICC for a variety of variables collected in a study of restorations placed in previously unrestored tooth surfaces by dental practitioners enrolled in the network.

Materials & Methods

Participants

The network has a wide representation of practice types, treatment philosophies, and patient populations, including diversity regarding race, ethnicity, geography and rural/urban area of residence of both its practitioners and their patients. Analyses of these characteristics confirm that network dentists have much in common with dentists at large (Makhija et al., 2009a), while also offering substantial diversity in these characteristics (Makhija et al., 2009b).

Details of the network’s study “Reasons for placement of the first restoration on previously unrestored permanent tooth surfaces” have been described elsewhere (Nascimento et al., 2010). Briefly, this was a cross-sectional study conducted by network dental practitioners in their offices. Clinicians recorded data on consecutive patients who had a restoration placed on a previously unrestored permanent tooth surface. The clinician could enroll up to 4 restorations per patient and collected data until information on 50 restorations was obtained. The total sample included 229 dental practitioners, placing a total of 9,890 restorations in 5,810 patients. Restorations for pediatric patients were not included in this analysis, yielding an analytic sample of 7,826 restorations in 4,672 patients, placed by 222 practitioners. A consecutive patient log was maintained to record information on each eligible restoration regardless of whether the lesion was enrolled. Copies of data collection forms are available at http://www.dpbrn.org/users/publications/supplement.aspx. The study protocol was approved by the Institutional Review Board of the University of Alabama at Birmingham and by each of the network’s regional boards.

Variables

Five patient characteristic variables were obtained: gender, age, race (coded as white or other for the calculation of ICC estimates), ethnicity (Hispanic or not Hispanic), and dental insurance coverage (coded yes or no).

Thirty-four variables were related to characteristics of individual restorations. For each restoration, clinicians indicated: the tooth number; which tooth surfaces were involved in the restoration (occlusal, mesial, distal, buccal or facial, lingual or palatal, incisal); reason for placing the restoration (caries or non-caries defect); use of specific diagnostic techniques (clinical assessment, radiographs, optical techniques); pre- and post-operative lesion depth estimates; restorative materials used (amalgam; directly placed resin composite; indirect resin composite; glass ionomer; ceramic or porcelain; cast gold or other metal; combined metal/ceramic; temporary restorative material); reasons for restoration of non-carious defects (abfraction/abrasion/erosion; developmental defect or hypoplasia; cosmetic reason; restore endodontically treated tooth; tooth fracture; other), use of base, lining, or bonding material (none; resin-based; glass ionomer; calcium-hydroxide-based; varnish; other); and whether or not a rubber dam was used when the restoration was placed.

Lesion depths were coded as ordinal variables. Missing values were possible for the two depth variables, placement reason and use of rubber dam. The other characteristics were recorded as separate dichotomous variables, coded as 1 if the clinician checked the corresponding box on the form or 0 if the characteristic was not indicated. For these variables, sample size was the total number of restorations enrolled; omitted responses would not cause a missing data value.

Statistical Methods

Estimates of ICC were calculated by a mixed-model analysis of variance (ANOVA) approach, implemented with the PROC MIXED procedure of the SAS^® Release 9.2 software (SAS Institute, Inc., Cary, NC, USA).

Separate analyses were conducted for each of the restoration-level variables. ICC estimates were calculated as ratios of the appropriate covariance parameter estimates, obtained based on the restricted maximum likelihood (REML) estimation, for two hierarchical models for each variable. In the first approach, a two-level model was used with clusters representing individual practitioners. ‘Practitioner’ was included in the model as a random effect, and ‘restoration’ as the within-cluster, or Level 1, unit. This yields a single ICC estimate reflecting similarity of observations on restorations treated by the same practitioner. A compound symmetric correlation structure was specified for this model. We calculated ICC by dividing the covariance parameter estimate for practitioner by the sum of the practitioner and residual covariance estimates. The second approach used a three-level model, with restorations as Level 1, patients as Level 2, and practitioner as Level 3, respectively. This approach yields two ICC estimates, one representing similarity of restorations within the same patient, and the other representing similarity of restorations in different patients made by the same dentist. A ‘variance components’ model was specified, with ‘practitioner’ and ‘patients within practitioner’ as random effects. We calculated the within-patient ICC by dividing the covariance estimate for the patient by the sum of the patient and residual covariance estimates. The practitioner-level ICC was calculated as the covariance estimate for practitioner divided by the sum of the covariance estimates for practitioner, patient, and residual terms. Region was included as a fixed effect in both modeling approaches, to avoid inflation of the ICC due to differences among region means. Adjustment for patient-level characteristics was not attempted. Confidence intervals were not calculated for the ICC estimates, due to the dichotomous or ordinal nature of the variables that were considered. The prevalence for each dichotomous variable was calculated, since ICC depends on prevalence for variables of this type (Crespi et al., 2011).

Results

The analysis included 7,826 restorations placed by 222 dentists in 4,672 patients. Dentists enrolled a mean of 35.3 restorations from a mean of 21.1 patients, yielding a mean of 1.7 restorations per patient. The numbers of restorations enrolled per dentist ranged from 1 to 71. The first quartile was 25 restorations, median 37.5, and the third quartile was 46. Patients received 1 (n = 2,710; 58.0%), 2 (n = 1,118; 23.9%), 3 (n = 496; 10.6%), or 4 (n = 348; 7.5%) restorations.

The overall prevalence of positive responses for each of the variables is shown in Table 1. These ranged from 0.3% for the use of cast gold or base metal to 67.3% for the use of clinical assessments in diagnosis. Quartiles of prevalence for dichotomous variables, calculated for individual practitioners, are shown in Table 2.

Table 1.

Numbers of Observations (N), Prevalence (%), and Intraclass Correlation Coefficients (ICC) for Two Hierarchical Models

			Two-level Model	Three-level Model
	Na	Prevalence (%)	ICC	Level 2 ICC (within Dentist)	Level 3 ICC (within Patient)
Restoration-level Dichotomous Variables
Tooth Surfaces
Occlusal surface	7,826	38.9%	0.1473	0.1309	0.5218
Mesial surface	7,826	26.3%	0.0327	0.0282	0.1452
Distal surface	7,826	31.4%	0.0378	0.0310	0.1861
Buccal surface	7,826	29.1%	0.0704	0.0511	0.4842
Lingual surface	7,826	15.0%	0.0711	0.0597	0.3891
Incisal surface	7,826	5.5%	0.0665	0.0337	0.5626
Main Reason for Restoration
Caries	7,781	82.7%	0.1512	0.1205	0.7617
Diagnostic Techniques
Clinical assessment	7,826	67.3%	0.1485	0.1267	0.6174
Radiograph	7,826	50.2%	0.1343	0.1257	0.5520
Optical technique	7,826	5.1%	0.3310	0.3135	0.6072
Reasons for Restoration of Non-caries Lesion
Abrasion/Abfraction/Erosion	7,826	9.3%	0.1538	0.1063	0.7321
Developmental defect	7,826	0.6%	0.0215	0.0071	0.4415
Cosmetic reasons	7,826	2.3%	0.1486	0.0945	0.6468
Restore endodontically treated tooth	7,826	0.6%	0.0581	0.0599	0.6839
Fractured tooth	7,826	5.3%	0.0809	0.0088	0.7851
Other reason	7,826	1.4%	0.0645	0.0300	0.7295
Use of Base, Lining, or Bonding Material
No base or lining material	7,826	27.1%	0.3923	0.3789	0.5955
Resin	7,826	48.1%	0.3872	0.3687	0.6761
Glass ionomer	7,826	9.9%	0.2859	0.2812	0.4127
CaOH-based cement/liner	7,826	4.3%	0.2807	0.2491	0.4756
Varnish	7,826	3.8%	0.4154	0.3996	0.5323
Other	7,826	11.2%	0.4207	0.4242	0.5989
Restorative Material Used
Amalgam	7,826	33.3%	0.1881	0.1594	0.6735
Direct resin	7,826	58.5%	0.2266	0.1998	0.6796
Glass ionomer	7,826	3.9%	0.1605	0.1235	0.5498
Ceramic or porcelain	7,826	0.6%	0.2029	0.1964	0.9382
Cast gold or base metal	7,826	0.3%	0.0255	0.0316	0.8738
Combined metal/ceramic	7,826	0.9%	0.1521	0.1590	0.7353
Temporary restorative material	7,826	1.2%	0.0823	0.0808	0.6402
Use of Rubber Dam
Rubber dam used	7,595	12.4%	0.3581	0.3792	0.7057
Restoration-level Ordinal Variables
Estimated Depth of Lesion
Pre-operative depth assessment	6,393		0.1818	0.1673	0.3831
Post-operative depth assessment	6,501		0.1809	0.1671	0.3284
Patient Variables
Age (yrs)	4,672		0.1364	–	–
Gender (Male)	4,658	45.7%	0.0043	–	–
Race (White)	4,340	85.5%	0.1980	–	–
Ethnicity (Hispanic)	4,406	4.7%	0.1182	–	–
Dental Insurance	4,661	77.6%	0.3124	–	–

^aVariation in sample sizes is due to the presence of missing values for some variables and whether the variable of interest is at the restoration level or the patient level.

Table 2.

Prevalence^a of Dichotomous Restorations and Patient Variables among 222 Dentists

	Prevalence (%) by Dentist
	25th Percentile	Median	75th Percentile	Maximum
Restoration Variables
Tooth Surfaces
Occlusal surface	21.1	38.7	56.3	100.0
Mesial surface	18.3	26.0	33.3	60.0
Distal surface	21.1	29.6	38.9	84.0
Buccal surface	18.0	26.4	37.8	100.0
Lingual surface	5.7	12.0	20.0	100.0
Incisal surface	0.0	2.4	7.7	35.7
Main Reason for Restoration
Caries	72.2	88.4	97.8	100.0
Diagnostic techniques
Clinical assessment	54.2	67.0	84.6	100.0
Radiograph	35.5	50.0	67.6	100.0
Optical technique	0.0	0.0	3.3	97.5
Reasons for Restoration of Non-caries Lesion
Abrasion/Abfraction/Erosion	0.0	4.1	13.3	82.6
Developmental defect	0.0	0.0	0.0	9.6
Cosmetic reasons	0.0	0.0	0.0	54.3
Restore endodontically treated tooth	0.0	0.0	0.0	17.9
Fractured tooth	0.0	2.0	7.5	40.7
Other reason	0.0	0.0	0.0	24.4
Use of Base, Lining, or Bonding Material
No base or lining material	0.0	10.8	43.9	100.0
Resin	17.5	50.0	87.0	100.0
Glass ionomer	0.0	0.0	10.6	100.0
CaOH-based cement/liner	0.0	0.0	2.7	100.0
Varnish	0.0	0.0	0.0	78.9
Other	0.0	0.0	4.2	98.1
Restorative Material Used
Amalgam	0.0	26.0	55.3	100.0
Direct resin
Glass ionomer	0.0	0.0	4.3	57.1
Ceramic or porcelain	0.0	0.0	0.0	37.5
Cast gold or base metal	0.0	0.0	0.0	10.4
Combined metal/ceramic	0.0	0.0	0.0	42.2
Temporary restorative material	0.0	0.0	0.0	33.3
Use of Rubber Dam
Rubber dam used	0.0	0.0	5.9	100.0
Patient Variables
Gender (Male)	38.3	47.9	56.3	100.0
Race (White)	75.8	91.4	100.0	100.0
Ethnicity (Hispanic)	0.0	0.0	6.3	100.0
Dental insurance	66.7	86.0	97.6	100.0

^aThe prevalence value is the percentage of a dentist’s enrolled restorations having a non-zero response for the item.

ICC values for tooth surfaces ranged from approximately 0.03 to 0.06 in two-level models, for all except the occlusal surface, which showed an ICC of 0.15 (Table 1). Three-level models showed similar values for Level 2 (within-practitioner) clusters, with higher values for Level 1 (within-patient) clusters. ICC ranged from 0.15 for the mesial surface to 0.56 for the incisal surface.

Main reasons for the placement of restorations showed a moderate ICC of 0.15 within practitioners. The three-level model showed an ICC of 0.11 within practitioners and 0.73 within patients.

The use of base, lining, or bonding materials showed an ICC ranging from 0.28 for CaOH-based cement/liner to 0.42 for varnish and for other material. In three-level models, these also showed high correlations of 0.25 to 0.42 within practitioners. Restoration materials showed ICCs from 0.16 for glass ionomer to 0.23 for direct resin. Respective prevalence values for these two materials were 3.9% and 58.5%. The lesion depth assessment measures showed similar ICC estimates of 0.18. Within-practitioner ICC was approximately 0.17 for both measures. Within-patient ICC was 0.38 for pre-operative and 0.33 post-operatively.

Practitioners reported using a rubber dam in 12.4% of restorations. The two-level model showed ICC = 0.36 within dentist, and the 3-level model showed ICC = 0.38 within practitioner and 0.71 within patient.

Discussion

The variables analyzed in this study might serve as either outcome or process variables, depending on the research question that is to be addressed. Besides the outcome variable, the research context (i.e., data collection in dental practices), sampling method, and participant characteristics may influence the correlation in a study. This suggests that the distribution of ICC values across multiple variables may be useful in study planning (Adams et al., 2004). ICC estimates have not been common in the scientific literature (Bland, 2000). Recently, estimates of ICC for patient- and practice-level characteristics have been published for demographic and behavioral variables in medical practice–based networks (Thompson et al., 2012). Estimates have also been published for cancer screening outcomes (Hade et al., 2010), the WHO Global Survey on Maternal and Child Health (Taljaard et al., 2008), community-based trials (Janjua et al., 2006), and primary care settings (Adams et al., 2004). Standards for reporting ICC estimates have been proposed (Campbell et al., 2004). For binary outcomes, generalizability is enhanced by presenting the prevalence of the outcome, since ICC for binary variables is related to prevalence (Gulliford et al., 2005; Crespi et al., 2011). Variables based on dentists’ choices, such as the use of particular restorative materials or diagnostic techniques, tended generally to show higher ICC than more restoration-specific variables, such as tooth surface.

As expected, restorations within patients were more highly correlated than restorations in different patients within the same dentist. High DE may result from either large numbers of observations within clusters or from high values of ICC. In this study, the maximum number of restorations per patient was 4, but the mean was only 1.7. Thus, the number of within-practitioner observations was the more important driver of the magnitude of ICC found in the two-level analysis, and of the resulting DE. If the two ICC estimates from the three-level model are used to calculate separate values of DE for each level, even the highest within-patient ICC observed in this study, 0.76 for primary caries given as the reason for restoration, would result in a Level 1 DE of only 1.5, given the small average number of restorations per patient. In a three-level study design, the three-level estimates could be used with expected sample sizes for each level in a proposed study to calculate an overall ICC and DE. Alternatively, separate effective sample sizes could be calculated for each level to obtain the effective total sample size.

The effect of clustering on sample size requirements can be substantial. Failure to account for clustering typically leads to underestimation of the needed sample size. If the cluster size is large, even low values of ICC have considerable impact. A study with clusters of 50 observations would require double the sample size relative to a study without clusters if ICC equals 0.02.

For ordinal and dichotomous variables, the ICC is analogous to the kappa statistic, and varies with the prevalence of the outcome, tending to increase with higher prevalence (Taljaard et al., 2008). The variables reported in this study may be considered process variables: characteristics of the patient or measures that are at least partially under the control of the practitioner. Process variables typically show larger correlations than would be expected for outcome variables (Taljaard et al., 2008). In the design of a cluster-sampled study, the ICC for the outcome variable, rather than process variables or covariates, would be used to determine the needed sample size. However, ICC for non-outcome variables can provide information regarding the general level of similarity among members of the same cluster.

There are potential limitations to this study. The study was not designed for the estimation of ICC, so neither the number of clusters nor the numbers of patients or restorations per cluster were specified. This resulted in relatively large variation in cluster size, which has the potential to bias the ICC estimates. Several items for which data were collected showed low prevalence in this sample, possibly leading to imprecision or bias in the estimation of ICC. No measures of precision were calculated for the ICC estimates, since these are not available from the estimation method that was utilized. While methods of interval estimation of ICC for variables that are not normally distributed exist, these typically use computer-intensive methods such as resampling. This is an area that is currently not settled in the field of statistical analysis. Previous studies have shown that network dentists have much in common with dentists at large (Makhija et al., 2009a), so we speculate that the distributions of dentist and practice characteristics in our sample are similar to those in the general population of dentists. This similarity may increase the generalizability of these estimates, as well as identifying possible adjustment variables for multivariable analyses, which could reduce the effect of dependent variable clustering and thus improve study power (Yelland et al., 2011). This study focused on restorations, with a limit of 4 per patient. Other sampling approaches might show substantially different correlations, so caution is warranted in using these estimates in different study contexts.

In summary, we found relatively large ICCs for characteristics that are influenced by clinician choice. ICCs for outcomes within individual patients, such as tooth surfaces affected by a caries lesion, tended to be smaller, but were still sufficiently large to substantially affect statistical power. Clustering should be taken into account in the design of oral health studies and derivation of statistical power estimates for these studies.