Calculating averted caries attributable to school-based sealant programs with a minimal data set

Abstract

Objectives

We describe a methodology for school-based sealant programs (SBSP) to estimate averted cavities,(i.e.,difference in cavities without and with SBSP) over 9 years using a minimal data set.

Methods

A Markov model was used to estimate averted cavities. SBSP would input estimates of their annual attack rate (AR) and 1-year retention rate. The model estimated retention 2+ years after placement with a functional form obtained from the literature. Assuming a constant AR, SBSP can estimate their AR with child-level data collected prior to sealant placement on sealant presence, number of decayed/filled first molars, and age. We demonstrate the methodology with data from the Wisconsin SBSP. Finally, we examine how sensitive averted cavities obtained with this methodology is if an SBSP were to over or underestimate their AR or 1-year retention.

Results

Demonstrating the methodology with estimated AR (= 7 percent) and 1-year retention (= 92 percent) from the Wisconsin SBSP data, we found that placing 31,324 sealants averted 10,718 cavities. Sensitivity analysis indicated that for any AR, the magnitude of the error (percent) in estimating averted cavities was always less than the magnitude of the error in specifying the AR and equal to the error in specifying the 1-year retention rate. We also found that estimates of averted cavities were more robust to misspecifications of AR for higher- versus lower-risk children.

Conclusions

With Excel (Microsoft Corporation, Redmond, WA, USA) spreadsheets available upon request, SBSP can use this methodology to generate reasonable estimates of their impact with a minimal data set.

Introduction

The goal of public health programs is to improve population health. Prioritizing interventions with evidence of effectiveness as found in systematic reviews is a first step to ensuring that interventions impact population health. Impact on population health will also depend upon whether the intervention is delivered to persons at risk for the disease and the quality of the intervention delivery. A program's primary mission is delivery of interventions to reduce disease. Collecting data to document health outcomes attributable to interventions can be costly and take needed resources away from a program. It is critical, however, for community-based programs to document their health impact in order to secure and maintain current levels of funding.

In this paper, we describe a methodology using a minimal data set to estimate the impact of school-based sealant programs (SBSP) on averted cavities, calculated as the difference in cavities with and without the program. To estimate cavities without the intervention, we use information on baseline first molar caries severity (decayed or filled first permanent molars) among children screened by the program. To estimate cavities with the intervention, we combine data on caries severity with data on sealant retention. Sealant retention rates provide good evidence of program effectiveness as retained sealants are 100 percent effective (1). Because there is limited evidence for the effectiveness of sealants beyond 9 years (2), our analytic horizon is 9 years. Our approach is similar to other studies that have used data from cross-sectional studies or expert opinion to estimate annual caries levels over several years with and without caries preventive interventions such as dental sealants or water fluoridation (3-5). One simulation of the cost-effectiveness of clinically delivered sealants also used a Markov model to estimate averted cavities (3).

Using this proposed methodology, SBSP will be able to estimate their impact with data collected at baseline screening, at sealant delivery, and at the retention checkup 1 year after placement. We next demonstrate the model using SBSP data. Finally, we examine the sensitivity of averted cavities derived from this methodology if a program over or underestimates their first molar attack rate (AR) or their 1-year retention rate.

Methods

Model

The methodology uses a Markov model with 1-year cycle times over 9 years to simulate caries progression in a permanent first molar (hereon referred to as first molar) with and without sealants. The following assumptions are used in the Markov model:

1. Retained sealants are 100 percent effective (1,6) in preventing caries.

2. The annual first molar AR is constant (7,8). For example, if a child had a 5 percent chance of developing caries in a sound first molar at 8 years of age, then the chance at age 9, 10, 11, and so on, of developing caries in a sound first molar would also be 5 percent.

3. The monthly loss rate after a resin-based sealant has been in place for more than 1 year decreases over time and takes on the functional form of 0.01 × e^{(−0.012×M)} where M equals the number of months since placement (3). For example, if program data indicated that the 1 year loss rate is 10 percent, 18 percent of sealants would be lost at 2 years (average annual loss rate = 9 percent); 34 percent would be lost at 5 years (average annual loss rate = 7 percent), and 45 percent of sealants would be lost at 9 years (average annual loss rate = 5 percent). The Markov cycle rather than the age of the sealant will determine the sealant loss rate.

4. The probability that a first molar that loses its sealant develops caries is the same as that for a first molar that never received a sealant (9).

5. All lost sealants are lost at the beginning of the cycle (i.e., year).

6. All carious teeth develop caries at the beginning of the cycle (from hereon, we will refer to cycles as years).

Because some SBSP may want to use averted cavities estimates from this methodology to calculate their cost effectiveness, we also estimate averted caries discounted at an annual rate of 3 percent as recommended by the Panel on Cost-Effectiveness in Health and Medicine (10).

The cavities averted in each year after sealant placement are the number of new cavities expected that year in the absence of the sealant program less the number of new cavities expected that year in the presence of the sealant program. We first consider the cavities expected without sealants (frame A of Figure 1). In the Markov model, a first molar without a sealant is initially in the sound-unsealed state. In each subsequent year, a first molar can stay in the sound state or move to the caries state. The annual AR is the probability that a first molar moves into the caries state, and thus, 1 minus AR is the probability a first molar remains sound.

Figure 1.

Markov model of caries progression (a) without sealant and (b) with sealant where AR indicates annual first molar (1M) attack rate and R_i indicates sealant retention rate in cycles 1 through 9. Solid black lines indicate transitions that occur in all cycles. Dashed gray lines indicate transitions that occur in cycles 2 through 9.

When estimating cavities expected with a sealant, a first molar is initially in the sound-sealed state (frame B of Figure 1). At the beginning of the first year, a first molar will retain its sealant and remain in the sound-sealed state with probability R₁ (first year retention rate) or lose its sealant and move to either the sound-unsealed state with probability (1 – R₁) × (1 – AR) or the caries state with probability (1 – R₁) × AR. From the second year onward, the model changes in two ways. First, because the sound-unsealed state is now populated; teeth in that state can stay in the sound-unsealed state with probability 1 – AR, or move to the caries state with probability AR. Second, the sealant loss rate changes from (1 – R₁) to 1 – R_i, where i represents years since sealant placement. The loss rate in years 2 through 9, (1 – R_i), equals $1-{\prod }_{M=12(i-1)+1}^{12i}(1-0.01\times {\mathrm{e}}^{(-0.012\ast M)})$; this converts the monthly loss rate to an annual rate. In all years after the first, teeth in the sound-sealed state can stay in that same state with probability R_i or leave that state and move to either the sound-unsealed state with probability (1 – R_i) × (1 – AR) or move to the caries state with probability (1 – R_i) × AR.

Discounted cavities averted in each year after sealant placement is calculated as the cavities averted during that year multiplied by 1/(1.03ⁱ). Summing the discounted cavities averted for each year provides an estimate of discounted cavities averted over the 9-year horizon.

Estimating first molar AR

To estimate averted cavities per first molar sealed, the Markov model requires two pieces of information – the annual first molar AR and the 1-year retention rate. Sealant program administrators can estimate first molar ARs from data collected when they screen children prior to sealant placement. Because first molars typically erupt at age 6 years (11) and we assume that the first molar AR is constant over time (Assumption 2), the annual AR for children of the same age can be calculated with the following equation:

$$FirstMolarA{R}_{Age}=1-{\left(1-\frac{DF1M}{4\times \#screened}\right)}^{\frac{1}{Age-6}}$$ (1)

Where:

• First Molar AR_Age represents annual first molar AR among children at a specified age.

• DF1M represents the number of first molars with cavities (decayed or filled) among children with no sealants at screening prior to sealant placement. We do not include children with evidence of past receipt of sealants because it is likely that most of their first molars have received the preventive benefits of sealants at some point. National data indicate that among 6- to 12-year-olds with at least one sealant, the average number of permanent teeth sealed is 3.23 (12). Including children with at least one sealant would thus underestimate the population risk for caries.

• # screened represents the number of children with no sealants on their permanent first molars at screening (multiplying this value by 4 represents the number of first molars initially at risk for caries).

• Age represents the children's age in years (so Age – 6 represents the time a first molar has been in the mouth); again, the equation is applied separately for each age represented in the program patient population.

The part of equation (1) in parentheses represents the cumulative probability, a never-sealed first molar remains sound during the time it has been in the mouth. To obtain the annual probability that a first molar remains sound, we raise the cumulative probability to 1 divided by the time the first molar has been in the mouth. Subtracting this annual probability from 1 yields the annual first molar AR.

Required data elements to be collected

To estimate the annual first molar AR for a group of children of different ages, sealant programs can calculate the first molar AR for all represented ages and then calculate the weighted average of these first molar ARs. Thus, calculating the first molar AR requires sealant programs to collect the following data at screening prior to sealant placement for each child a) whether child has at least one sealant present in mouth; b) total number of decayed and filled first molars (i.e., 0, 1, 2, 3, or 4); and c) age.

Estimating cavities with and without a sealant program using this methodology requires the first molar AR, the number of first molars sealants placed, and the number of those sealants retained approximately 1 year after placement.

Demonstration of model

To demonstrate the methodology, we used de-identified data from baseline screening by the Wisconsin Seal-A-Smile Program for the 2011-2012 school year, which was collected by the Wisconsin Department of Health Services as part of the program evaluation under Centers for Disease Control and Prevention cooperative agreement 5U58DP001480-05. This use of the data was a nonresearch evaluation of a performance tool intended to improve public health practice. With the exception of retention rate, which was provided by Wisconsin based on past retention findings, values used in the model were obtained by analyzing Wisconsin's data with SAS version 9.3 software (SAS Institute, Inc., Cary, NC, USA). We used an Excel version 14.0.6129.5000 spreadsheet for our simulation model.

Sensitivity analyses

We evaluate the sensitivity of estimated averted cavities to misspecifications of first molar AR and 1-year sealant retention. We first analyze how estimated averted cavities vary for different ARs if the program over or underestimated the AR by 10 percent. For this sensitivity analysis, we assumed a 1-year retention rate of 89 percent, which was obtained from the source of the functional form we used to estimate retention 2+ years after placement (3). We then examine how averted cavities vary for different ARs if the 1-year retention rate were 80, 85, 90, or 95 percent. We conduct all sensitivity analyses at the tooth level (first molar) as compared with the program level (total number of first molars sealed) to make interpretation of the findings more relevant to programs of all sizes.

Results

Demonstrating methodology: estimating annual first molar AR

To demonstrate how to estimate the first molar AR, we used data for the 10,275 children, aged 7-12 years, who were screened by the Wisconsin Seal-A-Smile Program and did not have any sealants. To illustrate how the AR for a specific age group is calculated, consider the 2,909 8-year-olds (Table 1). We first estimated the probability over 2 years of a first molar developing a cavity. The 2-year AR equals 14 percent – the total number of decayed or filled first molars (= 1,632) divided by the number of teeth at risk (4 × 2,909 = 11,636) and then converted to a percent. Alternatively, the probability a first molar stays sound over 2 years is 1 – 0.14 = 0.86. To obtain the annual first molar AR, we first obtain the annual probability a first molar remains sound, which equals the square root of 0.86 (i.e., 0.86^½). We take the square root because the first molar has been in mouth for 2 years (8 years-6 years). This value, which equals 0.93, is then subtracted from 1 and converted to a percent to obtain an annual first molar AR of 7 percent (i.e., over 1 year, seven of every 100 sound first molars will develop a cavity). We did the same calculations for 7-year-olds and 9 through 12-year-olds and then obtained the weighted average first molar AR (weighted by percent of children in each age cohort) of 7 percent (Table 1).

Table 1.

Estimated 1M AR by Age and Weighted Average for Children Aged 7-12 Years: Wisconsin Seal-A-Smile Programs 2011-2012

Reported age	Average time 1M in mouth	Number of children	Decayed and filled 1M among children	Attack rate
				Cumulative	1 year	Weighted*
7	1	3,411	1,035	0.0759	0.0759	0.0252
8	2	2,909	1,632	0.1403	0.0728	0.0206
9	3	1,395	1,190	0.2133	0.0768	0.0104
10	4	1,184	1,345	0.2840	0.0801	0.0092
11	5	937	1,184	0.3159	0.0731	0.0067
12	6	439	557	0.3172	0.0616	0.0026
Total		10,275	6,943			0.0748

^*Weighted by proportion of observations in age group.

AR, attack rate; 1M, first molar.

Demonstrating methodology: estimating averted cavities

To estimate cavities that would occur with and without the sealant program (Table 2), we used the following parameters in the model (Figure 1):

1. The annual first molar AR = 7 percent.

2. The first-year retention rate = 92 percent.

3. Number of first molar receiving sealants =31,324 sealants.

Table 2.

Cavities with and without Sealant Program over 9 Years, Select Wisconsin Sealant Programs 2010-2011^*

	No sealant program		Sealant program
Cycle	Sound unsealed	Caries	Sound sealed	Sound unsealed	Cavities	Averted cavities	Discounted averted cavities
0	31,324	0	31,324	0	0	0	0
1	28,983	2,341	28,912	2,232	180	2,161	2,098
2	26,816	4,508	26,250	4,528	546	1,801	1,697
3	24,812	6,512	24,145	6,137	1,042	1,509	1,381
4	22,957	8,367	22,460	7,237	1,627	1,270	1,128
5	21,241	10,083	21,097	7,957	2,269	1,073	926
6	19,653	11,671	19,984	8,392	2,947	910	762
7	18,184	13,140	19,068	8,612	3,643	773	629
8	16,825	14,499	18,310	8,671	4,344	659	520
9	15,567	15,757	17,677	8,608	5,039	562	431
Total averted cavities						10,718	9,572

^*First molar attack rate and retention in text rounded to two decimal places. Averted cavities calculated using first molar attack rate is equal to 7.475%, and 1-year retention is equal to 92.3%.

Over 9 years, the number of carious first molars without the program was estimated to be 15,757, and with the program, 5,039 (Table 2). Thus, averted cavities attributable to the sealant program would be 10,718 first molars (Table 2). The discounted value of averted cavities would be 9,572 first molars.

Sensitivity analysis

The solid line in Figure 2 shows that estimated cavities averted is higher for higher ARs when the sealant retention rate is held constant (i.e., 89 percent). Additionally, for a given absolute change in AR (horizontal movement), the absolute change in cavities averted (associated vertical movement) is smaller as the AR increases (further to the right in Figure 2). Thus, absolute errors in estimated averted cavities because of over or underestimating the AR will be smaller in the higher range of ARs.

Figure 2.

One-way sensitivity analysis – how averted first molar cavities vary if underestimate or overestimate annual attack rate by 10 percent assuming 1-year retention rate is 89 percent.

This holds true when the relative error (converted to percentage error by multiplying by 100) is also considered. As illustrated in Figure 2, we found that a 10 percent over or underestimate of the annual AR (dotted and dashed lines in Figure 2, respectively) produces an error in estimated averted cavities that is a smaller percentage error at higher AR values. Additionally, this percentage error is always less than 10 percent. For example, if a program's true AR was 3 percent and the program overestimated it by 10 percent (110 percent × 3 percent = 3.3 percent), averted cavities would be overestimated by 8.7 percent (because the estimated value would be 0.175 instead of 0.161). If the AR, however, was estimated to be 19.8 percent when it was actually 18 percent (again, a 10 percent overestimate), averted cavities would only be overestimated by 3.0 percent (because the estimated value would be 0.540 instead of 0.524).

For a given AR, the relative error in estimating cavities averted is exactly the same as the relative error in estimating the 1-year retention rate. This can be seen in Figure 3. For example, if the true retention rate were 80 percent, but was estimated to be 85 percent, retention rate would be overestimated by 6 percent [= 100 × (85 – 80)/80]. The corresponding percentage that estimated averted cavities deviates from actual averted cavities is always 6 percent regardless of the AR. Specifically, Figure 4 shows that for an AR of 15 percent, when retention rate is really 80 percent, but estimated to be 85 percent, cavities averted is overestimated by 0.027 (= 0.466 – 0.439) or 6 percent. Note that the vertical distances (i.e., absolute error in estimates of cavities averted) are larger at higher ARs (i.e.,further to the right in Figure 3) and smaller for lower ARs. However, because the base cavities averted is changing with the absolute change in cavities averted, the relative and percentage errors stay the same across the range of ARs.

Figure 3.

Two-way sensitivity analysis – how averted first molar cavities vary by attack rate and 1-year retention rate.

Figure 4.

Illustration of sensitivity analysis of misestimations for changes in retention rate.

The two-way sensitivity analysis in Figure 3 is presented, so program administrators may see how variations in AR and/or 1-year retention rate affect estimates of disease averted by their programs. The example above illustrates how this analysis may be used to gauge the magnitude of misestimation of averted cavities if retention rate is misestimated. Alternately, it may be used to gauge the magnitude of misestimation of averted cavities if AR is misestimated. Program managers can use it to estimate the effect on disease averted if they reach retention rate improvement goals or successfully target higher risk (AR) populations.

Discussion

In this paper, we have described a methodology based on assumptions from published studies that provides an estimate of averted cavities attributable to SBSP. This methodology requires programs to collect the following data for every child: sealant(s) present, number of decayed or filled first molars, and age at screening prior to sealant placement; number of first molars sealed; and number of sealants retained at 1-year. Programs that cannot collect 1-year retention data for each child can still employ this methodology by estimating retention from a subsample of children served or by using historical data. These programs, however, should keep in mind that biased estimates of 1-year retention will result in biased estimates of averted cavities, and thus, they should consider analyzing the sensitivity of their findings by allowing the 1-year retention rate to take on different values. An Excel spreadsheet that generates the numbers reported in Tables 1 and 2 is available upon request.

Estimated averted cavities can be used to evaluate the quality of sealant delivery. A recent review of current oral health performance measures conducted by a National Quality Forum panel found that in oral health, process measures were abundant but that well defined outcome measures were scarce (13). Among the process measures, utilization measures were most widespread, and the panel voiced concern about the difficulties associated with determining “appropriate” levels of utilization. Estimated averted cavities contains information on both the effectiveness/quality of sealants delivered in a school-based setting (i.e., retention) and whether the program is targeting children at high risk for caries (i.e., first molar AR).

In addition, discounted averted cavities estimated with the methodology can be combined with program data on sealant delivery costs and restoration costs for first molar cavities to estimate cost-effectiveness measured by cost per cavity averted:

$$\frac{\text{Program Cost}-\text{Cost of Restoration}\times \text{Discounted Averted Caries}}{\text{Discounted Averted Caries}}$$

Cost-effectiveness analysis allows funders to estimate the impact of their funding and also allows them to identify those programs that are using their resources to prevent first molar caries most effectively and efficiently.

The accuracy of the estimate for averted cavities obtained with this methodology will depend on how representative the input AR and 1-year retention rate are for children served by SBSP. Our sensitivity analysis indicated that the methodology was more robust to over or underestimates of the AR for higher-risk children who are more likely to be targeted by SBSP (14) than for lower-risk children.

How well this methodology performs will also depend on the accuracy of the six assumptions used to develop the Markov model. Assumptions 5 and 6 (see Methods) will only affect the estimate of discounted averted cavities as they relate to when disease and sealant loss occur not whether they occur. These assumptions result in a more conservative estimate of discounted averted cavities because sealant loss and first molar cavities occur at the beginning rather than the end of each year. Assumptions 1 and 4 have a strong evidence base as they are from National Institutes of Health Consensus Conferences statements (1,6) and a published systematic review (9), respectively. Assumptions 2 and 3 are supported by published studies: sealant retention from a study published in 2005 (3) and constant first molar AR from two studies published in 2002 (7) and 2001 (8). Averted cavities would be overestimated with the methodology if the annual first molar cavities AR decreased with age. We note that programs could also verify whether this assumption holds for children served by their programs by examining the estimated first molar AR for each age group (see Table 1) for which there is a sufficient number of students. If there were a clear trend that the first molar AR decreased with child's year of age, then SBSP could either not use this methodology or revise the annual AR downward accordingly. Finally, averted cavities would be overestimated with this methodology if the functional form of retention 2+ years after placement was constant, for example, instead of exponential.

The AR calculation assumed that first molars erupt at age 6. Because this estimate was obtained from a published analysis of data representative of US children (11), this assumption is likely to be correct for the average US child. If first molars were to erupt later (e.g., first molars erupt at age 6.5 rather than 6 years), then the AR would be underestimated because the teeth were in the mouth for a shorter period of time. For example, using the Wisconsin data, if eruption were at 6.5 rather than 6 years, the first molar AR would be 11 percent rather than the estimated 7 percent, and the resulting averted cavities would be underestimated. However, if the age at first molar eruption was overestimated (e.g., first molars erupt at 5.5 rather than 6 years), the first molar AR would be 6 percent rather than the estimated 7 percent and the resulting estimated averted cavities would be overestimated. Again, this assumption can be checked by examining whether the estimated ARs are similar for all age groups. The impact of mistaking the age of eruption on estimated ARs decreases with each age cohort. For example, again using the Wisconsin data, if first molars erupted at age 6.5 years, then the resulting AR for 6-year-olds would be 15 percent compared with 10 and 9 percent, respectively for 7- and 8-year-olds.

In conclusion, this methodology provides an effective way to measure SBSP impact using a minimal data set.