Validity of a Self-reported Periodontal Disease Measure

Abstract

Objectives

The purpose of this study was to determine the validity of a self-reported periodontal disease measure for use in the Health Professionals Follow-up Study.

Methods

Participating dentists responded to the question “Have you had periodontal disease with bone loss?” Radiographs obtained from 140 participants were evaluated for bone loss at 32 posterior sites and used as the standard. A site was positive if it had bone loss >2 mm and/or complete loss of crestal lamina dura. To avoid falsely classifying participants as positive, three blinded examiners independently evaluated each participant’s radiographs. An a priori decision rule was used to classify a participant positive if all examiners independently assessed the same two or more sites positive.

Results

The validity of the self-reported measure was good among dentists, with positive and negative predictive values of 0.76 and 0.74, respectively. Among nondentists, the self-reported measure showed discriminatory power by confirming associations with known risk factors such as age and smoking.

Conclusions

Dentists have a good perception of their periodontal status, and there is reasonable consensus among dentists regarding the threshold for defining periodontal disease. Self-reported measures might have potential for use in studies of other populations with substantial cost reduction, and deserve further evaluation.

Self-reported periodontal measures rarely have been used, as the general population is believed to be poorly informed about their periodontal status. Clinical or radiographic measures are likely to give better estimates of periodontal status. However, self-reported measures have great practical advantages in terms of cost, time, and convenience, and may be the only feasible measure obtainable from large study populations. A few studies have validated other self-reported oral health measures showing that people generally are well able to report their number of teeth (1), but have a low to moderate perception of other oral health measures (2–8). Only one study in the literature reported using a self-reported periodontal measure. However, the validity of the measure was not assessed (9).

This paper describes the validity of a self-reported measure of periodontal disease history among dentists participating in the Health Professionals Follow-up Study (HPFS). This measure is being evaluated for use in studying the associations of periodontal disease with diet and systemic diseases in this population. It is reasonable to expect dentists to value and maintain good dental care, be well aware of their dental condition, and thus be knowledgeable and able to report their periodontal status. Hence this initial validation study was limited to dentists because the validity is expected to be lower among other health professionals.

The self-reported measure consisted of the response to a single question: “Have you had periodontal disease with bone loss?” Since almost all adults have experienced some amount of bone loss, a threshold is generally defined in order to have a measure with some discriminatory power; hence, mild disease usually is ignored (10, 11). There is no universally accepted threshold for periodontal bone loss and no definition was provided in the questionnaire. The threshold was left to the judgment of each participant. Since the self-reported measure was based on diagnoses by different dentists across the country, the validity of this measure provides an estimate of the degree of consensus among dentists regarding the threshold for defining periodontal bone loss. Validation of this measure also provides an estimate of the periodontal status of this population of US dentists.

We assessed the validity of this measure using radiographs as the standard because they provide a feasible assessment of bone loss (12,131), and compare well with the true bone loss status (14,15). We also examined the associations of the self-reported measure with known risk factors such as age and smoking to see if the measure had discriminatory power among both dentists and nondentists in the HPFS. In addition, we compared the association between age and periodontal disease (as a measure of predictive validity) using the self-reported measure in the HPFS with the same association in another study that used an extensive radiographic measure of periodontal disease.

Methods

The HPFS was launched in 1986 primarily to study the dietary etiologies of coronary heart disease and cancer. The baseline questionnaire incorporated a single question on periodontal disease: “Have you had periodontal disease with bone loss?” Participants included 52,000 male health professionals, 58 percent of whom were dentists. This population was chosen to provide high response rates, good-quality data, and a relatively homogeneous socioeconomic group with minimum potential for confounding due to educational or economic factors.

In 1989 a subset of the dentists participating in the HPFS who completed the 1986 and 1988 questionnaires and responded to the periodontal question were considered for enrollment in a validation study (16). From this population, 250 participants who reported a positive periodontal disease history and 250 participants who reported a negative history were randomly selected. Supplemental questionnaires were sent to this sample in 1989 to obtain the name and address of the dentist who had their radiographs, and to obtain additional information pertaining to their oral health. Since bitewings are routinely obtained in clinical practice, participants were not asked to have radiographs taken specifically for this study; rather, existing posterior bitewing radiographs acquired prior to 1986 were requested. A supplemental bone loss measure was obtained from the questionnaire by asking participants to classify their periodontal status in 1986 as one of the following: (1) no bone loss, (2) mild bone loss, (3) moderate bone loss, (4) severe bone loss, or (5) don’t remember.

A recorder mounted the selected set of radiographs on standard mounting frames to avoid ambiguity and disagreements related to the orientation of the radiographs. When more than one set of radiographs was provided, one set of readable radiographs, preferably bitewings, closest to 1986 was selected for each participant. A radiograph viewing box, Vernier calipers, a millimeter scale, and a magnifying glass were used to aid radiograph assessment in a radiograph reading area with subdued ambient lighting. Using a slightly modified visual categorization method (17), each site was assigned a score as defined in Figure 1. A site was considered positive if it had a score of 2 or more, as up to 2 mm bone loss may be within a normal range (18). The most coronal point of bone was used as the crestal landmark, and in the presence of crowns or restorations, clinical judgment was used to estimate the probable location of the cemento-enamel junction. If the same site was seen on more than one radiograph, then the most severe reading was used. Unreadable sites and missing teeth were noted.

FIGURE 1.

Radiographic Assessment Protocol Used to Classify Each Site According to Bone Level, Measured as Distance from Cemento-enamel Junction to Alveolar Crest and Degree of Loss of Crestal Lamina Dura

The sensitivity and specificity of an individual reading of bone loss for a single site on bitewing radiographs have been reported as 86 percent and 69 percent, respectively (19). To avoid falsely classifying participants as positive based on a single false positive reading on any site, radiographic assessment was improved by using multiple examiners and the following a priori decision rule: a participant was positive for periodontal disease only if all three examiners agreed to the same two or more sites being positive based on independent evaluations of the radiographs for each participant. The examiners consisted of three licensed dentists—including a radiologist, a periodontist, and a public health dentist—all of whom are highly experienced in interpreting radiographs. As these dentists were likely to have different perspectives due to their different specialties, their errors were less likely to be correlated. All three examiners simultaneously participated in three training and standardization sessions.

Each session consisted of sets of radiographs for 20 participants to be read independently by an examiner. The examiners and recorder were blinded to the participants’ reported periodontal status. Each participant was assessed at all posterior inter-proximal sites present, excluding third molars, for a total of 32 possible sites. After the first examiner had completed one session, the recorder noted the missing teeth on the corresponding forms of the other two examiners to avoid disagreements due to tooth identification. The mounted sets of radiographs within the batch were shuffled prior to each session to get a random ordering. A few radiographs from each batch were incorporated into subsequent batches and were re-read by each examiner to obtain measures of within examiner reliability.

To assess possible bias due to non-response, the distribution of participants with respect to age, smoking, and number of teeth was computed for the HPFS dentists and for the validation sample. The validation sample was selected to be evenly divided on their self-reported periodontal status. To compensate for the higher prevalence of periodontal disease in this sample, the estimates for the validation study sample were standardized to the distribution of the self-reported periodontal status in the population of the HPFS dentists.

Associations of the self-reported measure with factors known to be associated with periodontal disease were explored among HPFS dentists and nondentists by comparing the prevalence of self-reported periodontal history in subgroups defined by age, smoking, and number of teeth.

Measures for between and within examiner reliability, namely percent agreement and kappa, were computed for each site for the dichotomized measure. Correlation coefficients between and within examiners for a continuous radiographic bone loss measure, obtained by averaging the ordinal site-level measure across all sites of a subject, and for number of positive sites were computed.

Validity estimates of the self-reported periodontal history in the 1986 HPFS questionnaire were computed using the participants’ radiographic assessments as the true measure according to the a priori decision rule. The self-reported questionnaire measure was tabulated against this dichotomous participant measure to compute predictive values. Predictive values were computed for subgroups defined by age, smoking, and type and timing of radiographs to assess the factors affecting validity. Validity estimates by timing of radiographs were computed to determine the extent of error in the standard caused by the fact that the radiographs were exposed at various times between 1976–89. An additional assessment of validity was obtained by comparing the distribution of the continuous radiographic bone loss measure (obtained by averaging the ordinal measure across all sites and all examiners) among people with and without self-reported periodontal disease and across levels of bone loss self-reported in the supplemental questionnaire. A slightly different and more stringent self-reported periodontal measure was obtained in the 1988 HPFS questionnaire—“Have you had professionally diagnosed periodontal disease?”—also was assessed for its validity.

Associations between periodontal disease and age among males were compared between the HPFS dentists and participants in the US Department of Veterans Affairs Dental Longitudinal Study (VA DLS) to compare the predictive validity of the self-reported HPFS measure with an extensive radiographic measure. The study characteristics and radiographic bone loss measures of the VA DLS have been described previously (20). The age-specific HPFS prevalence estimates were adjusted for misclassification in the self-reported measure, as assessed in this validation study. The adjusted prevalence was computed as the sum of (1) the percent of reported positives expected to be positive on the standard, and (2) the percent of reported negatives expected to be positive on the standard. The adjusted prevalence thus can be interpreted as the true positives, or percent of participants with two or more sites where all three examiners gave a score of 2 or higher. Positive history was defined for the VA DLS population as percent of participants with Schei scores (21) (inter-proximal bone loss as a percent of the radiographic root length or maximum bone height) of 20 percent or more on two or more teeth among the same 16 teeth used in the HPFS. Since the VA DLS measure could not be adjusted for misclassification, a more stringent cut-off was used. An overall prevalence was computed for the HPFS and VA DLS populations, standardized to the age distribution of the HPFS.

Since only posterior bitewings were requested, participants could be falsely classified as negative if they had bone loss only in the anterior teeth. Additional analyses on the VA DLS data were carried out to estimate the extent of this error. Among the VA DLS participants with no bone loss in the posteriors (less than two sites with ≥20% bone loss), about 13 percent had bone loss in anteriors (two or more sites with ≥20% bone loss). We therefore expect this source of error to result in false negative radiographic assessments only about 5 percent of the time (estimate adjusted to account for lower disease rates in the HPFS population and chance of false positives from available bitewings).

Results

Two hundred and sixty participants responded to the supplemental questionnaire, for a response rate of 52 percent. Of the 159 participants who provided names of participating dentists (to whom requests for radiographs were made), sets of radiographs were obtained for 77 participants with a positive self-reported status and 63 participants with a negative self-reported status.

The sample of 140 participants who provided radiographs (responders) were somewhat older, consisted of more former smokers, and had slightly more teeth compared to the total population of HPFS dentists (Table 1). The responders were slightly more conscientious (exercised more and had higher health care utilization). Responders were similar to partial responders (questionnaire but no radiographs) with respect to frequency of brushing or flossing, but were less likely to be under the care of another dentist than partial responders.

TABLE 1.

Subject Characteristics and Prevalence of Self-reported Periodontal Disease History by Age, Number of Teeth, and Smoking History

	Validation Sample (%)*	HPFS Dentists		HPFS Nondentists
		N (%)	% Perio POS.	N	% Perio POS.
Age group
40–44	1	5,629 (20)	8	4,523	7
45–50	17	4,240 (15)	12	2,974	10
50–54	16	4,489 (16)	17	2,888	12
55–59	14	4,251 (15)	22	3,066	14
60–64	16	4,176 (15)	27	3,096	17
65–69	21	3,353 (12)	29	2,376	16
≥70	15	2,286 (8)	34	1,467	17
Number of teeth
0	1	264 (1)	49	438	27
1–10	0	329 (1)	64	561	36
11–16	1	485 (2)	59	681	30
17–24	6	2,780 (10)	39	2,914	20
≥25	91	24,477 (86)	15	15,633	9
Smoking
Never	40	12,705 (47)	12	9,033	8
Former	53	11,872 (44)	25	8,655	16
Current	6	2,696 (10)	31	1,997	21

^*Standardized to the distribution of self-reported periodontal status of the HPFS dentists.

The relationship of the self-reported measure with known determinants of periodontal disease among HPFS dentists (Table 1) were consistent with expectations (11,22). Smokers were more likely to have periodontal disease history than nonsmokers, and prevalence of positive periodontal history increased with age. Participants with a history of periodontal disease had fewer teeth. Nondentists (Table 1) had lower reported periodontal history rates than dentists (12.5% vs 19.3%), and showed smaller associations than dentists of the self-reported measure with age, suggesting lower validity, but similar associations with smoking and number of teeth.

Tables 2 and 3 display the between and within examiner reliabilities for radiographic assessments, respectively. On a site level, the percent exact agreement between and within examiners for the dichotomized scores ranged from 78 percent to 87 percent and kappas ranged from 0.49 to 0.62. All Spearman correlations for participant mean bone loss were approximately 0.8. The number of sites ≥2 were highly correlated with mean bone loss (r approximately 0.9) and showed slightly lower reliabilities than mean bone loss. Pearson correlations were slightly higher.

TABLE 2.

Interexaminer Comparisons of Percent Agreement and Kappa Statistic for Radiographic Assessments of Each Site (Score of 0,1 vs 2,3)and Correlations on Subject Mean Bone Loss

	% Agreement (n=3,230)*	Kappa (n=3,230)*	Correlation Coefficients (n=140)
			Pearson	Spearman
Average	80	0.52	0.81	0.78
Examiners A and B	83	0.59	0.82	0.81
Examiners B and C	78	0.51	0.79	0.76
Examiners C and A	78	0.49	0.82	0.78

^*Average sample size.

TABLE 3.

Intraexaminer Comparisons of Percent Agreement and Kappa Statistic for Radiographic Assessments of Each Site and Correlations on Subject Mean Bone Loss

	% Agreement (n=481)*	Kappa (n=481)*	Correlation Coefficients (n=19)
			Pearson	Spearman
Average	84	0.55	0.82	0.77
Examiner A	86	0.58	0.84	0.77
Examiner B	87	0.62	0.80	0.79
Examiner C	80	0.49	0.83	0.75

^*Average sample size.

Validity estimates of the self-reported periodontal history measure in the 1986 HPFS questionnaire using the radiographic assessment as the standard are presented (Table 4). The positive predictive value is 0.76 and the negative predictive value is 0.74. Thus, of the participants who reported positive, 76 percent were found to have disease (defined as all three examiners calling the same two or more sites of a participant positive) and of participants who reported negative, 74 percent were found to be truly negative. The positive predictive values were higher and negative values were lower for the 1988 questionnaire measure compared to the 1986 measure. Only 9 percent of the dentists reported positive on the 1988 questionnaire.

TABLE 4.

Validity of Self-reported Periodontal History Measures

	Reported Positive		Reported Negative
	N (%)	PV+	N (%)	PV−
Overall	63 (45)	0.76	77 (55)	0.74
Severity of bone loss
None	2 (5)	0.50	39 (95)	0.87
Mild	22 (44)	0.59	28 (56)	0.64
Moderate	29 (81)	0.90	7 (19)	0.43
Severe	9 (100)	0.89	0 (0)	–

PV+ = positive predictive value and PV− = negative predictive value.

Participants who reported no bone loss on the supplemental questionnaire generally reported negative periodontal history at baseline, and participants who reported moderate to severe disease generally reported positive disease. Among participants who reported mild bone loss, 56 percent had reported a negative history of bone loss, whereas 44 percent had reported a positive history. Thirty-eight percent of participants with mild bone loss were misclassified compared to only 16 percent of participants with none or moderate to severe bone loss; the positive predictive values increased and the negative predictive values decreased with severity of disease.

Box plots of mean bone loss (average score for a participant across all sites and examiners) by levels of the self-reported measure are shown in Figure 2 and by levels of bone loss from the supplemental questionnaire in Figure 3. Using a threshold based on the distribution, 75 percent of those who reported negative had mean bone loss scores of 1.2 or less and over 75 percent of those who reported positive had mean bone loss scores above 1.2. Men who reported moderate to severe bone loss tended to have more bone loss than men who reported no bone loss. Participants who reported mild bone loss showed a wide range of mean bone loss (Figure 3).

FIGURE 2. Box Plots of Subject Mean Bone Loss Score (Averaged Across All Sites and All Examiners) by Self-reported Periodontal Disease History.

[The dotted lines are the 25th, 50th, and 75th percentiles. The mean is denoted by a plus sign and the zeros and asterisk are minor and major outliers.]

FIGURE 3. Box Plots of Subject Mean Bone Loss Score (Averaged Across All Sites and All Examiners) by Levels of Self-reported Bone Loss from Supplemental Questionnaire.

[The dotted lines denote the 25th, 50th, and 75th percentiles. The mean is denoted by a plus sign and the zeros and asterisk are minor and major outliers.]

Small variations in validity were seen by age and smoking—positive predictive values were higher among smokers and older participants, whereas the reverse was true for negative predictive values. The year in which radiographs were acquired did not have much affect on validity. The only subgroup with unacceptable validity was the 28 participants with radiographs other than bitewings, suggesting error in the standard.

Periodontal bone loss status of males in the HPFS and VA DLS populations by age is displayed in Figure 4. The association of the periodontal disease measure with age is similar in both groups. The age-standardized periodontal prevalence was 36 percent for the HPFS dentists and 50 percent for the VA DLS population.

FIGURE 4. Percent of Male HPFS and VA Dental Longitudinal Study Participants with Periodontal Disease by Age Group.

Prevalence for DLS subjects computed as percent of VA DLS subject with bone loss of 20 percent or more on two or more teeth among all premolars and first and second molars only. The HPFS measure was adjusted for misclassification.

According to probability theory: Adjusted prevalence = P(True positive) = [P(True positive I reported positive). P(reported positive)] + P(True positive I reported negative). P(reported negative)]

Since P(True positive I reported negative) = 1–P(True negative I reported negative)

P(True positive) = [(Predictive value positive). P(reported positive)] + (1–predictive value negative). P(reported negative).

Thus, adjusted prevalence=(0.76).(0.19)+(0.26).(0.81)=0.35.

Discussion

Dentist participants in the HPFS correctly reported their disease status according to the criteria used to define periodontal disease about 75 percent of the time. The validity was lower among participants with mild or borderline disease, whereas the extremes of disease severity were much better classified, suggesting a lower impact on attenuation of the relative risks than if the misclassification were completely random. Age and smoking were positively related to disease severity; hence, the predictive values by age and smoking reflect the finding that severity of bone loss was positively related to positive predictive value and inversely associated with negative predictive value. Timing of radiographs had little effect on the validity. Overall the validity was reasonable in all subgroups and the measure therefore is suitable for use in epidemiologic studies in this population. Since the questionnaire did not provide a threshold to define periodontal disease, the validity of this measure suggests that not only are the dentists in this study aware of their periodontal status, but that they use a similar threshold to define periodontal disease.

Bone loss in the buccal or lingual areas or in extracted teeth would not be detectable by radiographs and thus could lead to a false negative radiographic reading if the participant did not have detectable bone loss in areas covered by the radiographs. This problem is expected to be small, since most of the disease occurs on proximal sites (23). On the other hand, in rare cases a false positive assessment may occur in participants without periodontal disease who might have experienced alveolar bone loss due to other local pathology or systemic diseases.

Examiner reliabilities for radiographic assessments are high, although there is still an element of error in the standard. Part of the misclassification might be due to radiograph quality as suggested by the lower validity among participants with radiographs other than bitewings. The self-reported status was based on periodontal status over the lifetime and was more likely to be based on clinical assessment of pocket depth and attachment level. Hence, errors in radiographic assessment are unlikely to be associated with errors in the self-reported measures, thus resulting in a conservative estimate of their validity (24). Misclassification in the standard due to limitations of radiographs and to examiner or recorder error is thus likely to result in an underestimate of the observed validity estimates; hence, the actual estimates are expected to be higher. The strong associations of the self-reported measure with known determinants of periodontal disease lend additional support to these findings. The comparability of the association of the self-reported measure and age, with the association of the extensive radiographic measure from the VA DLS study with age adds credibility to the self-reported measure.

Since the response rate was low, some bias in the validity estimates in the opposite direction theoretically could exist if the 140 participants whose radiographs were obtained knew and/or reported their periodontal status more accurately than the remaining 360 who were sampled. The participants in the validation study were slightly older, were more likely to be former smokers, had slightly more teeth, and reported less periodontal disease. However, these factors showed little relation to validity. Participants might be slightly more conscientious and thus have a slightly better validity. Interestingly, participants who claimed that their self-reported status was based on self-diagnoses actually had a better validity, and being under the care of another dentist did not affect the validity. Also, given that the disease history includes the whole life span and that these are dentists who have several other means to assess their periodontal status, it seems less likely that not having radiographs leads to poorer self-reported periodontal status.

The bias due to nonresponse is likely to be small compared to the bias due to errors in the standard; hence the true validity estimates, after accounting for all sources of bias, may be slightly higher, but are unlikely to be lower than the observed estimates.

The difference in periodontal disease prevalence between the HPFS dentists and the DLS population seems smaller than one would expect, if periodontal disease was largely governed by home care and professional care. The dentists seem to have a similar amount of disease. In what might be expected to be a population with the best oral health in the nation, periodontal bone loss seems to be a phenomenon similar to hair loss: with increasing age, everybody has some and some have more than others.

As the validity of self-reported periodontal disease is expected to be lower in the general population, the estimates from this study should be considered an upper bound for this measure. A greater proportion of dentists reported periodontal disease compared to nondentists. This finding suggests that other health professionals have more false negatives due to lesser awareness of the subtle signs of periodontal disease. The associations of the self-reported measure with known determinants of periodontal disease among nondentists, although lower than among dentists, was fairly good. This finding suggests that such measures might have potential for use in nondentist groups. We only had a single binary measure of periodontal disease. More detailed questions and clearer criteria for disease classification are likely to lead to higher validity, especially among those with mild disease. Additional studies are needed to evaluate the potential use of such self-reported measures in other populations, which would be extremely useful and result in significant savings in time and costs in large epidemiologic studies. For survey research the individual misclassification would be much less of a concern and one could obtain reasonably good estimates, provided the systematic bias of the measure was minimal. Valid self-reported measures would be useful for conducting survey research and for large-scale epidemiologic studies with limited resources.