Inter-examiner Agreement in the Radiologic Identification of Apical Periodontitis/Rarefying Osteitis in the National Dental PBRN PREDICT Endodontic Study

Abstract

Introduction:

Periapical images are routinely made in endodontics to support diagnosis and treatment decisions, but conventional imaging may not readily demonstrate inflammatory changes. This study aims to 1) quantify disagreement in the radiologic interpretation of apical periodontitis/rarefying osteitis between 2 expert examiners; and 2) determine if differences exist based on anatomical location.

Methods:

We used 1717 pre-treatment periapical images made prior to orthograde endodontic treatment as part of the Predicting Outcomes of Root Canal Treatment (PREDICT) study conducted within the National Dental Practice-Based Research Network. Periapical changes were assessed independently by two board-certified specialists, an oral and maxillofacial radiologist and an endodontist, blinded to other clinical information. If the examiners disagreed about whether a diagnosis of apical periodontitis/rarefying osteitis was justified, an adjudication was made by a third examiner.

Results:

The overall prevalence of this radiologic diagnosis in the periapical images was 55% and inter-examiner agreement measured with Cohen’s kappa was calculated to be 0.56 (95% Confidence Interval 0.52, 0.60). Diagnostic disagreements between the two examiners occurred for 377 teeth (22%) with disagreements more frequent for jaw location (p=0.038) and tooth type (p=0.021). Differences between root number (p=0.058), and jaw location and tooth groups (p=0.069) were found not to be statistically significant.

Conclusions:

The variability of diagnostic disagreements across anatomical location and tooth type may reflect the inability of periapical images to reveal bone changes masked by the complexity and density of overlying anatomical structures; a limitation that could potentially be overcome with the use of three-dimensional imaging.

Introduction

Radiologic image interpretation relies on the ability of clinicians to visualize anatomical changes and then relate them to specific biological processes (1). This visualization can only occur when the clinician is able to perceive a difference in x-ray beam attenuation between normal tissue and an area of disease. In endodontics, periapical radiography is commonly used to assess periradicular and periapical tissues for signs of disease; the periodontal ligament space, the lamina dura and the cancellous bone architecture. As well, periapical radiography is also used to monitor osseous healing following endodontic therapy.

For two-dimensional projection techniques like periapical radiography, the ability to detect changes to normal anatomical structures can be attributed to the differential passage of the incident x-ray beam through tissue, and the ability of an image receptor to capture these changes in the form of exiting photons. Greater than 30% of generalized cortical and cancellous bone mineral content must be lost before a radiographic change can be seen (2). However, for localized changes like apical periodontitis/rarefying osteitis, this threshold may be much lower; 12.5% cortical bone loss with a 6.6% mineral bone loss. If these thresholds are not met, apical periodontitis/rarefying osteitis may not be seen, and this may affect therapeutic decision making (2–6).

Once captured and displayed, the radiologic image must be evaluated for the presence of disease. The normal periodontal ligament space is reported to vary between 0.15 mm and 0.21 mm (7), and widening of this space may occur uniformly around the root or more locally. The latter type of widening may demonstrate a concentric or hydraulic pattern of widening with a discrete epicenter, without or with alterations to the appearance of the lamina dura (8). In oral and maxillofacial radiology, the interpretation of this change is referred to as rarefying osteitis (8), and in endodontics, apical periodontitis.

Given the nature of radiologic image interpretation, there may be disagreements between clinicians. One study reported interpretation disagreement between 3 examiners (1 oral and maxillofacial radiologist and 2 endodontists) for 27% of cases of apical periodontitis, while disagreement for individual examiner pairs ranged from 14% to 18%. For these examiner pair comparisons, Cohen’s kappa (κ) was reported to be 0.55 and 0.58 (“moderate” agreement) (9). A follow-up examination of cases from this study deemed to be “borderline” due to uncertainty in decision-making stressed the importance of careful analysis of periapical and periradicular tissues (10).

This study is a post hoc analysis of radiographic images collected as part of the Predicting Outcomes of Root Canal Treatment (PREDICT) study conducted in the offices of general dentists and endodontists enrolled in the National Dental Practice-Based Research Network (the “Network”) (11) to investigate diagnostic disagreements on periapical images made by 2 independent expert examiners. The aims of this study were to 1) quantify disagreement in the radiologic interpretation of apical periodontitis/rarefying osteitis between 2 independent expert examiners; and 2) determine if differences exist based on anatomical location.

Materials and Methods

Human research ethics approval was granted by the University of Alabama at Birmingham (Protocol X161121003), the University of Minnesota (Protocol RNI00002290), and the University of Toronto (Protocol 34105).

Patient clinical and radiographic data were collected within the Network. During the study, 156 practitioners (115 general dentists and 50 endodontists) used a consecutive enrollment strategy to enroll patients over a 6 month period (April to September 2017). The study asked each general dentist to enroll a target of 7 patients and each endodontist to enroll a target of 15 patients over 10 to 14 weeks. Enrollment targets differed between general dentists and endodontists to reflect differences in the number of root canal treatments performed in a typical week.

Study data were collected from practitioners before root canal treatment, at the completion of root canal treatment, and at a 12-month follow-up visit. The full study protocol and all data forms used in this study are publicly available (https://www.nationaldentalpbrn.org/study-results/#1589313316092-edfe4c32-6b00). Practitioners uploaded 1 to 2 periapical images at each time point for images taken as part of standard care. For this analysis, images were taken from the initial visit before root canal treatment.

Quality assurance guidelines for the periapical images were determined by the study team at the outset of the study prior to data acquisition, and acceptable periapical images were defined as those in which the tooth root apex or apices could be visualized with 3 mm of cancellous bone beyond the apex or apices. Network dentists were required to submit at least 1 pre-treatment image that met these standards. The periapical images were reviewed by a board-certified oral and maxillofacial radiologist (Examiner 1). If the image did not meet quality assurance guidelines, the practitioner was notified to submit a replacement image. This occurred in fewer than 3% of cases. Also at this time, a meeting that included Examiner 1 as well as a second examiner, a board-certified endodontist (Examiner 2) and a board-certified specialist in Orofacial Pain (Examiner 3) was held to determine the criteria for periapical image review, the purpose of which was to provide confirmation of the presence or absence of a periapical change, defined as a localized change in the apical periodontal ligament space width beyond the normal anatomic range of 0.15 to 0.21 mm (7), exhibiting a concentric or hydraulic pattern of widening with an epicenter. Observations were made by Examiner 1 and Examiner 2 during the patient recruitment phase of the study during the first year. The diagnosis of Examiner 1 was not known to Examiner 2, and vice-versa. If the subject database flagged a disagreement between the two primary examiners, the assessment was adjudicated Examiner 3.

A subset of 50 randomly chosen cases was re-reviewed by Examiner 1 and Examiner 2, 18 months after the initial review to determine intra-examiner reliability. Overall percentage agreement and Cohen’s kappa (κ) (19) statistics (standard error and p-value) were calculated using STATA v15 (StataCorp, College Station, TX) to determine inter- and intra-observer examiner agreements with a 95% confidence interval. Chi-square statistics were calculated using Microsoft Excel (Microsoft Corporation, Redmond, WA). A significant difference was defined as p < 0.05.

Results

A total of 1717 periapical images were independently reviewed by the two primary examiners. The most common teeth in the study were maxillary molars (n=593) and mandibular molars (n=443), followed by maxillary premolars (n=306), mandibular premolars (n=165), maxillary incisors and canines (n=161), and mandibular incisors and canines (n=49). The mandibular left first molar (n=186) was the most common tooth enrolled, followed by the mandibular right first molar tooth (n=170), and then the maxillary right first molar tooth (n=151). The least common teeth enrolled were third molars (n=6 mandibular and no maxillary third molars). Examples of submitted images are shown in Figure 1.

Figure 1:

Periapical images of selected participant pre-treatment images. Teeth 29 (A) and 15 (B) were interpreted as having normal periapical/periradicular structures. Teeth 29 (C) and 18 (D) were interpreted as having changes consistent with apical periodontitis/rarefying osteitis.

A total of 948 (55%) were classified by at least one examiner as having apical periodontitis/rarefying osteitis (Table 1). The two primary examiners agreed overall on the periapical diagnosis in 1340 teeth (Figure 2). Percentage agreement of a normal appearance was most frequent for mandibular premolars (64%) and maxillary molars (64%). Percentage agreement of apical periodontitis/rarefying osteitis was most frequent for mandibular incisors and canines (63%), and inter-examiner Cohen’s kappa (κ) was “moderate” (0.56 [95% confidence interval] CI 0.52, 0.60) for the 1717 teeth examined. The intra-examiner agreement was higher for Examiner 1 (κ=0.72, CI 0.52, 0.92) than Examiner 2 (κ=0.30, CI 0.04, 0.56).

Table 1:

Agreement and disagreement dataset for 2 observers in the detection of apical periodontitis/rarefying osteitis. The calculated Cohen’s kappa is 0.56 (95% Confidence Interval 0.52, 0.60).

		Examiner 1 (Oral and Maxillofacial Radiologist)
		Present	Absent
Examiner 2 (Endodontist)	Present	571	106	677
	Absent	271	769	1040
		842	875	1717

Figure 2:

Distribution of enrolled teeth (by tooth number) in the study. The grey portion of each bar represent the number of teeth where the two primary examiners agreed on the interpretation, and black portions represent the number of teeth in disagreement.

With the input of the Examiner 3 in cases of diagnostic disagreement, 57% of teeth were interpreted as having normal periapical appearances and apical periodontitis/rarefying osteitis was interpreted in 43% of cases. The two primary examiners disagreed on the periapical diagnosis in 377 or 22% of the 1717 teeth (Figure 2). Table 2 summarizes the statistical analyses of the agreement and disagreement data grouped by jaw location of the tooth (maxilla, mandible), tooth type (molar, premolar, incisor/canine), tooth root number (single root, multi-rooted), and jaw location and tooth type together (mandibular molars, maxillary molars, maxillary premolars, etc.). Our results demonstrate statistically significant differences between teeth in the maxilla and mandible (p=0.038) and “moderate” agreement with inter-examiner κ of 0.59 (CI 0.54, 0.65) for the maxillae and 0.52 (CI 0.47, 0.58) for the mandible. As well, we demonstrated statistically significant differences for tooth type (p=0.021) with “moderate” inter-examiner κ of 0.52 (CI 0.47, 0.58) for molar teeth and 0.58 (CI 0.51, 0.66) for premolar teeth. “Substantial” inter-examiner κ was found for incisor/canine teeth (κ=0.70, CI 0.60, 0.79).

Table 2:

Agreement and disagreement differences in the detection of apical periodontitis/rarefying osteitis (data grouped by anatomical site)

Anatomical group	Diagnostic Agreement	Diagnostic Disagreement	Cohen’s kappa (95% Confidence Interval)	Chi-Square test
Maxilla	728	182	0.59 (0.54, 0.65)	χ2 = 4.327
Mandible	612	195	0.52 (0.47, 0.58)	p = 0.038
Molar teeth	790	246	0.52 (0.47, 0.58)	χ2 = 7.707
Premolar teeth	372	99	0.58 (0.51, 0.66)	p = 0.021
Incisor/Canine teeth	178	32	0.70 (0.60, 0.79)
Multi-rooted teeth+	894	271	0.53 (0.49, 0.59)	χ2 = 3.601
Single rooted teeth	446	106	0.61 (0.55, 0.68)	p = 0.058
Mandibular molar teeth	442	151	0.51 (0.44, 0.57)
Maxillary molar teeth	348	95	0.55 (0.47, 0.63)
Maxillary premolar teeth	243	63	0.60 (0.51, 0.69)	χ2 = 10.236
Maxillary incisor/canine teeth	137	24	0.47 (0.35, 0.60)	p = 0.069
Mandibular premolar teeth	129	36	0.55 (0.42, 0.68)
Mandibular incisor/canine teeth	41	8	0.66 (0.44, 0.87)

⁺Multi-rooted teeth were all molar teeth and maxillary first premolars.

The results of our Chi-square analyses for single and multi-rooted teeth (p=0.058), and tooth group and location (p=0.069) did not demonstrate statistical significance. For these groupings, we found “substantial” inter-examiner κ (0.61, CI 0.55, 0.68) for single rooted teeth and “moderate” inter-examiner κ (0.53, CI 0.49, 0.59) for multi-rooted teeth. For jaw location and tooth type together, we found “moderate” inter-examiner κ for maxillary molars (0.55, CI 0.47, 0.63), mandibular molars (0.51, CI 0.44, 0.57), maxillary premolars (0.60, CI 0.51, 0.69), mandibular premolars (0.55, CI 0.42, 0.68) and maxillary incisors (0.47, CI 0.35, 0.60). “Substantial” inter-examiner κ was found for mandibular incisors (0.66, CI 0.44, 0.87).

Figure 3 demonstrates the disagreement data by jaw and tooth type. As a group, mandibular molars were found to have the highest percentage disagreement (26%), followed by mandibular premolars (22%), maxillary molars (21%), maxillary premolars (21%), mandibular incisors and canines (16%) and maxillary incisors and canines (15%). More specifically as shown in Figure 4, the highest percentage disagreement was observed for the mandibular left second molar (30%) followed by the maxillary left second molar (26%) and the mandibular left first molar (23%).

Figure 3:

Percent distribution of teeth (by tooth number) where the two primary examiners disagreed on the radiologic interpretation.

Figure 4:

Percent distribution of teeth (by tooth group) where the two primary examiners disagreed on the radiologic interpretation.

Discussion

The diagnosis of apical periodontitis/rarefying osteitis can be challenging, particularly if the patient’s clinical examination findings and diagnostic test results are contradictory or nonspecific (13). Although every participant in the PREDICT study had a clinical diagnosis and underwent orthograde endodontic treatment, this manuscript focuses exclusively on the interpretation of the periapical image data only.

In observational studies such as this, there may be significant variations in the abilities of examiners to identify and reconcile the characteristic features of disease on imaging. And in such instances, diagnostic disagreements may arise (14). Therefore, it is important for examiners to develop a disciplined approach disease diagnosi (15). At the outset of the PREDICT study and prior to data acquisition, there was a meeting of the 3 examiners to determine the criteria for radiographic review.

This study, which included 1717 independent observations by 2 primary examiners, reported a Cohen’s kappa (κ) for inter-examiner agreement of 0.56 ([95% Confidence Interval] CI 0.52, 0.60), which reflects “moderate” agreement. Multiple factors can affect Cohen’s kappa, including observer accuracy (which may affect the frequency of false positive and false negatives), the prevalence of the finding and observer independence, to name a few. Lower Cohen’s κ (i.e., “fair” or “slight” agreement) can reflect an increase in type 2 errors (i.e., false negatives), and in the context of this study, a false negative would mean that if an area of apical periodontitis/rarefying osteitis were to be present, it would not be observed. Low agreement would also reflect low statistical power, calculated as 1-β (where β is the frequency of a type 2 error). In the current study, we reported inter-examiner agreement to be “moderate”, and identified 22% (377) cases of diagnostic disagreement. This level of disagreement between the two primary examiners was deemed to be substantial enough to undertake this post hoc analysis to investigate potential source(s) of diagnostic disagreement.

Intra-examiner Cohen’s κ were determined to be 0.72 and 0.30. Molven et al (9) reported a similar inter-examiner κ (0.53), and intra-examiner reliabilities of 0.54 and 0.57. Uraba et al (16) reported inter-examiner κ of 0.68 and 0.65, and 0.91 and 0.73 for intra-examiner κ. The differences between the intra-examiner agreement of our 2 examiners is worth noting. Typically, observational studies with multiple observers specify that an a priori level of agreement be achieved prior to the start of the study. In this study, this a priori process was not done because this study was not prospectively planned to determine inter-examiner agreement of the radiologic diagnosis. As noted earlier, this study was undertaken only after identifying the magnitude of disagreement between the two primary examiners after the initial data was analyzed. As intra-examiner agreement is typically conducted between 1 and 3 weeks following an initial set of observations, the low intra-examiner κ of the Examiner 2 may be due to the long 18 month period between the two sets of observations. Another potential explanation for the difference in intra-examiner agreement may be related to differences in examiners’ clinical education and training, and the manner in which image interpretation is undertaken in the two specialties. We cannot, however, determine with certainty, how much disagreement is due to these differences.

Of the 377 teeth where there was a diagnostic disagreement, Examiner 1 interpreted the presence of rarefying osteitis in 271 teeth (72%) where Examiner 2 did not. In contrast, Examiner 2 interpreted the presence of apical periodontitis/rarefying osteitis in 106 teeth (28%), and the Examiner 1 did not. For disagreements, Examiner 3 showed no strong agreement preference with either examiner (in 62% of these disagreements, the Examiner 3 agreed with Examiner 1). An obvious limitation of this multi-observer study is that there is no true gold standard for the radiologic diagnosis; histopathologic examination of the periapical tissues or in the case of Uraba et al (16), cone beam computed tomography (CBCT). We found more diagnostic disagreements between the maxilla and mandible (p=0.038), with “moderate” inter-examiner κ in each jaw; 0.59 (CI 0.54, 0.65) for the maxillae and 0.52 (CI 0.47, 0.58) for the mandible. These disagreements may be due to the anatomical and radiologic complexity of the maxilla; the sinus in both health and disease, the zygomatic process of the maxilla and zygomatic bone, and the soft tissue of the nose may contribute greatly to the appearance of the periapical image. In the mandible, differences may be attributed to the thickness of cortical bone, particularly in the posterior mandible. Image interpretation may be further complicated by the bisecting-the-angle technique, particularly if the angulation of the incident x-ray beam is steep enough to be attenuated by the anatomical complexity of an area, including the thickness of the bone and soft tissues.

Our study also found a statistically significant difference between molar, premolar and incisor/canine teeth (p=0.021). For these tooth types, we found “moderate” inter-examiner κ for molar and premolar teeth (0.52 [CI 0.47, 0.58] for molar teeth and 0.58 [CI 0.51, 0.66) for premolar teeth), and “substantial” agreement for incisor/canine teeth (0.70 [CI 0.60, 0,79]). Images of maxillary molar teeth, and in some cases maxillary second premolar teeth may be more prone to superimpositions of hard tissue while the maxillary incisor teeth may be more prone to superimposition of soft tissue structures like nose. Uraba et al (16) reported an overall disagreement between examiners in 41 of 178 teeth compared to their CBCT gold standard. Of note was that the majority of their disagreements were clustered in the maxillary incisor and maxillary canine teeth (16/41 or 39%) and maxillary molar teeth (9/41 or 22%). It is also worth noting that maxillary incisors accounted for the greatest proportion of the Uraba et al (16) study’s teeth (38.7%), and maxillary molars accounted for only 14.6%. In contrast, our study included 161 maxillary incisor and maxillary canine teeth (9% of all teeth), and maxillary molars accounted for 443 teeth (26% of all teeth).

In the mandible, our study included observations from a total of 807 mandibular teeth (593 molar, 165 premolar and 49 incisor and canine teeth) compared to the 39 mandibular teeth in the Uraba et al study (16). For the 377 cases of disagreement, the magnitude of disagreement was highest for mandibular molars as a group (26%) followed by mandibular premolars (22%); specifically, the mandibular left second molar (30%) and the mandibular left first (25%) molar. Although not as anatomically complex as the posterior maxilla, the buccal and lingual cortices of the mandible are both dense and thick. In the mandibular molar area, the external oblique ridge forms a substantial bony buttress merging with the buccal cortical plate, and on the lingual surface, the internal oblique and mylohyoid ridges located near the levels of the tooth root apices attenuate the beam further. The thick anterior border of the overlying masseter muscle may add additional attenuation to the incident beam. Bender and Selzer (2) demonstrated experimentally that as much as 1.25 mm of cortical bone must be removed from the endosteal surface of an 8 mm thick cortex in order for examiners to distinctly visualize a localized change in radiolucency of the bone so it is likely that significant amounts of cortical bone may be obscuring changes in the periapical areas of mandibular molar teeth. Areas of the mandible free from superimpositions of prominent anatomy such as in the mandibular incisor/canine areas may be less susceptible to disagreements, which is what we found.

In our study, we also analyzed the agreement and disagreement data based on tooth root number (i.e., single or multi-rooted), and jaw location and tooth type together. Although our dataset of 377 teeth was larger than Uraba et al (16), we were unable to demonstrate statistically significant differences based on root number (p=0.058). We did, however, find “substantial” inter-examiner agreement of 0.61 (CI 0.55, 0.68) for single rooted teeth. With jaw location and tooth type together, we did not find statistically significant differences between groups (p=0.069). For these data, we did, however, find “substantial” inter-examiner κ of 0.66 (CI 0.44, 0.87) for mandibular incisor/canine teeth; the tooth group least susceptible to anatomic superimpositions on periapical images and “moderate” κ for all other groups. The primary examiners disagreed on the periapical diagnosis in maxillary molars in 21% of cases with the highest disagreement found for the maxillary left second molar (26%). Our findings in the posterior maxillary teeth are in agreement with previous studies (16–19).

CBCT has gained widespread application in endodontics. Unfortunately, the PREDICT study’s web-based radiographic upload system did not have the capacity to upload CBCT image data. Several studies have previously demonstrated the diagnostic superiority of CBCT over periapical radiography to visualize apical periodontitis/rarefying osteitis (17–24), however, the American Association of Endodontists and the American Academy of Oral and Maxillofacial Radiology do not support routine CBCT use for diagnosis or screening (25). Although some evidence-based guidelines have been developed for CBCT use in endodontics, the evidence base that supports their use continues to develop and evolve (26).

Uraba et al (16) have suggested that anatomical complexity could be used as a basis for developing more focussed guidelines for CBCT use, and our results provide further support for this. In the anterior mandible where our examiners recorded fewer disagreements than in the maxilla or more posterior regions of the mandible, periapical imaging may be sufficient for diagnosis. However, in other areas, potentially obscured by a multitude of adjacent anatomical structures, the adjunctive use of CBCT may be useful, potentially depicting changes that might otherwise be difficult to identify in periapical radiographs. While we support the current recommendations of the American Association of Endodontists and American Academy of Oral and Maxillofacial Radiology’s guidelines for CBCT use in endodontics (25), our findings suggest that further refinements could be made for CBCT use in endodontics.

Conclusions

In this large practice-based study, we identified disagreements between examiners in the interpretation of apical periodontitis/rarefying osteitis on periapical images. Statistically significant diagnostic differences were found between teeth in the maxilla and mandible, and between molar, premolar and incisor/canine teeth. Our work suggests that a more anatomically specific approach be considered for CBCT use in endodontics, and future guidelines should take this into consideration.