The effect of a deep-learning tool on dentists’ performances in detecting apical radiolucencies on periapical radiographs

Abstract

Objectives:

To determine the efficacy of a deep-learning (DL) tool in assisting dentists in detecting apical radiolucencies on periapical radiographs.

Methods:

Sixty-eight intraoral periapical radiographs with CBCT-proven presence or absence of apical radiolucencies were selected to serve as the testing subset. Eight readers examined the subset, denoted the positions of apical radiolucencies, and used a 5-point confidence scale to score each radiolucency. The same subset was assessed by readers under two conditions: with and without Denti.AI DL tool predictions. For the two sessions, the performance of the readers was compared. The comparison was performed with the alternate free response receiver operating characteristic (AFROC) methodology.

Results:

Localization of lesion accuracy (AFROC-AUC), specificity and sensitivity (by lesion) detection demonstrated improvements in the DL aided session in comparison with the unaided reading session. Subgroup performance analysis revealed an increase in sensitivity for small radiolucencies and in radiolucencies located apical to endodontically treated teeth..

Conclusion:

The study revealed that the DL technology (Denti.AI) enhances dental professionals' abilities to detect apical radiolucencies on intraoral radiographs.

Advances in knowledge:

DL tools have the potential to improve diagnostic efficacy of dentists in identifying apical radiolucencies on periapical radiographs.

Introduction

The term “artificial intelligence (AI)” was coined in 1956, at a workshop that took place at Dartmouth College. ¹ Scientists hypothesized that a machine can be trained to learn through experimental “trial and error” much like humans do. ² Since then, AI has been progressing very rapidly in the medical field and is showing the most promise in providing non-specialists with easily accessible, expert-level predictions. ³

Deep learning (DL) is a class of artificial intelligence algorithms that allows a computer program to learn from input data for further interpretation of previously unseen samples. In dentistry, multiple computer-aided detection and DL tools have emerged recently for the assessment of dental caries. ^4–7 Additionally, neural networks and DL algorithms have been utilized in applications that include predicting dental pain, ⁸ teeth numbering and classification, ⁹ in deciding if extractions are necessary prior to orthodontic treatment, ¹⁰ and for detecting periodontal bone loss on panoramic radiographs. ^11,12

In DL, a convolutional neural network (CNN) is a class of artificial neural network most commonly applied to analyze visual imagery. Deep CNNs have been used in the field of endodontics to detect apical lesions on panoramic radiographs and CBCTs. ^13–17

In addition, they have been utilized in analyzing the root morphology of mandibular first molars and to evaluate vertical root fractures on panoramic radiographs. ^18,19

Denti.AI (Denti.AI Technology Inc., Toronto, CA, http://denti.AI) is a software that uses DL technology to assist in the detection of carious lesions, apical radiolucencies, tooth numbering, and dental charting. ⁹ The apical radiolucency detection module is based on DL techniques, specifically deep CNNs. The key aspect of these CNNs is that these detection features are not designed by humans, but automatically extracted and learned from the raw data, such as pixels of images. ^20,21

While computer-aided detection of apical radiolucencies might benefit both experienced radiologists and general practitioners, we anticipate that this would be more relevant to the latter population due to their greater numbers and need for efficient screening tools.

In this study, we aim to investigate the effectiveness of using the Denti.AI system to assist dentists in the task of detecting apical radiolucencies on periapical radiographs.

Methods and materials

Case selection and ground truth

Ethical approval was granted by the University Biomedical Institutional Review Board. The dental school’s Oral and Maxillofacial Radiology CBCT referral database was searched for all volumes acquired for endodontic purposes between August 2014 and March 2019. The finalized radiology reports were analyzed for findings related to apical radiolucencies including: “apical rarefying osteitis”, “apical radiolucent lesions”, or “apical widening of the PDL space”. All radiology reports of those volumes were written by one of the three board-certified oral and maxillofacial radiologists with at least 10 years of experience. If a report was found to contain evidence of an apical radiolucency, the accompanying CBCT volume was downloaded and re-examined by the primary investigator. Measurements of the lesions were then recorded in all dimensions; mesiodistal, buccolingual, and apico-coronal. Apical radiolucencies measuring less than 2 mm at their widest dimension were excluded to reduce the potential imperfect reference standard bias subsequent to beam hardening artifacts resulting from endodontic treatment fill-material. ²² In addition, the exclusion criteria encompassed radiographs of patients under 18 years old.

Once a case was considered, the patients’ records were searched for a corresponding, same-site intraoral periapical radiograph to be added to the study. The inclusion criteria for periapical radiographs comprised diagnostically acceptable radiographs acquired within a six-month window of the accompanying CBCT. For control cases, periapical radiographs of teeth with CBCT verified sound periodontium were uploaded in the negative subgroup. The intraoral images were all verified to have no evidence of apical radiolucencies on CBCT volumes and associated reports. At the time of this writing, the authors are not aware of any previous investigation utilizing CBCT as a reference standard. Previous studies have used consensus panels, which we considered not as robust as a CBCT reference standard.

After applying the inclusion and exclusion criteria, a total of (n = 184) positive intraoral radiographs were collected and divided into a testing (n = 130) and a model-tuning (n = 54) subsets. One hundred and thirty-two periapical radiographs with sound apical periodontium were collected to serve as controls. The final testing subset was comprised of 68 images that were randomly selected from the testing positive (130) and control sets (132) by assigning a number to each case and using a random number generator. Table 1 displays an overview of the distribution of cases between positive and control, number of lesions, along with the extent of radiolucencies and endodontic treatment status.

Table 1.

General Descriptive Statistics. Provides an overview of the positive and control samples and the distribution of radiolucent lesions by extent and endodontic treatment status

Cases	Number of periapical radiographs	Number of teeth	Number of lesions
Positive cases	38	152	56
Control cases	30	116	0
Total cases	68	268	56
56 lesions extent and treatment status
Extent		Number of lesions
Small (2–5 mm)		31
Large (>5 mm)		25
Treatment status		Number of lesions
Endodontically treated		31
Not endodontically treated		25

The AI model was pre-trained ahead of this study on images obtained from sites other than the dental schools’ clinics (>1000 images). The model-tuning subset (n = 54) of images obtained in the dental school’s clinics was added to further adapt the model to the devices used in the clinics.

The model-tuning and testing sets were de-identified and uploaded to Denti.AI. The collected radiographs were annotated using the Denti.AI labeling tool.

Apical radiolucencies were then divided by:

1. Extent: small radiolucency (2–5 mm) or large radiolucency (≥ 5 mm)

2. Treatment status: endodontically-treated or untreated teeth.

Corresponding tags were added for each annotated lesion showing the extent of the lesion and treatment status of the tooth.

Devices and imaging instruments

The CBCT volumes were all previously acquired using either Orthophos XG 3D, Orthophos SL 3D (Dentsply Sirona, Charlotte, NC), CS 9000 or CS 9300 (Carestream Dental, Atlanta, GA). The 68 periapical radiographs were obtained using photostimulable phosphor (PSP) plates scanned using ScanX (Air Techniques, Hicksville, NY), Soredex Digora Optime (Kavo Dental, Charlotte, NC), or using Sirona Schick33 Direct Digital Sensor (Dentsply Sirona, Charlotte, NC), or XDR Anatomic Sensor (Cyber Medical Imaging, Los Angeles, CA). The inclusion of multiple units and sensor types allowed for additional generalizability of the results.

Reader study execution

Eight readers performed a cross-over reading scenario. The eight readers were comprised of six operative dentistry residents, a general dentist and an endodontist. Each reader analyzed the same testing subset collection of 68 images under two conditions; without and with the aid of AI predictions. The reading sessions were separated by a washout period of more than one month.

Readers underwent a calibration session conducted by the primary investigator. The training included sample images and practice annotations. For the first session, without the assistance of AI predictions, the readers were instructed to use the labeling tool to draw bounding boxes around sites with suspected apical radiolucencies, i.e., annotate apical radiolucencies (AR). Boxes had to cover the finding with the margin not exceeding 2 mm beyond any side of the AR. In addition, they were asked to add confidence score tags that reflected their confidence regarding the presence of their decision; a (1-5) confidence rating scale was used for this step. The tags were as follows: C1: not confident, C2: slightly confident, C3: somewhat confident, C4: moderately confident, and C5: very confident.

For the second reading session, the task was to review the same periapical radiographs and either confirm or modify the bounding box, add a new finding or delete the prediction generated by the Denti.Ai system based on the output of the DL tool.

Statistical analysis

RJafroc R-library (version 2.0.1 https://dpc10ster.github.io/RJafroc/) was used to evaluate the performance. ²³ Dorfman-Berbaum-Metz with Hillis' improvements (DBMH) method of analysis was applied which is a method for multireader multicase (MRMC) analysis that uses the jackknife technique and conventional analysis of variance (ANOVA). ²⁴ MRMC analysis allows for the assessment of the significance of shift in dentists’ performance after using AI predictions and provides a quantitative measure of the performance of a diagnostic test across varying reader skills. The random-reader random-case (RRRC) option of the MRMC analysis was evaluated. Instead of the significance of shift in performance applying only to the participating group of dentists or radiographs included in the study, RRRC allows for the results to be more generalizable to encompass other dentists reading new cases.

The following common definitions were applied:

1. Lesion: an apical radiolucency that is shown on the image in the form of a bounding box; each lesion annotated by the reader is supported by a confidence score.

2. Case: an image that is interpreted by the reader, i.e. a periapical radiograph. The inferred receiver operating characteristic rating paradigm was applied to define a confidence rating for the case annotated by the reader: the highest rating was used in the case of multiple lesions shown.

Alternative Free-Response Receiver Operating Characteristic (AFROC) AUC metric was evaluated as the primary endpoint for comparing the performance of the readers for the two reading scenarios. AFROC AUC measures the area under the AFROC curve. This curve shows a trade-off between lesion localization fraction and the false-positive fraction for the range of decision thresholds. The lesion localization fraction shows how many ground truth lesions were correctly detected by the reader, i.e. sensitivity on the by-lesion basis.

False-positive fraction shows how many actual negative cases were mistakenly classified as positive, i.e. 1 - specificity on a by-case basis. AFROC was chosen as the main metric of performance as it both provide aggregated measurement over different thresholds and accounts for localization accuracy.

Secondary endpoint analysis included the following metrics: sensitivity (by case) specificity (by case), and sensitivity (by lesion). The specificity (by lesion) metric was not calculated. That is mainly because this metric depends on true-negatives and there is no meaningful way to calculate true-negative lesions, as there are an undefined number of locations where the lesions might be shown on an image. Subgroup performance analysis was conducted to measure the effect of the DL tool for different characteristics of lesions. The analysis was stratified based on apical radiolucency extent and endodontic treatment status of the tooth.

Results

Primary and secondary endpoints

Table 2 shows the primary and secondary endpoint results. With the AI-aided session, AFROC indicated a clinically important 8.6% improvement over the unaided reading session (p = 0.005). Figure 1 displays the improvement in the aggregated measure of performance (AFROC AUC) between the aided and unaided reads. Figure 2 shows examples of improved localization and specificity in the detection of apical radiolucencies using the software. Secondary endpoints results showed improved specificity by case (p = 0.005) and sensitivity by lesion (p = 0.01) between the two sessions—a 23.1% improvement in specificity by case along with an 8.2% improvement in sensitivity by lesion was demonstrated after the AI-aided session. There was no difference in the sensitivity by case metric between the two sessions (p = 0.712).

Table 2.

Primary and Secondary Endpoints. Provides statistics on Primary and Secondary study endpoints: AFROC AUC, Sensitivity (by case), Specificity (by case), and Sensitivity (by lesion)

Primary and Secondary Endpoints (68 images, 56 lesions, eight readers)
	AFROC	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.892	0.833	0.951	0.071	0.022	0.119	0.005	8.6%
Read 1	0.822	0.749	0.894
	Sensitivity by Case	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.931	0.884	0.978	−0.007	−0.043	0.030	0.712	−0.7%
Read 1	0.938	0.904	0.971
	Specificity by Case	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.733	0.644	0.822	0.138	0.048	0.277	0.005	23.1%
Read 1	0.596	0.506	0.685
	Sensitivity by Lesion	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.888	0.831	0.946	0.067	0.017	0.117	0.010	8.2%
Read 1	0.821	0.759	0.884

Figure 1.

Averaged AFROC-AUC Curves The AFROC curves are averaged for the eight readers and two reading modes: unaided (blue) and aided (red). The gap between the two curves reflects the improvement in the aggregated measure of performance (AFROC-AUC) between the AI-aided readings and the unaided readings. LLF: lesion localization fraction (sensitivity), FPF: false-positive fraction (1 - specificity).

Figure 2.

The effect of Denti.AI software: examples. Top image: an additional radiolucency was detected by readers during the AI-aided read. Bottom image: a false-positive lesion was removed during the second read, corresponding with AI prediction results.

Subgroup analysis stratified by lesion characteristics

Table 3 shows the subgroup analysis by extent and endodontic treatment status. There was an improvement in the sensitivity by lesion detection between the two sessions for small apical radiolucencies (p < 0.001), and for radiolucencies associated with endodontically treated teeth (p < 0.001). With the aid of AI, the sensitivity for small lesions increased by 14.5% and by 15% in endodontically treated teeth.

Table 3.

Subgroup Statistics. Provides statistics on cohorts of lesions differentiated by the extent (small and large), and tooth treatment status (endo-treated and non-endo-treated)

Subgroup Statistics (eight readers)
Small Extent (31 lesions)
	Sensitivity by Lesion	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.859	0.808	0.910	0.109	0.058	0.160	<0.001	14.5%
Read 1	0.750	0.684	0.816
Large Extent (25 lesions)
	Sensitivity by Lesion	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.925	0.876	0.974	0.015	−0.021	0.051	0.409	1.6%
Read 1	0.910	0.866	0.954
Endodontically treated (31 lesions)
	Sensitivity by Lesion	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.956	0.933	0.978	0.125	0.069	0.181	<0.001	15%
Read 1	0.831	0.762	0.899
Non-endodontically treated (25 lesions)
	Sensitivity by Lesion	CI Lower	CI Upper	Read 2 – Read 1	CI Lower	CI Upper	P-Value	%
Read 2 (Aided by AI)	0.805	0.726	0.884	−0.005	−0.053	0.043	0.827	−0.6%
Read 1	0.810	0.754	0.866

Discussion

In accordance with the purpose of this research, the findings showed that using Denti.AI resulted in an improvement in the localization of apical radiolucencies by dentists. This is clinically relevant for dentists as they are responsible for selecting the most appropriate radiographic modality, interpreting the results, and making appropriate treatment decisions based on these interpretations. If radiographs are viewed inattentively, this could result in patients being over- or undertreated. ²⁵ Subgroup analysis results displayed an improvement in the detection of small lesions and lesions apical to endodontically treated teeth. Further studies with larger sample sizes are necessary to validate the above findings.

In endodontics, CNNs have been applied to tasks that include the detection of apical lesions with panoramic radiography and CBCT. In one study, a moderately deep CNN trained on a limited set of panoramic images showed satisfactory ability to detect apical lesions. Based on the consensus of six examiners, the AUC of the CNN was 85%. Sensitivity and specificity were 65% and 87%, respectively. A subgroup analysis for tooth type was also performed showing a higher sensitivity in molars than in other tooth types, whereas specificity was lower. ¹³ Another study that was based on clinically validated ground truth investigated the detection of periapical radiolucencies on panoramic radiographs. The periapical radiolucencies included infections, granulomas, cysts, and tumors. Results demonstrated that the DL algorithm achieved a better performance than 14 of 24 participating oral and maxillofacial surgeons within the cohort. ¹⁴ Excellent results were found in a study that used a DL algorithm for the automated segmentation of CBCT images and the detection of periapical lesions with lesion detection accuracy of 93%. ¹⁶ In another attempt to evaluate a CNN method for detecting apical pathosis with CBCT, 153 periapical lesions obtained from 109 patients were included, and the AI system was able to detect 92.8% (142/153) of the periapical lesions. ¹⁷

The previous studies were designed to assess the diagnostic efficacy of DL systems in detecting periapical lesions as standalone systems compared with clinicians. Our investigation is novel in that it seeks to determine the effect of a specific DL system on the performances of dental professionals. We posit that assessing the ability of the DL software to assist the clinician is of equal value to the readership.

Various observer studies have been conducted to evaluate the accuracy of periapical radiographs, panoramic radiographs, and CBCT images in the diagnosis of apical radiolucent lesions. While studies showed varying results, we used CBCT volumes as a reference standard since they have been shown to have much higher diagnostic accuracy than 2D imaging modalities. In one study, small and large artificial periapical lesions were prepared in the periapical region of the distal root of six molar teeth in human mandibles. The ROC Az (AUC) values were 0.79 and 1.00 for intraoral radiographs and CBCT, respectively. ²⁶ It was also reported that (20%–39%) of AP radiolucencies were diagnosed with CBCT and missed with intraoral radiography. ²⁷ In another study of 156 roots, CBCT detected more lesions (34%) than periapical radiographs (p < 0.001). ²⁸ Furthermore, one study assessed 120 untreated apical periodontitis sites and 120 healthy sites on panoramic radiographs. Findings included a low sensitivity of 34.2%, a diagnostic accuracy of 65% and a high specificity 95.8%. ²⁹ A major advantage of using CBCT as the reference is the ability to detect apical radiolucencies in anatomically challenging areas such as the posterior maxilla, where anatomic overlap takes place in two-dimensional images. ³⁰ CBCT can display the details of the lesions and adjacent structures and provide correct clinical diagnosis as it shows destruction of cortical bone that could not be detected with periapical radiography. ³¹ Furthermore, the prevalence of apical pathology undetected on periapical radiographs is concerningly high, with 30–50% of mineral loss being needed to visualize the lesions. ³² Thus, the limitations of periapical radiographs as a diagnostic tool should not be disregarded, mainly to reduce false-negative results. ^33,34

For the average performance for all eight readers, the results of this study with the AI-aided session showed an 8.6% improvement in AFROC AUC compared with the unaided reading session (p = 0.005). Additionally, specificity by case and sensitivity by lesion showed 23.1 and 8.2% improvements, respectively. These results reflect that the detection of apical radiolucencies by clinicians can be improved using the Denti.AI system. Sensitivity by case did not show a clinically important difference between sessions (0.7%, p = 0.712). In the context of this study, a case is the radiograph itself. The study’s limitations include a small testing sample size and a limited number of observers.

Conclusions

This study demonstrates Denti.AI system’s potential in aiding dentists in detecting and localizing apical lesions on intraoral images. The 8.6% improvement in AFROC AUC is clinically relevant. Further studies with a more diverse and larger group of readers and cases would add more robust evidence on the effects of DL on particular cohorts of lesions and readers.