Comparison between observer-based and AI-based reading of CBCT datasets: An interrater-reliability study

Abstract

Objective

To assess the performance of human observers and convolutional neural networks (CNNs) in detecting periodontal lesions in cone beam computed tomography (CBCT), a total of 38 datasets were examined. Three human readers and a CNN-based solution were employed to evaluate the presence of periodontal pathologies in these datasets.

Materials and Methods

Datasets were acquired with a Veraview X800 L P (JMorita Mfg. Corp., Kyoto, Japan). Three general dentists, previously calibrated by a general principal investigator, read the datasets in 3D MPR mode using Horos(LGPL license at Horosproject.org and sponsored by Nimble Co LLC d/b/a Purview in Annapolis, MD, USA) as a DICOM reader. All pathological changes including vertical bone loss, furcation involvement, and periradicular osteolysis were detected. Furthermore, the same datasets were analyzed automatically by Diagnocat (Diagnocat LLC, Prague, Czech Republic), a deep CNN. Finally, the performance of the dentists and the CNN were compared and evaluated.

Results

The CNN’s performance was significantly lower compared to the human readers in the search for different types of lesions. The human observers achieved good to very good interobserver agreement, except for the evaluation of the vertical lesions, which resulted in a moderate agreement.

Conclusion

The CNN used in this study was found to be ineffective in identifying periodontal lesions and was not adequately trained to offer significant assistance in the automated evaluation of periodontal lesions in CBCT datasets.

1. Introduction

1.1. Clinical and radiographic parameters of periodontitis therapy

For successful periodontitis therapy, a clinical periodontal diagnosis and radiographic assessment of hard tissue are fundamental. The clinical parameters are generally based on probing pocket depth, probing attachment level, and probing of the furcation entrance. Dental radiographs are necessary to evaluate the periodontal bone level, the presence of periapical lesions, and subgingival calculus (Braun et al., 2014, Kalkwarf and Reinhardt, 1988, Woelber et al., 2018). The drawback of using two-dimensional images is that they can only capture a simplified representation of three-dimensional anatomy. The resulting superimposition of structures in the missing third dimension causes limited accessibility to periodontal structures (Reddy, 1992). The rotation and changes in the angle of consecutively obtained radiographs may also lead to different two-dimensional images of the same three-dimensional situation (Eickholz et al., 1998). Tissue inflammation, probe type or diameter, and probing force regarding periodontal probing may lead to over- or underestimated scores of clinical attachment loss, which has a central role in defining a successful periodontal treatment plan (Akesson et al., 1992, Listgarten, 1980, Suomi et al., 1968, Tugnait et al., 2000). For this reason, accurate standardized radiographs are needed to get a more precise diagnosis resulting in a better treatment plan for periodontal diseases (Walter et al., 2015).

1.2. CBCT in periodontology

Cone beam computed tomography (CBCT) provides access to high-quality three-dimensional imaging. Its main benefits are the absence of distortion or variations in angles and the ability to assess anatomical structures effectively in all three dimensions (Ludlow et al., 2003, Mah et al., 2003). Even though the radiation dose of CBCT is significantly reduced compared to conventional CT, a general discussion is necessary to determine whether it could be used for general dental issues, such as periodontal treatments. High-quality CBCT images are easily available nowadays, but the interpretation of these images depends on the examiner.

1.3. The application of artificial intelligence (AI) in dentistry

AI refers to the use of computer technology to perform tasks that are typically done by humans. In dentistry, we have seen an increased use of AI, with a particular focus on machine learning (See Fig. 1).

Fig. 1.

Aspects of artificial intelligence.

Machine learning utilizes artificial neurons, which are mathematical non-linear models inspired by human neurons. These neurons are interconnected to form a network that can be trained to perform specific tasks, such as image classification. In dentistry, this technology can be used to determine whether a radiologic image shows a decayed tooth or not (Schwendicke et al., 2020). Popular fields of machine learning are the deep-learning-based convolutional neural network (CNN) and the artificial neural network (ANN) (Ahmed et al., 2021, Khanagar et al., 2021, Schwendicke et al., 2020).

AI can address the daily challenges of clinicians with improved precision, reduced staffing requirements, and fewer errors (Chen et al., 2020). It has been proven to be effective in classifying and detecting dental restorations and several maxillofacial pathologies (Abdalla-Aslan et al., 2020, Chen et al., 2020). Several studies included in reviews have demonstrated that AI systems based on automation perform exceptionally well and can even surpass the expertise of dental specialists (Ahmed et al., 2021, Khanagar et al., 2021).

2. Material and methods

2.1. Group size

Thirty-eight CBCT datasets were selected from an archive of a private dental imaging center. For all datasets, patients gave their signed informed consent regarding further anonymized scientific use. Initially, 40 datasets were selected for evaluation, but two datasets had to be excluded due to extensive metal artifacts.

2.2. Data extraction

All datasets were acquired previously with a Veraview X800 L P (JMorita Mfg. Corp., Kyoto, Japan). The acquisition volumes varied between 10 cm × 8 cm (diameter × height) and 8 cm × 8 cm. The exposure parameters were set at a tube voltage of 100 kV and an exposure time of 9.4 s, while the tube current varied between 4 mA and 8 mA. The voxel dimension was a combination of 0.25 mm for slice thickness (z-direction) and 0.125 mm for length and width (x and y). The datasets were imported into Horos™ (LGPL license at Horosproject.org and sponsored by Nimble Co. LLC d/b/a Purview in Annapolis, MD, USA) and anonymized by the principal investigator who was not involved in the reading of the datasets. Thus, the datasets were blinded for the human readers.

2.3. Analyzing CBCT datasets

Three general dentists with an average CBCT-reading experience of five years served as readers. They used Horos™ as the Digital Imaging and Communications in Medicine (DICOM) reader, imported the datasets, and read them in the 3D Multiplanar Reformation (MPR) mode. Prior to reading, the principal investigator used five additional anonymized datasets to calibrate the readers. They were allowed to adjust brightness, contrast, and slice thickness and to rotate the reference axes to angulate the different viewing planes accordingly. They were asked to look for the following pathological changes:

• 1.Vertical bone loss, which is defined as a reduction of the alveolar ridge more than 2 mm away from the enamel-cement-junction
• 2.Furcation involvement (FI), which is defined as bone loss between the roots starting from the roof of the furcation of multirooted teeth
• 3.Periradicular osteolysis, which is defined as a complete loss or absence of the bone surrounding a root.

2.4. Documentation

For documentation purposes, a web application (Simplymates GmbH & Co. KG, Freiburg, Germany) was used by all the readers. The aforementioned pathological changes were recorded in the documentation platform as follows:

• 1.All missing teeth were checked out to avoid reading errors.
• 2.Vertical bone loss was saved for four different sides around each affected tooth in three different stages: slight (cervical third of root), medium (middle third of root), and severe (apical third of root).
• 3.FI was saved in two different stages: moderate (incomplete loss of bone in furcation) and severe (complete absence of bone in furcation).
• 4.Periradicular osteolysis was saved for each affected tooth. There was no discrimination against each root.

Next, all datasets were uploaded to a deep CNN called Diagnocat (Diagnocat LLC, Prague, Czech Republic). Then, the datasets were analyzed automatically regarding the periodontal lesions detected by the system. These were summarized in a downloadable PDF named Radiology Report. The following periodontal lesions were used for comparison:

• 1.Periodontal bone loss (PBL), vertical type (mild, moderate, severe)
• 2.Periodontal bone loss (PBL), mixed type (mild, moderate, severe)
• 3.Furcation lesion
• 4.Detected missing teeth, implants, and pontics

These findings were transferred by one investigator from the Diagnocat output to the edit framework to generate a comparable readout.

Vertical and mixed types of vertical bone loss were transcribed as vertical bone loss without designating a specific root surface. Furcation lesions were transcribed as severe furcation involvement in the edit. Missing teeth, implants, and pontics were transcribed as missing teeth in edit.

2.5. Statistical analysis

The results from the readers and AI analysis were exported from the edit as CSV files and were processed for statistical evaluation in Excel (Microsoft Excel, Microsoft, Redmond, Wa, USA).

Statistical analysis was performed with DATAtab (DATAtab Team (2023) DATAtab: Online Statistics Calculator, DATAtab e.U. Graz, Austria. URL: https://datatab.net). For nonparametric data, the Wilcoxon test was used to compare the output of human observers and the CNN. Interrater reliability analysis was performed for the dependent samples of the three observers by calculating Fleiss’ kappa.

3. Results

Thirty-eight datasets were read by the observers and analyzed by CNN. Therefore, all results referred to a potential maximum number of 1,216 teeth.

Only one pair showed no significant difference (O3 – CCN for vertical lesions) while the Fleiss’ kappa value dropped accordingly to a moderate agreement in the observer group (see Table 1, Table 2).

Table 1.

Descriptive results – total findings (O1 = observer 1, O2 = observer 2, O3 = observer 3, periradicular = periradicular osteolytic changes, vertical = vertical lesions, furc = furcation involvement).

Finding	O1	O2	O3	CNN
Periradicular	14	17	22	0
Vertical 1	48	138	141	24
Vertical 2	70	49	119	56
Vertical 3	94	69	109	168
Initial furc	15	13	7	0
Severe furc	7	17	22	28

Table 2.

Comparison of all Wilcoxon test results (z value|p) for all pairs.

Finding	O1 – CNN	O2 - CNN	O3 - CNN	Fleiss kappa
Periradicular	−3.74|<0.001	−4.12|<0.001	−4.69|<0.001	0.94
Vertical	−4.54|<0.001	−5.38|<0.001	−1.37|<0.169	0.43
Initial furc	−3.87|<0.001	−3.61|<0.001	−2.65|<0.008	0.8
Severe furc	−5.29|<0.001	−3.58|<0.001	−3.13|<0.002	0.88

4. Discussion

This study aimed to evaluate whether a human examiner or an AI-based software was more efficient in detecting periapical osteolysis, vertical infrabony defects, and/or FI in CBCT datasets.

4.1. Periapical osteolysis

Regarding the detection of periradicular osteolysis, the results showed very high interrater reliability in comparison to Diagnocat. The AI-based software was not able to detect any periradicular osteolysis (p-value < 0.001). Periradicular osteolysis results in a poor prognosis for the survival of the specific tooth. Thus, the integration of this feature into Diagnocat would be useful.

4.2. Furcation involvement

This study also aimed to compare the accuracy of dentists in detecting FI in CBCT datasets compared to the AI-based software Diagnocat.

Compared to molars without FI, molars with FI Class I have a 100 % increased risk of tooth loss during supportive periodontal treatment for up to 10–15 years (Nibali et al., 2016). If a FI Class III is compared to a FI Class II, the relative risk of tooth loss increases to 3.13 (Dannewitz et al., 2006, Graetz et al., 2015, Johansson et al., 2013, McGuire and Nunn, 1996, Nibali et al., 2016, Salvi et al., 2014).

Regenerative therapy is recommended only in Class II FI. This shows a necessity for early diagnosis of FI to achieve higher long-term survival rates of periodontally treated teeth and better treatment options (Reddy et al., 2015).

The clinical diagnosis of beginning FI by periodontal probing often leads to wrong over- or underestimated scores of clinical attachment level (CAL). Conventional two-dimensional radiographs lead to the superimposition of periodontal structures. Therefore, the beginning FI is clinically very hard to detect and there is a need for better and more accurate diagnostics concerning FI of multirooted teeth (Müller and Eger, 1999).

The use of CBCT shows a high accuracy in detecting FI (Qiao et al., 2014, Walter et al., 2016). It can also identify several morphologic variations and pathologic observations pertinent to the decision-making process.

In this survey, the interrater reliability was high in detecting severe FI, which is clearly defined as a complete loss of bone in the furcation area. The performance of Diagnocat was significantly worse compared to the dentists in this survey (maximum p < 0,008).

4.3. Vertical bone loss

Besides FI, the detection of vertical bone loss was another aim of this survey. Deep and narrow bone defects enable regenerative periodontal treatment procedures that lead to radiographic bone gain and gain of CAL, as several studies have shown (Figueira et al., 2014, Nibali et al., 2021). The detection of these defects should have a central role in periodontal treatment planning. This is because periodontal regenerative therapy of infrabony defects results in high tooth survival rates even in long-term observation studies (Stavropoulos et al., 2021).

CBCT showed a higher accuracy in the detection of infrabony defects in comparison to 2D intraoral images, even though no statistically significant differences were found between clinical and CBCT measurements (Feijo et al., 2012, Raichur et al., 2012, Walter et al., 2010). The analysis of the buccal and lingual/palatal surfaces is only possible by CBCT (de Faria Vasconcelos et al., 2012).

4.4. CBCT versus conventional radiographs in periodontitis therapy

There is some evidence that CBCT seems to be a helpful supporting tool in regenerative periodontal surgery and furcation therapy of maxillary molars (Woelber et al., 2018). Nonetheless, the radiation dose of CBCT is relatively high in comparison to conventional radiographs. It is about 3–6 times a panoramic radiograph, 8–14 times an IO radiograph, and comparable to a full-mouth radiographic examination (Vandenberghe et al., 2008). Furthermore, dental restorations containing metal alloys can be a source of artifacts (e.g. extinction artifacts, beam hardening) that may interfere with the diagnostic process performed on CBCT (Schulze et al., 2011).

4.5. AI: A valuable assistant in periodontal diagnosis

This study aimed to evaluate whether a human examiner or an AI-based tool is more efficient and accurate in detecting PBL on CBCT datasets. Most of the published studies comparing dentists with AI reflect only on intra-oral or panoramic radiographs.

Lee et al. involved 693 intra-oral radiographs from randomly selected periodontal patients. These radiographs were examined by three independent dental specialists, including two periodontists and one periodontal resident. The results showed that there was no significant difference in radiographic bone loss percentage measurements between the examiners and the deep learning model, whereas the time required was significantly longer for the dentists (Lee et al., 2022).

To compare dentists versus a deep learning approach on panoramic radiographs, Krois et al. synthesized a set of 2001 image segments. The reference test was the percentual measurement of PBL. In total, six dentists examined the image segments for PBL. The CNN showed at least a similar discrimination ability compared to the dental specialists in examining PBL on panoramic radiographs. The study concluded that dentists’ diagnostic efforts could be reduced by applying an AI-based tool. Furthermore, the limited agreement between the dentists was shown in the study mentioned (Krois et al., 2019).

In general, severe FI and periradicular osteolysis are clearly defined as yes or no decisions that depend on the presence or absence of bone in the furcation area or around the root. In this study, the interrater reliability for these two defect morphologies was high. Compared to FI and periradicular osteolysis, infrabony defects could be differentiated into mild, moderate, and severe bone loss categories. The interrater reliability in the case of infrabony defects in this study was poor. This result could be due to different skills in reading CBCT datasets and the absence of an existing classification system that grades infrabony defects in CBCT datasets. Furthermore, depending on the orientation of each tooth in relation to the projection plane, the depiction of infrabony defects on CBCT datasets varied, especially regarding the depth and angulation of the infrabony defect.

In conclusion, the diagnostic performance of Diagnocat for periodontal diseases in this study was poor. The interrater reliability between the human examiners was at a good level in most cases. Diagnocat cannot replace the dentist as a diagnostician, but it could be a useful supporting tool to facilitate the diagnosis of CBCT datasets.

Ethical statement

For all datasets informed consent about the further anonymized scientific use has been given and signed by every patient.

Declaration of generative AI in scientific writing

During the preparation of this work the author(s) did not use any tool or service in order to prepare the manuscript.

CRediT authorship contribution statement

Dirk Schulze: Conceptualization, Methodology, Software, Formal analysis, Writing – review & editing, Supervision, Project administration. Lutz Häußermann: Validation, Investigation, Data curation, Writing – original draft. Julian Ripper: Validation, Data curation, Writing – original draft. Thomas Sottong: Validation, Investigation, Data curation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.