Performance of ChatGPT 3.5 and 4 on U.S. dental examinations: the INBDE, ADAT, and DAT

Abstract

Purpose

Recent advancements in artificial intelligence (AI), particularly tools such as ChatGPT developed by OpenAI, a U.S.-based AI research organization, have transformed the healthcare and education sectors. This study investigated the effectiveness of ChatGPT in answering dentistry exam questions, demonstrating its potential to enhance professional practice and patient care.

Materials and Methods

This study assessed the performance of ChatGPT 3.5 and 4 on U.S. dental exams - specifically, the Integrated National Board Dental Examination (INBDE), Dental Admission Test (DAT), and Advanced Dental Admission Test (ADAT) - excluding image-based questions. Using customized prompts, ChatGPT’s answers were evaluated against official answer sheets.

Results

ChatGPT 3.5 and 4 were tested with 253 questions from the INBDE, ADAT, and DAT exams. For the INBDE, both versions achieved 80% accuracy in knowledge-based questions and 66-69% in case history questions. In ADAT, they scored 66-83% in knowledge-based and 76% in case history questions. ChatGPT 4 excelled on the DAT, with 94% accuracy in knowledge-based questions, 57% in mathematical analysis items, and 100% in comprehension questions, surpassing ChatGPT 3.5’s rates of 83%, 31%, and 82%, respectively. The difference was significant for knowledge-based questions (P=0.009). Both versions showed similar patterns in incorrect responses.

Conclusion

Both ChatGPT 3.5 and 4 effectively handled knowledge-based, case history, and comprehension questions, with ChatGPT 4 being more reliable and surpassing the performance of 3.5. ChatGPT 4’s perfect score in comprehension questions underscores its trainability in specific subjects. However, both versions exhibited weaker performance in mathematical analysis, suggesting this as an area for improvement.

Introduction

Artificial intelligence (AI) has become a trending topic in recent years, and its applications are expanding exponentially across various fields worldwide.1 AI, or artificial intelligence, involves using computers to mimic human intelligence, performing tasks that typically require human capabilities such as understanding, reasoning, and decision-making.1,2

AI software will eventually provide an efficient way for the world of education and learning. AI programs are frequently used across a variety of fields such as engineering, marketing, and medicine.3 Numerous tasks have become easier as a result of using AI in daily life.2 Despite the advantages of AI, many people are still unfamiliar with its principles.

OpenAI is a private research laboratory that was founded in December 2015,4 with the goal of making rapid progress in AI technologies.4 One of its notable innovations is ChatGPT (short for “Generative Pre-Trained Transformer”), a publicly accessible tool.4,5,6 ChatGPT is based on the GPT language model.7 Alongside Microsoft Bing and Google Bard, ChatGPT is recognized as a prominent AI chatbot. It utilizes deep learning AI techniques to generate responses that closely mimic human interaction in natural language.8 ChatGPT is an example of a large language model (LLM), which is trained on text and produces textual content.9,10 LLMs have been applied to various areas in the medical field, and these models represent a notable advancement in the field of AI.11

In dentistry, ChatGPT offers a variety of services for medical personnel, including diagnosis, disease prevention, medication management, and reduction of medical errors.11 These applications have the potential to significantly improve healthcare and dentistry by promoting patient engagement and self-determination.

ChatGPT is a chatbot that utilizes the GPT-3 and GPT-4 language models.4,7 This tool is primarily designed to gener-ate human-like responses to text inputs and fulfill a wide range of text-based requests, from answering complex questions to generating brief texts that mimic human language.5,12 Users can access it through various platforms, including mobile apps and websites, either by text or voice.5,6,10

Recent publications have demonstrated that ChatGPT can accurately answer exam questions, including those from the United States Medical Licensing Examination. The authors have also challenged it to respond to a variety of other exam questions.8

This study aimed to further test and evaluate both the qualitative and quantitative performance of ChatGPT on exam questions in the field of dentistry in the U.S., to deter-mine if ChatGPT is capable of reaching the passing thre-shold. The null hypothesis was that ChatGPT cannot assist dental students and dentists in answering dental questions.

Material and Methods

Official question samples from 3 U.S. dental examinations - specifically, the Integrated National Board Dental Examination (INBDE), Dental Admission Test (DAT), and Advanced Dental Admission Test (ADAT) - were collected.

INBDE and ADAT questions were categorized into two types: Knowledge-based questions and Case history questions. Additionally, questions that included images were excluded. Similarly, DAT questions were divided into three categories: Knowledge-based questions, Mathematical analysis questions, and Comprehension questions, with image-based questions also being excluded. For the INBDE and ADAT examinations, the following prompts were used to solicit responses from ChatGPT 3.5 and 4. For knowledge-based questions, the prompt was: “You are a dentist, which is taking a dental examination, please chose the best answer for the following question.” For case history questions, the prompt was: “You are a dentist, which is taking a dental examination, based on the following case history, chose the best answer.”

For the DAT examination, the following prompts were used to ask ChatGPT 3.5 and 4 to answer the questions. For knowledge-based questions, the prompt was: “You want to apply for dental school and need to pass the DAT (Dental admission test) exam, please answer the following questions from the DAT exam as best as you can.” For mathe-matical analysis questions and chemical equation questions, the prompt was: “You want to apply for dental school and need to pass the DAT (Dental admission test) exam, please answer the following mathematical questions from the DAT exam.” For comprehension questions, the prompt was: “You want to apply for dental school and need to pass the DAT (Dental admission test) exam, please answer the following questions from the DAT exam as best as you can. Based on the passage provided in the next prompt please answer the questions.” The distribution of each question type for each examination is shown in Table 1.

Table 1. Breakdown of each examination.

Both ChatGPT 3.5 and 4 were asked to answer the questions. The accuracy of their responses was checked based on the answer sheet that was provided by the examiner.

Each examination was evaluated separately according to the categories of the questions. Initially, the percentage of correct answers was assessed for each of the four categories: knowledge-based, case history, mathematical, and comprehension questions. The chi-square and Fisher exact tests were used to evaluate the performance of ChatGPT 3.5 com-pared to ChatGPT 4. The significance level was set at 0.05.

Results

The ChatGPT 3.5 and ChatGPT 4 models were used to examine a total of 49 questions from the INBDE examination, 80 questions from the ADAT examination, and 124 questions from the DAT examination.

ChatGPT 3.5 and ChatGPT 4 performed well on the INBDE, particularly in the knowledge-based questions, where both versions answered correctly 80% of the time. In the case history questions, ChatGPT 4 had a slightly higher success rate, answering 69% correctly, compared to ChatGPT 3.5, which answered 66% correctly (Fig. 1).

Fig. 1. Performance of ChatGPT 3.5 and 4 on the INBDE (A), ADAT (B), and DAT (C) examinations.

In the ADAT questions, ChatGPT 4 correctly answered 83% of the queries, while ChatGPT 3.5 had a success rate of 66%. Regarding the case history questions, both ChatGPT 3.5 and ChatGPT 4 achieved a correct response rate of 76% (Fig. 1).

For the DAT, in the knowledge-based questions, Chat GPT 4 correctly answered 94% of the questions, while ChatGPT 3.5 had a correct response rate of 83%. In mathematical analysis, ChatGPT 4 achieved a correct answer rate of 57%, compared to 31% for ChatGPT 3.5. The chi square test indicated that ChatGPT 4’s improved performance was statistically significant (P=0.021). In chemistry calculations, ChatGPT 4 answered 68% of the questions correctly, whereas ChatGPT 3.5 answered 56% correctly; however, this difference was not statistically significant (P=0.465). For comprehension questions, ChatGPT 4 had a 100% correct response rate, significantly outperforming ChatGPT 3.5, which answered 82% correctly. The Fisher exact test revealed no significant difference in performance on these questions (P=0.227) (Fig. 1).

Overall, in the knowledge-based question section, Chat GPT 4 correctly answered 88% of the questions, while ChatGPT 3.5 correctly answered 78%. The chi-square test revealed that ChatGPT 4’s performance was significantly better (P=0.009). In the case history questions section, ChatGPT 4 correctly answered 71% of the questions, compared to 70% for ChatGPT 3.5. The chi-square test indicated no significant difference in performance between the two versions (P=0.841) (Fig. 2).

Fig. 2. Overall topics performance on question types.

Interestingly, both ChatGPT 3.5 and ChatGPT 4 provided the same incorrect answers for 6 out of the 9 knowledge-based questions they answered incorrectly, representing a 66% overlap. In the case history questions section, they both answered 12 questions incorrectly, with identical errors in 7 of these, indicating a 58% overlap. In the mathematical analysis section, they gave the same incorrect responses to 6 out of 11 incorrectly answered questions, a 54% overlap. Lastly, in the chemistry calculation questions section, they both answered 4 questions incorrectly, with 2 being the same, showing a 50% overlap. This pattern suggests that ChatGPT 3.5 and ChatGPT 4 tend to make similar errors (Fig. 3).

Fig. 3. Overlap in mistakes between different sections.

Discussion

Since its establishment in December 2015,4 OpenAI has become a key player in the development of AI technology. A significant aspect of this advancement is the creation of the ChatGPT series,7 which is based on the Generative Pre-Trained Transformer architecture. Notable advancements were made with the release of ChatGPT 3.5 and ChatGPT 4, with each version improving performance through training on increasingly larger text datasets.

To enhance the accuracy and intricacy of its responses, ChatGPT 3.5 was trained on a broader variety of materials, serving as a transitional enhancement over its earlier versions. The most current version, ChatGPT 4, builds on this foundation by incorporating an even larger and more diverse training sample. As a result, the model now delivers answers that are far more nuanced and contextually rich.4

In this study, ChatGPT - particularly ChatGPT 4 - demonstrated promising results in taking U.S.-based dental examinations. In the knowledge-based questions, ChatGPT 4 correctly answered 88% of the items, showing a statistically significant improvement over ChatGPT 3.5. Additionally, in the case history questions section, ChatGPT 4 correctly answered 71% of the questions. These results suggest that ChatGPT 4 can easily pass the dental board examination tests.

In the comprehension section, text was provided to ChatGPT 4 and it was asked questions. The model answered all questions correctly, demonstrating its capability to be trained effectively to meet the specific needs of this field.

Although ChatGPT demonstrated proficiency in knowledge-based, case history, and comprehension questions, its performance on mathematical questions was less impressive. In the ADAT mathematical questions section, Chat GPT 4 correctly answered 57% of the questions, while ChatGPT 3.5 managed only 31%. Despite the improved performance of ChatGPT 4 over ChatGPT 3.5 in mathematical analysis questions, it still cannot be considered a reliable resource for assisting students with these types of questions.

Another point to consider is the possibility of hallucination in LLMs, such as ChatGPT. Hallucination in an LLM is defined as “generated content that is nonsensical or unfaithful to the provided source content,” which can degrade system performance and fail to meet user expectations in many real-world scenarios.12 This issue may arise when a user questions the LLM with prompts like, “Are you sure?” or “I think your answer is incorrect, can you double-check?” Such inquiries can confuse the LLM, leading it to change a correct response to an incorrect one. Hallucination can also occur in contexts such as dental examinations, highlighting the need for caution to prevent it when using these systems.

ChatGPT 3.5 and 4 have been developed and trained using data available on the internet up to the September 2021, and January 2023 respectively. It can address a wide range of questions effectively, demonstrating strong performance in knowledge-based and case history queries. Notably, it achieves a perfect score of 100% in the comprehension questions section. This underscores the capability of LLMs like ChatGPT to be trained extensively in specific subject areas, thereby maximizing their potential.

Many aspects of natural language processing software and LLMs remain to be tested. However, the current study demonstrates the potential of ChatGPT in assisting dentists and dental students during dental examinations. However, there is a need to develop more LLMs that are specifically trained and tailored for medical and dental subjects.

In conclusion, both ChatGPT 3.5 and ChatGPT 4 are capable of answering knowledge-based, case history, and comprehension questions effectively, achieving good scores. ChatGPT 4 outperformed ChatGPT 3.5 in most aspects, demonstrating greater reliability. Notably, ChatGPT 4 achieved a perfect score on the comprehension questions, reflecting its robust training in the user-selected subject matter. However, both versions exhibited weaker performance in mathematical analysis questions, underscoring the need for further improvement in this area.