Examining the Role of Artificial Intelligence in Assessment: A Comparative Study of ChatGPT and Educator‐Generated Multiple‐Choice Questions in a Dental Exam

ABSTRACT

Aim

To compare the item difficulty and discriminative index of multiple‐choice questions (MCQs) generated by ChatGPT with those created by dental educators, based on the performance of dental students in a real exam setting.

Materials and Methods

A total of 40 MCQs—20 generated by ChatGPT 4.0 and 20 by dental educators—were developed based on the Oral Diagnosis and Radiology course content. An independent, blinded panel of three educators assessed all MCQs for accuracy, relevance and clarity. Fifth‐year dental students participated in an onsite and online exam featuring these questions. Item difficulty and discriminative indices were calculated using classical test theory and point‐biserial correlation. Statistical analysis was conducted with the Shapiro–Wilk test, paired sample t‐test and independent t‐test, with significance set at p < 0.05.

Results

Educators created 20 valid MCQs in 2.5 h, with minor revisions needed for three questions. ChatGPT generated 36 MCQs in 30 min; 20 were accepted, while 44% were excluded due to poor distractors, repetition, bias, or factual errors. Eighty fifth‐year dental students completed the exam. The mean difficulty index was 0.41 ± 0.19 for educator‐generated questions and 0.42 ± 0.15 for ChatGPT‐generated questions, with no statistically significant difference (p = 0.773). Similarly, the mean discriminative index was 0.30 ± 0.16 for educator‐generated questions and 0.32 ± 0.16 for ChatGPT‐generated questions, also showing no significant difference (p = 0.578). Notably, 60% (n = 12) of ChatGPT‐generated and 50% (n = 10) of educator‐generated questions met the criteria for ‘good quality’, demonstrating balanced difficulty and strong discriminative performance.

Conclusion

ChatGPT‐generated MCQs performed comparably to educator‐created questions in terms of difficulty and discriminative power, highlighting their potential to support assessment design. However, it is important to note that a substantial portion of the initial ChatGPT‐generated MCQs were excluded by the independent panel due to issues related to clarity, accuracy, or distractor quality. To avoid overreliance, particularly among faculty who may lack experience in question development or awareness of AI limitations, expert review is essential before use. Future studies should investigate AI's ability to generate complex question formats and its long‐term impact on learning.

1. Introduction

Humanity stands at the threshold of a remarkable transformation in education, driven by the advent of large language models (LLMs)—a specialised form of generative artificial intelligence (GenAI). These technologies are transforming how knowledge is shared and applied, challenging traditional methods and creating new opportunities for learners and educators [1, 2, 3]. Advanced systems like ChatGPT (OpenAI, San Francisco, United States) demonstrate the power of LLMs to transform learning, assessment and research in higher education, and research through advanced natural language processing and vast datasets [4]. Demonstrating versatility, ChatGPT has surpassed passing thresholds on high‐stakes exams such as the Uniform Bar [5] and Medical Licensing Exams [6], with OpenAI benchmarking its performance across various professional and academic tests, including dental exams [7, 8, 9, 10]. In classrooms, these models are revolutionising traditional teaching by generating diverse and tailored educational materials, such as essay prompts and case studies, designed to meet the unique needs of individual students [11, 12]. For instance, in dental and medical education, LLMs create realistic patient scenarios that enable students to practise diagnostic and decision‐making skills in low‐stakes, simulated environments [13]. Additionally, adaptive learning platforms powered by LLMs personalise content delivery, addressing specific knowledge gaps and reinforcing areas where students need additional support [14]. However, integrating AI into dental and medical education presents challenges, including concerns over chatbot reliability, data privacy, and the risk of student overreliance that may hinder critical thinking and problem‐solving [15, 16]. Additionally, educators often face difficulties due to limited knowledge or training in effectively incorporating AI technologies into their teaching. In line with these concerns, a global survey of dental educators found that while 31% had already used AI chatbots, 64% recognised their potential to enhance knowledge acquisition, research and clinical decision‐making [17]. Nonetheless, many educators pointed to the lack of clear guidance and insufficient training as key barriers to effective implementation, highlighting the need for clear dental guidelines [17, 18].

One of the most promising applications of LLMs lies in their ability to generate exam questions, producing a wide range of question types, including multiple‐choice questions (MCQs), case‐based scenarios and clinical problem‐solving exercises [19, 20, 21]. This is particularly important for addressing the heavy workload of dental or medical university staff, who balance responsibilities such as patient care, teaching, student assessment, research and administrative duties [20, 21, 22]. Additionally, the rising demand for high‐quality MCQs underscores the need for fair, accurate and objective‐aligned questions—a complex task requiring expertise, creativity and attention to bias, diversity and resource limitations [23]. These challenges highlight the potential of AI tools, like LLMs, to support educators by addressing these limitations while maintaining assessment rigour and quality.

Recent studies have explored the potential of ChatGPT in generating medical MCQs across various disciplines, including dermatology, physiology, neurosurgery and dental caries [20, 24, 25, 26, 27, 28, 29, 30, 31, 32]. In most of these studies [20, 25, 26, 27, 28, 30, 32], human experts or specialists evaluated these questions based on parameters such as validity, difficulty and accuracy, while others did not include any formal assessment [29, 31]. In one of these studies [20], questions generated by ChatGPT were compared to those created by university staff, showing that ChatGPT produced 50 MCQs 10 times faster than human examiners, with no significant difference in the quality of the questions. Additionally, two studies evaluated the performance of various LLMs in generating medical MCQs [27, 32]. One study [27] found that ChatGPT produced the most valid but least difficult questions compared to Bing and Bard, while the other [32] reported no significant differences in question quality between ChatGPT and Bard. The percentage of inaccurate or invalid questions generated by LLMs has been reported in a few previous studies [20, 25, 28], with most studies merely providing examples of irrelevant or inaccurate MCQs [26, 27, 29, 31]. Additionally, a phenomenon known as ‘ChatGPT's Hallucinations’ has been identified, where ChatGPT is reported to generate factually incorrect information in various studies [33, 34], highlighting the need for users to verify its outputs against reliable sources. During the writing of this paper, the most significant limitation identified in previous studies is that, to the best of our knowledge, only one study [24] has evaluated LLM‐generated MCQs in real‐world settings, such as actual examinations. Recently, Laupichler et al. [24] compared expert and ChatGPT generated MCQs on neurophysiology in a medical exam setting, reporting no significant difference in item difficulty but significantly higher discriminatory index for the expert‐generated questions.

Automated exam question generation by LLMs has the potential to support assessment design and reduce educator workload, but question quality must be evaluated before use in exams. The questions produced by these models must not only align with the learning objectives of the relevant curriculum but also meet quality standards, including valid content, appropriate difficulty and acceptable discriminatory index [19]. Therefore, the aim of this study was to compare the item difficulty and discriminative indices of MCQs generated by ChatGPT with those created by dental educators based on the performance of dental students in a real exam setting.

2. Materials and Methods

2.1. Generation of MCQs

Two distinct sets of 20 text‐based MCQs, each consisting of five answer options—one correct answer and four distractors—were prepared by dental educators and ChatGPT 4.0 for this cross‐sectional, comparative study (Figure 1). The same criteria were applied to both dental educators and ChatGPT 4.0 for generating the questions. The topics were determined beforehand and based on the content covered in the Oral Diagnosis and Radiology course lectures, which encompass subjects from both oral radiology and oral medicine. The MCQs primarily assessed foundational knowledge and concept application, with some incorporating clinical reasoning. The first set of 20 questions was generated by a panel of three dental educators specialising in Oral Diagnosis and Radiology, with an average of 17 years of experience. The educators were tasked with preparing 14 questions on oral radiology and 6 questions on oral medicine from scratch, ensuring a range of difficulty levels categorised subjectively as easy, medium and difficult (Table 1).

FIGURE 1.

The flowchart of the study.

TABLE 1.

Predetermined topics for MCQ generation, and question distribution across difficulty levels.

Topics and the corresponding number of questions for each domain	Difficulty level	Total (Educator + ChatGPT 4.0)
Oral radiology
Radiographic imaging methods, principles of imaging techniques (2 MCQs) Radiographic diagnosis of anatomical structures (2 MCQs) Radiation physics, radiographic image production, artefacts (4 MCQs) Harmful effects of radiation, radiation protection (1 MCQ) Radiographic diagnosis and/or findings of medical disorders (5 MCQs)	2 easy 8 medium 4 difficult	4 easy 16 medium 8 difficult		28 MCQs
Oral medicine
Diagnosis of oral mucosal lesions (2 MCQs) Oral findings of medical disorders (3 MCQs) Intra and extraoral examination (1 MCQ)	2 easy 2 medium 2 difficult	4 easy 4 medium 4 difficult		12 MCQs

ChatGPT 4.0 was accessed via a personal computer, and the course lectures were uploaded. One of the investigators, who is an expert in Oral Diagnosis and Radiology and was not involved in generating the educator‐written MCQs, prompted ChatGPT to generate 20 questions. The same criteria, topics and difficulty levels were applied to ensure consistency across both sets (Table 1). The model was prompted to generate complete MCQs from scratch to produce questions that included the stem, one correct answer and four distractors. No fine‐tuning or parameter adjustment was applied during the generation process, and the model was used in its default configuration via the standard OpenAI interface. As recommended in the literature [35], clear and concise prompts were used to ensure that each question aligned with the intended learning outcomes and adhered to an appropriate structure [36]. The prompt stem specified the question type, the targeted topic and the intended student audience (senior dental students). Additional instructions were presented as bullet points, outlining the educational level, the number of answer choices, the rationale for the correct answer and any constraints. The complete set of prompts used for ChatGPT 4.0‐generated questions is included as File S1. The questions were copied directly without any modifications. Furthermore, a new chat session was initiated for each question request to minimise memory retention bias.

2.2. Blinded Assessment of the MCQs

An independent, blinded panel of three educators specialising in Oral Diagnosis and Radiology, with an average of 15 years of experience and not involved in generating either the educator‐written or ChatGPT‐generated questions, was tasked with evaluating all the MCQs. All panel members have undergone training in measurement and evaluation, and prior to the evaluation, the panel participated in a calibration session to align on the evaluation criteria. The questions were presented in random order, and the evaluators were blinded to their origin. The assessment for accuracy/relevance was classified in two groups as ‘the question can be used’ and ‘the question cannot be used’. Inaccurate, irrelevant, or unclear questions were disqualified. The questions were then evaluated for their level of difficulty and classified as easy, medium, or difficult. In cases of disagreement during this process, the opinion of a fourth independent educator was consulted. If the disqualified questions were prepared by the educators, they were required to create new questions on the same topic and same difficulty level. Similarly, if the disqualified questions were generated by ChatGPT, the model was tasked with producing additional questions for that specific domain. The MCQ generation and blinded assessment process was repeated until each group achieved a total of 20 questions (14 oral radiology, 6 oral medicine), appropriately distributed across the specified difficulty levels (Table 1).

2.3. Voluntary Exam for 5th Grade Dental Students

After the MCQ generation process was finalised, an onsite, online exam consisting of a total of 40 questions was administered through a digital institutional education management and planning system (KEYPS, Kapitta Joint Stock Company, Ankara, Türkiye). This system, widely used for assessment and evaluation in undergraduate education within the faculty, automatically randomises questions for each student during online exams, eliminating the need for further randomisation. Fifth‐grade students were invited for the voluntary exam as they had completed all the theoretical courses related to Oral Diagnosis and Radiology curriculum. The students were informed both verbally and through messages and emails about the date/time and general conditions of participation in the study, emphasising that participation in the onsite exam was voluntary and would not affect their grades in subsequent evaluations. All of the students accessed the exam through mobile devices by logging into the system using their credentials, under supervised conditions. They were given 1 h to complete the exam and were blinded to the origin of the questions. Ethical approval for this part of the study was obtained from the Research Ethics Committee of the Faculty of Medicine, Ege University (No. 24‐3.1T/55), and informed consent was obtained from all participants.

After the exam, the reliability of the online test was determined using the Kuder–Richardson 20 (KR‐20) method. Subsequently, the item difficulty and discriminative indices were calculated for both educator‐generated and ChatGPT‐generated MCQs. The discriminative index for each question was determined using the point‐biserial correlation because the items were dichotomous [37], whereas the difficulty index was calculated based on the mean item score using classical test theory (CTT). RStudio (Version R‐4.4.2, JJ Allaire, Massachusetts, USA) was used for statistical analysis. The data was analysed using the Shapiro–Wilk test, paired sample t‐test and independent t‐test; p was set at 0.05.

3. Results

During the 2023–2024 academic year, our faculty had 176 5th‐year students, of whom 80 participated in the voluntary exam, representing 45% of the total 5th‐year cohort. All students completed the voluntary exam, consisting of 40 MCQs, within the provided timeframe, and no students' results were excluded. The reliability of the exam section prepared by the educators, calculated using the KR‐20 method, was 0.62, while the ChatGPT‐generated portion showed a reliability of 0.63, indicating the internal consistency of the exam [38].

Table 2 presents the questions generated by educators and ChatGPT 4.0, along with their correct answers, distractors and difficulty levels as evaluated by the independent, blinded panel. The educators developed an initial set of 20 MCQs in a single session. During the blinded evaluation conducted by the independent expert panel, three questions were disqualified due to unclear phrasing in the question stem, ambiguity among distractors, and insufficient alignment with the intended learning objectives. These questions were not revised or edited; instead, they were entirely discarded. The educators were requested to generate three new questions from scratch to replace the disqualified items. The entire process of finalising 20 educator‐generated questions was completed in approximately 2.5 h. The panel's rejection rate for the educator‐generated MCQs was 15%.

TABLE 2.

Questions generated by educators and ChatGPT 4.0 on predetermined topics, and difficulty levels as evaluated by the independent panel. The correct answers are marked with red colour.

	Topic	Difficulty level
Educator generated MCQs
#1 Which of the following statements regarding PET imaging are correct? I. PET scanners are similar to computed tomography (CT) units, but unlike CT, the radiation source in PET imaging is inside the body, II. The most commonly used radioactive tracer is 99mTc, III. CT can be combined with PET to determine the anatomical localization of radionuclides. (A) I and II, (B) I, (C) I, II and III, (D) I and III, (E) II and III	Oral radiology: Imaging methods	Difficult
#2 Which of the following imaging techniques do not use ionising radiation during imaging? I. Magnetic resonance imaging, II. Computed tomography, III. Ultrasonography, IV. Dental volumetric tomography, V. Fluoroscopy (A) I and IV, (B) II and V, (C) II and IV, (D) I and III, (E) III and V	Oral radiology: Imaging methods	Easy
#3 In a periapical radiograph taken from the canine tooth region of the upper jaw, which of the following anatomical formations is least likely to be seen? (A) Lateral fossa, (B) Nasolabial groove, (C) Anterior wall of the maxillary sinus, (D) Nasal fossa (E) Zygomatic arch	Oral radiology: Radiographic anatomy	Easy
#4 The patient who came to the radiology clinic to have their radiograph taken is asked to first turn their face towards the film receptor, then to tilt their head forward. The patient, with their frontal region touching the receptor, is asked to open their mouth. The central beam is directed perpendicular to the receptor and parallel to the midsagittal plane. In the radiography technique described above, which area is primarily intended to be visualised? (A) Maxillary sinuses, (B) Nasal cavity, (C) Mandibular condyles, (D) Coronoid processes, (E) Zygomatic arches	Oral radiology: Radiographic anatomy	Difficult
#5 Which of the following are characteristic features of the ghost image in panoramic views? I. It is blurred compared to the object's real image, II. It is on the opposite side of the object's real image, III. It is lower than the object's real image, IV. It has the same morphology as the object's real image. (A) Only I, (B) II and IV, (C) II, III and IV, (D) I and II, (E) I, II and IV	Oral radiology: Radiation physics, image production	Medium
#6 Which of the following are effects that can be observed when the kilovoltage (kVp) of the X‐ray machine is increased? I. Increase in the number of photons, II. Increase in the strength of beam penetration, III. Increase in image density (A) Only I, (B) Only II, (C) I, II and III, (D) I and II, (E) I and III	Oral radiology: Radiation physics, image production	Medium
#7 In digital imaging, contrast resolution is expressed by which of the following? (A) Bit depth, (B) Voxel size, (C) Pixel size, (D) Number of line pairs per mm, (E) Detector width	Oral radiology: Radiation physics, image production	Medium
#8 Which of the following provides the correct sequence of stages in the formation of X‐rays in a dental X‐ray machine? I. High‐speed electrons striking the tungsten target, II. Electrons separating from the heated filament, III. Increasing the potential difference between the anode and cathode (A) I‐II‐III, (B) II‐I‐III, (C) II‐III‐I, (D) III‐I‐II, (E) III‐II‐I	Oral radiology: Radiation physics, image production	Medium
#9 Which of the following is not effective in radiation protection? (A) Film speed, (B) Object‐film distance, (C) Collimation type, (D) Filtration, (E) Use of film holders	Oral radiology: Radiation protection	Difficult
#10 In the panoramic image of a 23‐year‐old male patient presenting with toothache in the maxillary molar region, a well‐defined, asymptomatic radiolucent lesion extending from the periapical area to the interradicular area is observed in the mandibular premolar region. When the clinician reaches the lesion, they find that the cavity is almost empty. Which of the following is the most likely diagnosis for this lesion? (A) Aneurysmal bone cyst, (B) Simple bone cyst, (C) Residual cyst, (D) Focal osteoporotic bone marrow defect, (E) Stafne bone cavity	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#11 Which of the following statements regarding anatomical markers and pathologies utilised in radiographic evaluation is incorrect? (A) Superior displacement of the inferior alveolar canal can be seen in fibrous dysplasia., (B) Enlargement of the inferior alveolar canal along with cortical bone may indicate vascular and neural‐origin benign lesions within the canal., (C) Irregular expansion with cortical bone destruction in the inferior alveolar canal may indicate malignant formations within the canal., (D) Irregular expansion of the periodontal ligament area can be seen in malignant diseases., (E) The presence of a radiolucent lesion typically suggests slowly developing lesions.	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#12 In the panoramic radiograph of a 45‐year‐old female patient taken for the assessment of wisdom teeth, bilateral, well‐defined, radiolucent areas with radiopacities in the centre associated with the roots of mandibular premolars and molars are observed. Clinical examination reveals unilateral composite fillings in the bilateral 1st and 2nd molars, with no complaints of pain or swelling related to these teeth. Vitality tests are performed on the teeth associated with the lesion, and it is determined that these teeth are vital. What is the most likely diagnosis for this patient? (A) Osteoporosis, (B) Cemento‐ossifying fibroma, (C) Condensing osteitis, (D) Fibrous dysplasia, (E) Florid cemento‐osseous dysplasia	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#13 Which of the following systemic diseases is most likely to show hypercementosis on radiological examination? (A) Hypophosphatasia, (B) Hypophosphatemia, (C) Renal osteodystrophy, (D) Hyperpituitarism, (E) Osteoporosis	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#14 Which of the following statements regarding the radiographic findings of leukaemia is incorrect? (A) It usually gives more frequent radiographic findings in developing tooth areas. (B) If it involves periodontal structures, crestal bone loss is observed. (C) It causes expansion. (D) It is seen as radiolucent areas in the jaws. (E) Rarely, periosteal new bone formation may be observed.	Oral radiology: Diagnosis and/or findings of medical disorders	Difficult
#15 A 45‐year‐old male patient presents with a white lesion on his right buccal mucosa that has been present for 1.5 years. Clinical examination reveals that the lesion is a homogeneous white plaque, raised from the surface, well‐defined, and does not scrape off. From his history, it is learned that the patient underwent a kidney transplant 4 years ago. What is the most likely diagnosis for this patient? (A) Chronic hyperplastic candidiasis, (B) Reticular lichen planus, (C) Pemphigus vulgaris, (D) White sponge nevus, (E) Frictional hyperkeratosis	Oral medicine: Diagnosis of oral mucosal lesions	Medium
#16 Which of the following is not a characteristic of oral squamous cell carcinoma? (A) Palpable induration is observed. (B) It is associated with chronic inflammation or irritation. (C) It is usually seen in middle‐aged and older men. (D) It can develop in all oral mucosa, especially on the floor of the mouth and lateral aspect of the tongue., (E) The 5‐year survival rate reaches 80% with surgical treatment, chemotherapy and radiotherapy.	Oral medicine: Diagnosis of oral mucosal lesions	Difficult
#17 A 28‐year‐old male patient presents with complaints of recurrent oral ulcers for the past 2 years. From his history, it is learned that he has abdominal and joint pain, weight loss and diarrhoea. Upon examination, aphthous lesions on the oral mucosa, as well as foldings in the lower buccal sulcus mucosa, gingivitis, angular cheilitis and granulomatous cheilitis are observed. What is the most likely preliminary diagnosis for the patient described above? (A) Wegener's granulomatosis, (B) Crohn's disease, (C) Behçet's disease, (D) Syphilis, (E) Systemic lupus erythematosus	Oral medicine: Oral findings of medical disorders	Medium
#18 A fifty‐year‐old female patient presents with complaints of painful sores inside the mouth. The patient describes that the sores inside the mouth first appear like blisters and then burst to become ulcers, and similar lesions have started to appear on her body as well. Upon intraoral examination, widespread ulcerative lesions are observed throughout the oral mucosa. When the mucosa adjacent to the erosive‐ulcerative areas is scratched with a probe, vesicle formation is noted. Histological examination reveals intraepithelial vesiculation along with acantholysis. The ELISA test shows positivity for anti‐dsg3. What is the most likely preliminary diagnosis for this patient? (A) Pemphigus vulgaris, (B) Erosive lichen planus, (C) Erythema multiforme, (D) Toxic epidermal necrolysis, (E) Wegener's granulomatosis	Oral medicine: Oral findings of medical disorders	Difficult
#19 Given the symptoms and findings below, what is the most likely diagnosis? Arthralgia, Positive pathergy test, Recurrent genital ulcers healing with scarring, Retinal vasculitis (A) Herpes zoster, (B) Behçet's disease, (C) HIV, (D) Crohn's disease, (E) Celiac disease	Oral medicine: Oral findings of medical disorders	Easy
#20 During the extraoral examination of the parotid gland region, which lymph nodes should be examined? (A) Submental lymph nodes, (B) Submandibular lymph nodes, (C) Occipital lymph nodes, (D) Preauricular lymph nodes, (E) Cervical lymph nodes	Oral medicine: Intra and extra oral examination	Easy
ChatGPT generated MCQs
#1 In panoramic radiography, what is the primary reason for the ‘tomographic movement’ of the X‐ray source and film? (A) To increase the field of view, (B) To reduce radiation exposure, (C) To focus on a specific plane and blur out other structures, (D) To enhance the contrast of the image, (E) To decrease the exposure time	Oral radiology: Imaging methods	Medium
#2 Which of the following is a characteristic of a charge‐coupled device (CCD) sensor used in digital intraoral radiography? (A) Employs a photodiode array to convert X‐rays into electrical signals, (B) Requires immediate chemical processing after exposure, (C) Utilises a digital sensor with a complementary metal‐oxide‐semiconductor (CMOS) design, (D) Exhibits high susceptibility to image distortion, (E) Has higher spatial resolution compared to film‐based receptors	Oral radiology: Imaging methods	Medium
#3 In dental radiography, which technique is employed to visualise buccolingual aspects of the maxillary and mandibular posterior teeth? (A) Bitewing radiography, (B) Periapical radiography, (C) Occlusal radiography, (D) Panoramic radiography, (E) Sialography	Oral radiology: Radiographic anatomy	Medium
#4 On a panoramic radiograph, what radiographic feature appears as a radiolucent line between the maxillary central incisors, representing the passage for nasopalatine nerve and vessels? (A) Anterior nasal spine, (B) Inferior nasal concha, (C) Incisive foramen, (D) Nasopalatine canal, (E) Median palatine suture	Oral radiology: Radiographic anatomy	Easy
#5 In dental radiography, what is the purpose of a grid? (A) Reduce patient exposure to radiation, (B) Enhance image contrast by reducing scattered radiation, (C) Produce sharper images of soft tissues, (D) Eliminate radiographic artefacts, (E) Improve the efficiency of the X‐ray machine	Oral radiology: Radiation physics, image production	Difficult
#6 What is the main purpose of the aluminium filter in a dental X‐ray tube? (A) Increase the energy of the X‐rays, (B) Absorb low‐energy X‐rays, (C) Focus the X‐ray beam, (D) Reduce the amount of scatter radiation, (E) Cool down the X‐ray tube	Oral radiology: Radiation physics, image production	Difficult
#7 In panoramic radiography, the ‘focal trough’ refers to: (A) The area of the film where the image is sharpest, (B) The path taken by the X‐ray beam, (C) The most radiopaque area on the film, (D) The area outside the patient's head, (E) The area within which structures are clearly imaged	Oral radiology: Radiation physics, image production	Medium
#8 What term refers to the reduction in radiation intensity as it passes through tissues? (A) Scatter radiation, (B) Attenuation, (C) Absorption, (D) Fluorescence, (E) Remission	Oral radiology: Radiation physics, image production	Difficult
#9 What is the primary risk associated with ionising radiation exposure in dentistry? (A) Skin burns, (B) Genetic mutations, (C) Cataract formation, (D) Increased blood pressure, (E) Allergic reactions	Oral radiology: Radiation protection	Easy
#10 A 70‐year‐old male patient visits the orthopaedic clinic with complaints of bone pain and deformity in his right leg. Clinical examination reveals warmth, tenderness and an increased circumference of the affected limb. The patient reports hearing loss and occasional headaches. A bone scan and X‐rays show areas of increased bone density, thickened cortices and a ‘cotton‐wool’ appearance. What is the most likely diagnosis for this patient based on the clinical and radiological findings? (A) Osteoporosis, (B) Osteosarcoma, (C) Rheumatoid arthritis, (D) Fibrous dysplasia, (E) Paget's disease	Oral radiology: Diagnosis and/or findings of medical disorders	Difficult
#11 Which radiographic feature is characteristic of an odontogenic myxoma? (A) Well‐defined unilocular radiolucency with a corticated border around an unerupted tooth, (B) Irregular radiolucency with scalloped borders in the periapical region, (C) Radiopaque mass with a ‘ground‐glass’ appearance in the mandible, (D) Multilocular radiolucency with soap bubble or honeycomb appearance, (E) Periapical radiolucency with loss of lamina dura and irregular borders	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#12 What radiographic characteristics are indicative of cherubism? (A) Well‐defined radiolucencies with scalloped borders, (B) Ground‐glass appearance, (C) Punched‐out radiolucencies, (D) Periapical radiolucency, (E) Thickening of the lamina dura	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#13 What malignant bone tumour most commonly affects the jaws, often presents as a rapidly growing, painful swelling, and exhibits a sunburst pattern on radiographic imaging? (A) Chondrosarcoma, (B) Ewing's sarcoma, (C) Osteosarcoma, (D) Fibrosarcoma, (E) Osteochondroma	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#14 What radiographic sign is commonly associated with rheumatoid arthritis affecting the temporomandibular joint (TMJ)? (A) Erosion of the articular surfaces, (B) Enlargement of the condyle, (C) Sclerosis of the joint margins, (D) Thinning of the cortical bone, (E) Subchondral cyst formation	Oral radiology: Diagnosis and/or findings of medical disorders	Medium
#15 Which of the following lesions is characterised by clustered vesicles on an erythematous base in the oral cavity (A) Aphthous stomatitis, (B) Herpetic stomatitis, (C) Mucocele, (D) Eosinophilic ulcer, (E) Pemphigus vulgaris	Oral medicine: Diagnosis of oral mucosal lesions	Easy
#16 A 40‐year‐old patient presents with a painless white lesion on the oral mucosa discovered during a routine dental examination. The lesion appears well‐defined and asymptomatic. There is no history of trauma, and the patient has no systemic conditions. What is the most common cause of the observed painless white lesion on the oral mucosa? (A) Candidiasis, (B) Leukoplakia, (C) Lichen planus, (D) Erythroplakia, (E) Geographic tongue	Oral medicine: Diagnosis of oral mucosal lesions	Easy
#17 A 38‐year‐old patient with a history of Addison's disease presents for a dental examination. The patient reports generalised weakness, weight loss and increased pigmentation of the skin and mucous membranes. During the intraoral examination, distinct pigmented lesions are observed on the buccal mucosa and gingiva. Considering the presented case, what type of oral pigmentation is commonly associated with Addison's disease? (A) Blue‐black macules, (B) White patches, (C) Red plaques, (D) Greenish discoloration, (E) Yellow nodules	Oral medicine: Oral findings of medical disorders	Medium
#18 A 30‐year‐old patient presents with a history of multiple odontogenic keratocysts detected during routine dental imaging. Upon further investigation, the patient reports having basal cell carcinomas on the skin, rib anomalies and a positive family history for similar conditions. Which rare genetic disorder is specifically associated with the presence of odontogenic keratocysts in the oral cavity? (A) Gardner syndrome, (B) Peutz‐Jeghers syndrome, (C) Gorlin‐Goltz syndrome, (D) Cowden syndrome, (E) Ehlers‐Danlos syndrome	Oral medicine: Oral findings of medical disorders	Difficult
#19 A 45‐year‐old patient presents with noticeable gingival enlargement during a routine dental examination. The patient's medical history reveals no significant dental concerns. Upon further inquiry, the patient mentions experiencing rapid weight loss, increased heart rate and difficulty sleeping. Considering the presented case, which systemic condition is commonly associated with the observed gingival enlargement? (A) Diabetes mellitus, (B) Hypertension, (C) Crohn's disease, (D) Hyperthyroidism, (E) Sjögren's syndrome	Oral medicine: Oral findings of medical disorders	Difficult
#20 In intraoral dermatoscopy (intraoral diascopy), what specific types of lesions exhibit diagnostic changes when pressure is applied? Identify two examples of oral lesions where intraoral diascopy is particularly useful in revealing characteristic alterations. (A) Fibroma and mucocele; demonstrating colour changes under pressure, (B) Erythroplakia and leukoplakia; emphasising vascular patterns, (C) Amalgam tattoo and melanotic macule; highlighting pigmentation alterations, (D) Lichen planus and pemphigus vulgaris; revealing characteristic blister formation, (E) Hemangioma and pyogenic granuloma; displaying blanching upon pressure	Oral medicine: Intra and extra oral examination	Medium

The generation of MCQs using ChatGPT was completed in 3 separate sessions, taking approximately 30 min in total, to achieve the required number of accurate/relevant questions, appropriately distributed across the specified difficulty levels. A total of 36 MCQs were generated by the model, of which 20 were considered accurate/relevant for inclusion in the exam. The remaining 16 questions, representing 44% of the total, were considered inaccurate/irrelevant by the panel. The most common issues involved the quality of distractors, with the model often providing repetitive or overly obvious options, which diminished the overall quality of the MCQs. Additionally, some ChatGPT‐generated questions exhibited bias, while a minority contained incorrect details.

During the voluntary exam, students correctly answered 40.4% of the educator‐generated MCQs and 41.9% of the ChatGPT‐generated MCQs (95% confidence interval [CI], −1.48 to 4.60). The normality of the students' answers for both groups was assessed using the Shapiro–Wilk test, which confirmed a normal distribution of the data (W = 0.975, p = 0.129). A paired sample t‐test was conducted to compare the number of correctly answered questions, revealing no statistically significant difference between the two groups of MCQs (t = 1.0235, df = 79, p = 0.309) (Figure 2). The raw data, including all student responses to each question, can be accessed through Files S2 and S3.

FIGURE 2.

Box plot presenting the distribution of correctly answered questions for ChatGPT 4.0‐generated and educator‐generated MCQs. Each box represents the range between the 25th percentile (lower quartile) and the 75th percentile (upper quartile), with the horizontal line indicating the median (50th percentile) for each group.

The normality of the difficulty (Educators: p = 0.242, ChatGPT: p = 0.094) and discriminative indices (Educators: p = 0.597, ChatGPT: p = 0.37) for two question groups was assessed separately using the Shapiro–Wilk test. Since all data followed a normal distribution (p > 0.05), comparisons were conducted using an independent t‐test.

Table 3 presents the difficulty and discriminative index values for questions generated by both educators and ChatGPT. The difficulty index ranges from 0 to 1, with the difficulty of a question increasing as the value approaches zero. Items with a difficulty level of 0.30 and below are classified as difficult; those with values between 0.31 and 0.70 are considered to have an acceptable difficulty, and values above 0.71 are categorised as easy [39, 40]. Based on these criteria, 45% (n = 9) of the educator‐generated MCQs were classified as difficult, while 55% (n = 11) fell within the acceptable range. For ChatGPT‐generated questions, 30% (n = 6) were categorised as difficult and 70% (n = 14) as acceptable (Table 3). Notably, neither group contained any questions categorised as easy, and the difficulty ratings assigned by the independent panel prior to the exam partially aligned with the actual exam results. The mean difficulty index for educator‐generated MCQs was 0.41 ± 0.19, compared to 0.42 ± 0.15 for ChatGPT‐generated MCQs. Both question groups demonstrated mean difficulty indices within the acceptable range. The questions prepared by the educators were slightly more difficult than those created by ChatGPT; however, this difference was not statistically significant (p = 0.773).

TABLE 3.

Difficulty and discriminative indices for MCQs generated by ChatGPT 4.0 and educators. Blue‐coloured cells indicate items with acceptable difficulty levels (0.31–0.70), while green‐coloured cells denote items with acceptable discriminative power (above 0.20).

ChatGPT 4.0 generated MCQs	Difficulty index	Discriminative index	Good quality MCQs	Educator‐generated MCQs	Difficulty index	Discriminative index	Good quality MCQs
#1	0.36	0.21	1	#1	0.13	0.22
#2	0.25	0.12		#2	0.53	0.53	1
#3	0.66	0.43	2	#3	0.61	0.25	2
#4	0.48	0.46	3	#4	0.29	0.25
#5	0.50	0.18		#5	0.24	−0.09 a
#6	0.35	0.25	4	#6	0.38	0.38	3
#7	0.26	0.25		#7	0.19	0.25
#8	0.18	0.06		#8	0.11	0.06
#9	0.68	0.44	5	#9	0.30	0.13
#10	0.68	0.50	6	#10	0.61	0.31	4
#11	0.38	0.50	7	#11	0.65	0.38	5
#12	0.25	0.28		#12	0.25	0.44
#13	0.53	0.25	8	#13	0.69	0.16
#14	0.26	0.37		#14	0.38	0.47	6
#15	0.29	0.34		#15	0.30	0.09
#16	0.43	0.09		#16	0.40	0.44	7
#17	0.55	0.50	9	#17	0.30	0.38
#18	0.35	0.21	10	#18	0.63	0.44	8
#19	0.49	0.31	11	#19	0.58	0.47	9
#20	0.50	0.65	12	#20	0.58	0.41	10
Mean	0.42 ± 0.15	0.32 ± 0.16			0.41 ± 0.19	0.30 ± 0.16

^a Item #5 generated by the educators exhibited negative discriminatory power.

The discriminative index, on the other hand, ranges from −1 to 1, with values closer to 1 indicating greater discriminative power of the item. While values above 0.20 represent an acceptable discriminative index, values between 0.30 and 0.39 are considered good, and those above 0.40 are regarded as excellent [39, 40]. Accordingly, 25% (n = 5) of the educator‐generated questions were classified as having an acceptable discriminative index, 15% (n = 3) as good and 35% (n = 7) as excellent (Table 3). Five questions (25%) prepared by the educators had discriminative index values below 0.20, indicating that these questions were unable to effectively distinguish between high‐performing and low‐performing students. For ChatGPT‐generated questions, 30% (n = 6) demonstrated an acceptable discriminative index, 15% (n = 3) were classified as having good discriminative power, and 35% (n = 7) were categorised as excellent (Table 3). Four questions (20%) prepared by ChatGPT had discriminative index values below 0.20, indicating poor discriminative power. The discriminative index of the educator‐generated MCQs, as expressed by the point‐biserial correlation, averaged 0.30 ± 0.16, while the mean value for ChatGPT‐generated questions was calculated as 0.32 ± 0.16. Although the questions generated by ChatGPT exhibited slightly higher mean discriminative index values, both groups fell within the good discriminative range, with no statistically significant difference between the two (p = 0.578). One reason for the lower mean discriminative index for educator‐generated questions was item #5, which had a negative discriminatory power of −0.09 (Table 3), favouring lower‐performing students over higher‐performing ones. Upon detailed evaluation of this question and the students' responses, it was observed that most students selected option E instead of the correct answer D. This was likely due to the presence of a strong distractor (IV: It has the same morphology as the object's real image) (Table 2), which not only influenced their choices but also made the question difficult, as reflected by its low difficulty index of 0.24.

Although the selection of questions based on their difficulty or discriminative power may vary depending on the specific requirements of the exam and the individual preferences of the educator, analysis of Table 3 focusing solely on items meeting both acceptable difficulty and discriminative index thresholds revealed that 12 (60%) ChatGPT‐generated questions and 10 (50%) educator‐generated questions qualified as ‘good quality’ [39, 40]. These questions effectively demonstrated balanced difficulty and strong discriminative performance, reflecting their suitability for reliable assessment.

4. Discussion

Current discussions on AI in education often focus on academic dishonesty, particularly the misuse of tools like ChatGPT for plagiarism and institutions' efforts to counter it with AI‐based detection systems [41]. However, when used wisely and strategically, incorporating AI technologies into educational practices offers substantial potential to enhance teaching methods and boost learning outcomes. These advancements create opportunities to promote more efficient, adaptable and inclusive learning environments, benefiting both educators and students in reaching academic success [1, 2]. In parallel with these developments, recent efforts have emphasised the importance of training dental professionals to critically evaluate and responsibly integrate AI into both educational and clinical settings [17, 18]. Schwendicke et al. [42] proposed a core curriculum for dental AI education, developed through a Delphi process involving 28 experts, which outlines foundational learning outcomes designed for flexibility and institutional adaptation. To complement such initiatives, a recent scoping review of university‐level GenAI guidelines revealed significant variability in policy development and implementation. While institutions like Harvard, Michigan and Zurich advocate for responsible and transparent AI use, professional dental education bodies such as ADEA and ADEE still lack comprehensive guidance. Key barriers to GenAI integration include cost, infrastructure limitations, resistance to change and ongoing concerns over academic integrity [17]. These findings highlight the need for clear policies, faculty training and curricular updates to ensure AI enhances rather than compromises the quality and ethics of dental education [18].

One emerging application of these models is the generation of exam questions, a task that is both demanding and complex [19, 20, 21]. This is particularly relevant for MCQs, which have been widely used for knowledge‐based assessment since the early 20th century. They are a crucial and essential component of both undergraduate and postgraduate examinations due to their standardisation, fairness, objectivity, cost‐efficiency and reliability [43, 44]. Compared to essay or short‐answer questions, MCQs enable the evaluation of a wider range of content, as exams can include a large number of questions [43]. However, a major drawback of the MCQ format is the difficulty and considerable effort required to develop high‐quality questions. This process requires solid dental or medical knowledge, thoughtful integration, and careful attention to avoid pitfalls such as repeated use of questions or design flaws that reduce assessment reliability [19].

To alleviate these challenges, especially in healthcare education, LLMs like ChatGPT are increasingly used to assist in MCQ development [20, 24, 25, 26, 27, 28, 29, 30, 31, 32]. In some of these pioneering studies, only a small number of medical MCQs (fewer than five) have been generated by the model, primarily for experimental and observational purposes [26, 29, 31]. Other studies have used various LLMs to generate MCQs on medical topics [20, 24, 25, 27, 28, 30], with one study specifically focusing on dental caries [32]. Cheung et al. [20] conducted the first and currently the only study comparing LLMs to humans in the generation of MCQs. ChatGPT‐3.5 Plus was used without post‐editing, and while it performed reasonably well, human‐generated questions were significantly superior in both quality and validity. Some studies involved educators in assessing model‐generated questions, evaluating validity, difficulty and accuracy using different rating scales [25, 26, 27, 28, 29, 30, 32]. Findings from these studies underscore the potential of LLMs to efficiently generate high‐quality dental and medical MCQs in a significantly shorter time [20, 25, 27, 28, 30, 32]. However, it was also reported that LLMs are not yet ready to replace human educators in the question development process and that all AI‐generated questions should undergo rigorous review and validation before being used in assessments [25, 27, 28, 30]. None of the studies mentioned have evaluated LLM‐generated MCQs in a real‐world setting, such as actual examinations administered to dental or medical students.

This study implements ChatGPT‐generated MCQs in a real dental examination setting, specifically administered to 5th‐year dental students. In a recent study with a similar design, Laupichler et al. [24] compared expert‐generated and ChatGPT‐generated MCQs on neurophysiology within a medical exam setting. However, the model‐generated questions were not reviewed before the exam, leading the authors to exclude 16% (n = 4) of the ChatGPT‐generated questions due to completely or partially incorrect answer options identified after the exam had concluded [24]. Their results revealed no significant difference in item difficulty; however, the expert‐generated questions demonstrated a significantly higher discriminatory index. In the present study, a thorough evaluation of both educator‐generated and ChatGPT‐generated MCQs was conducted prior to their inclusion in the actual exam. Educators prepared 20 MCQs in approximately 2.5 h, with only three requiring minor revisions after review. In contrast, ChatGPT generated 36 questions in three brief sessions, taking around 30 min in total. Of these, 20 (56%) were accepted as accurate and relevant, while 16 (44%) were excluded by the independent panel due to repetitive distractors, factual inaccuracies, or subtle bias. Despite this exclusion rate, ChatGPT successfully produced a complete and usable set of MCQs in a fraction of the time required by human educators. This outcome underscores both the efficiency of LLMs in generating assessment content and the essential role of expert review in ensuring item quality. This trade‐off between speed and quality control is a key consideration when evaluating the practical role of LLMs in assessment design.

A suggested by previous studies [20, 27, 35], ChatGPT 4.0 was provided with clear and precise prompts to ensure that the generated questions aligned with the learning outcomes and were properly structured. The MCQs were directly used in the exam without any modifications if they were accurate, relevant and appropriately distributed across the designated difficulty levels. This approach was implemented to accurately assess the model's performance in generating exam‐ready MCQs without any human intervention or correction. The voluntary exam was designed to balance easy, moderate and difficult questions, aiming to assess a wide spectrum of knowledge and skills. This structure enables fair evaluation across proficiency levels, measuring both foundational knowledge and advanced problem‐solving skills [23]. However, the difficulty ratings assigned by the independent panel before the exam did not fully align with the actual exam results, as students perceived some questions labelled as ‘easy’ to be of medium or even higher difficulty compared to the educators' evaluations. This outcome was not entirely unexpected, as the exam was conducted on a voluntary basis and scheduled outside the mid‐term period, resulting in students being less prepared for the assessment. Nevertheless, the mean difficulty index values for both educator‐generated and ChatGPT‐generated MCQs remained within the acceptable range [39, 40]. Although the educator‐generated questions were slightly more difficult, there was no statistically significant difference between the two groups.

In terms of discriminatory power, ChatGPT‐generated questions showed slightly higher mean discriminative index values; however, both groups remained within the good discriminative range [39, 40], with no significant difference between them. The number of questions with poor discriminatory power was similar in both groups. Notably, one of the educator‐generated MCQs (#5, Table 2) exhibited negative discriminatory power, contributing to a lower mean discriminative index for the educator‐generated questions. This was largely due to the effectiveness of a distractor that significantly misled students and made the question more challenging than intended. The item's low difficulty index further supported this (Table 3), indicating that even high‐performing students struggled with the question. The decision to include a difficult question with negative discriminatory power in an actual exam can be determined by the educator, depending on the exam's scope, specific objectives and individual preferences. However, this finding underscores the importance of carefully designing distractors and highlights the need for rigorous pre‐assessment review, even for educator‐generated questions, to ensure valid, fair and reliable evaluation.

The concept of a ‘quality question’ involves multiple factors that determine how effectively a question assesses the intended knowledge or competencies [43]. Key criteria contributing to question quality include clarity and accuracy, relevance, discriminatory power, fairness and the absence of bias, the effectiveness of distractors, conciseness, measurability and an appropriate level of complexity and challenge [43, 44, 45, 46]. This study assessed two key aspects of question quality—difficulty and discriminative index values—for both educator‐generated and ChatGPT‐generated questions, using established standards and predefined thresholds [39, 40]. Among the questions meeting acceptable difficulty and discriminative index standards, 12 ChatGPT‐generated and 10 educator‐generated questions were classified as ‘good quality’, demonstrating balanced difficulty and strong discriminative performance suitable for reliable assessment. Nevertheless, the choice of individual question selection ultimately depends on the exam's objectives, the specific competencies being assessed, and the educator's professional judgement. It is important to note that while the efficiency and quality of AI‐generated MCQs are promising, our results underscore a critical caveat: overreliance on generative AI may inadvertently compromise assessment quality, especially among less experienced or unaware faculty members [20, 33]. In dental education, where creating nuanced and context‐appropriate questions is essential, limited awareness of AI's limitations, including inaccuracies, repetitive distractors, or bias, can lead to suboptimal exam items if not carefully reviewed [33, 34]. Therefore, it is essential that AI‐generated questions undergo thorough expert validation to ensure they align with curricular objectives and maintain high standards of reliability. Although this study focused on difficulty and discriminative index as primary indicators of question quality, future research should build upon these findings by examining additional quality factors and assessing the long‐term effects of AI‐generated assessments on student learning outcomes. Additionally, exploring AI's potential to develop more complex question types, such as case‐based or key feature questions, could further expand its role in enhancing educational assessments.

5. Conclusion

This study is among the first to evaluate the use of ChatGPT‐generated MCQs in a real‐world dental examination setting. Our findings revealed no statistically significant differences between educator‐generated and ChatGPT‐generated MCQs in terms of difficulty and discriminatory power, with both sets of questions falling within acceptable ranges. While ChatGPT‐generated questions showed slightly higher mean discriminative index values, educator‐generated items were marginally more difficult. However, it is important to note that 44% of the initial ChatGPT‐generated MCQs were excluded by the independent panel due to issues related to clarity, accuracy, or distractor quality. These results suggest that LLMs, particularly ChatGPT, may support educators by significantly reducing the time required for assessment development, thereby enabling greater focus on teaching and individualised student support. Nonetheless, expert review remains essential to ensure content quality and alignment with educational standards.

Author Contributions

Conception or design of the study: B.İ., P.G. Collection, analysis, or interpretation of the study data: N.E.Ö., G.B., Y.B., B.İ., P.G. Writing or critical review with important intellectual contribution: B.İ., P.G., Y.B. Final approval of the version to be published: N.E.Ö., G.B., Y.B., B.İ., P.G.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

File S1: Complete set of prompts used for ChatGPT‐generated MCQs.

File S2: Raw data; Chat GPT‐generated questions and students responses.

File S3: Raw data; educator‐generated questions and students responses.

Özer N. E., Balcı Y., Bölükbaşı G., İlhan B., and Güneri P., “Examining the Role of Artificial Intelligence in Assessment: A Comparative Study of ChatGPT and Educator‐Generated Multiple‐Choice Questions in a Dental Exam,” European Journal of Dental Education 30, no. 3 (2026): 881–895, 10.1111/eje.70034.

Data Availability Statement

The data that supports the findings of this study are available in the Supporting Information of this article.