A competency-based assessment system for undergraduate oral and maxillofacial surgery internship: development and pilot validation

Abstract

Background

Traditional assessment in undergraduate oral and maxillofacial surgery (OMFS) internships mostly relies on single-skill examinations, focusing only on technical proficiency while ignoring communication ability, clinical reasoning and medical documentation. This study aimed to develop and pilot-validate a comprehensive competency-based assessment system for undergraduate OMFS outpatient internships.

Methods

This mixed-methods development and pilot validation study identified six core clinical competency domains through literature review and semi-structured interviews with five senior OMFS faculty members. A multi-dimensional assessment framework was established, including theoretical examination, integrated clinical operation evaluation, phased skill assessment aligned with national licensing standards, patient-related indicators and daily performance evaluation. Forty-six students in the development cohort (2024–2025) adopted the new system; 48 students in the reference cohort (2023–2024) received traditional assessment. Each student was independently evaluated by three faculty members from an eleven-person rater panel. Inter-rater reliability was calculated using the intraclass correlation coefficient. Questionnaire feedback and adverse event data were compared between groups.

Results

Six core competency domains were finalized. Student ratings of the comprehensiveness of the two assessment systems were not significantly different (8.14 vs. 6.52, P = 0.086). Faculty ratings showed a significant difference in comprehensiveness (9.64 vs. 6.28, P = 0.043). The overall inter-rater reliability was good (ICC = 0.78, 95% CI: 0.65–0.87). Based on participant feedback, the weight of theoretical examination and patient satisfaction was reduced, and more objective indicators were added. No significant between-group differences were found in adverse event incidence.

Trial registration

Not applicable.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-10026-y.

Introduction

Competency-based education (CBE) has become the mainstream reform direction in global health professions education, shifting from time-centered training to outcome-oriented competency cultivation [1]. In dentistry, this shift has placed new demands on assessment design: evaluation systems must now capture not only technical proficiency but also the abilities of clinical reasoning, communication, and professionalism that together constitute clinical competence [2]. This challenge is amplified in procedure-intensive specialties such as oral and maxillofacial surgery (OMFS), where sound assessment must simultaneously reflect a trainee’s ability to perform surgical procedures, diagnose correctly, and deliver holistic patient care [3].

Nevertheless, most domestic undergraduate OMFS internships still adopt single-item skill assessment represented by impacted tooth extraction [4, 5]. Such evaluation methods are relatively fragmented and subjective, failing to systematically examine students’ overall clinical competency [6]. These essential abilities remain largely unmeasured in current OMFS internship evaluation. In addition, the gap is further compounded by significant regional and institutional disparities in teaching resources, patient populations, and assessment standards across China [7, 8]. Consequently, internship evaluations may lack the standardization needed to serve as valid measures of clinical competence. It is urgent to construct a standardized, multi-dimensional and operable competency-based assessment system suitable for local undergraduate OMFS internship teaching.

Therefore, this study aimed to: (1) construct a targeted clinical competency assessment framework for OMFS undergraduates; (2) implement the system in a real teaching environment and collect feedback from students and instructors; (3) preliminarily verify its feasibility and reliability, and optimize the framework based on practical experience. This project was positioned as a developmental and formative evaluation research, rather than a strict controlled causal comparison [9].

Materials and methods

Study design

This study employed a mixed-methods design integrating literature research, expert interviews, framework construction and prospective cohort pilot evaluation. Due to fixed academic training arrangements, random grouping was not feasible. A sequential two-cohort design was adopted: the previous year cohort served as the historical reference, and the subsequent year cohort implemented the new assessment system, focusing on framework optimization and feasibility evaluation rather than causal inference.

Settings and participants

This study was conducted in the oral and maxillofacial surgery specialty clinic of the General Dentistry II at Peking University School and Hospital of Stomatology, a dedicated outpatient teaching unit specifically established for undergraduate dental student internships. This clinic operates as a standardized training site where patient selection, case allocation, and teaching protocols are aligned with educational objectives.

Two consecutive cohorts of final-year undergraduate dental students who completed the full OMFS outpatient internship were included. The reference cohort comprised students from August 2023 to July 2024 under the traditional assessment method. The development cohort comprised students from August 2024 to July 2025 under the new CBE assessment system. Exclusion criteria included incomplete internship training or failure to finish all assessment items. Eleven full-time teaching instructors were involved in teaching and questionnaire evaluation for each academic year respectively.

Development of the assessment system

Identification of clinical competency elements

On the basis of literature review on dental CBE competency frameworks, semi-structured interviews were conducted with five senior faculty members with more than 10 years of teaching experience. After consensus discussion, six core competency domains were confirmed: outpatient history-taking and doctor-patient communication, oral examination and basic outpatient procedures (including aseptic techniques and patient care), diagnosis and treatment of common OMFS outpatient conditions, prevention and management of common outpatient complications (including patient care and minimally invasive principles), medical record documentation, and teamwork and collaboration. Each competency domain corresponds to clear clinical internship requirements:

1. Outpatient history-taking and doctor-patient communication: the ability to collect complete medical history, communicate treatment risks and prognosis with patients, and resolve patient concerns appropriately.

2. Oral examination and basic outpatient procedures: standardized intraoral inspection, strict aseptic operation, and gentle perioperative patient protection during routine minor surgeries.

3. Diagnosis and treatment of common OMFS outpatient conditions: accurate differential diagnosis of prevalent lesions and formulation of individualized, evidence-based treatment plans.

4. Prevention and management of common outpatient complications: recognition of perioperative adverse reactions, minimally invasive operation concepts, and timely intervention to reduce surgical complications.

5. Medical record documentation: standardized, complete, accurate writing of outpatient medical records consistent with hospital and disciplinary specifications.

6. Teamwork and collaboration: effective communication with nursing staff, senior residents and attending surgeons during combined diagnosis and treatment.

Construction of the assessment blueprint

Based on these six competency domains, a multi-component assessment blueprint was designed. It comprised five assessment items: (1) an end-of-rotation theory examination covering diagnosis and treatment of common OMFS diseases, (2) an end-of-rotation integrated clinical operation assessment requiring students to independently manage a complete patient encounter (history-taking, communication, diagnosis and treatment planning, procedural execution, and medical record documentation), (3) phased operational skill assessments conducted weekly using standardized criteria aligned with China’s national dental licensure examination and standardized residency training assessments, (4) patient satisfaction and follow-up data on adverse events, and (5) daily performance and attendance evaluation. Each item was assigned an initial percentage weighting.

Scoring criteria

Each student was independently rated by three faculty members selected on a rotating schedule from a pool of eleven attending surgeons, who completed the scoring rubric without discussion or score exchange. Detailed scoring rubrics were developed for each assessment item. For the phased operational skill assessments, scoring standards were adopted directly from the national dental licensure examination. A clear 100-point scoring rubric was formulated for each assessment item with detailed deduction criteria.

Implementation and data collection

The reference cohort completed internships under the traditional assessment method, which included routine performance and attendance evaluation plus a final operational examination on extraction of an erupted maxillary third molar. The scoring standard for this single operation assessment directly referred to the tooth extraction scoring rubric of the national dental licensing examination. This historical evaluation lacked standardized multi-domain scoring tools covering the six core clinical competency domains defined in the present study. No systematic evaluation of doctor-patient communication, clinical diagnosis, medical record writing, complication management or teamwork performance was conducted in the prior cohort.

The development cohort implemented the full set of multi-dimensional CBE assessment throughout the internship. After the rotation, both students and instructors filled in anonymous structured questionnaires covering the system’s comprehensiveness, objectivity and ability to distinguish competency levels. The questionnaire items were drafted based on the six core competency domains identified via expert interviews, then subjected to internal pilot testing among 5 interns and 2 instructors before formal distribution. Three senior OMFS attending surgeons independently reviewed all items to evaluate content relevance, logical coherence and scoring rationality, with minor wording revisions implemented. This feedback tool was designed for internal teaching quality monitoring rather than a standardized psychometric scale; thus internal consistency testing (Cronbach’s α) and full construct validity analysis were not performed in this pilot work.

Patient satisfaction was collected via postoperative telephone follow-up for all patients treated by each intern. Due to the nature of the outpatient teaching clinic, each student was allocated a limited and variable number of independent single-session patient encounters during the rotation; the specific encounter count per student was not standardized and was not statistically analyzed as a separate variable in this pilot study. Feedback incompleteness resulted from patient refusal and occasional missed follow-up calls. Adverse events such as dry socket, residual root fragments and intraoperative anesthesia-related reactions were recorded and statistically compared between the two cohorts.

Data analysis

All statistical analyses were performed using SPSS (version 27.0; IBM Corp., Armonk, NY, USA). Baseline characteristics and demographic data were descriptively summarized. Continuous variables, including comprehensive competence scores and procedural volume, were presented as mean ± standard deviation and compared between cohorts using the independent-samples t-test. Categorical variables, including adverse event rates and questionnaire feedback proportions, were analyzed using the chi-square test. All statistical tests were two-tailed, with significance set at P < 0.05.

Results

General information

A total of 46 students in the development cohort and 48 in the reference cohort met the inclusion criteria. Valid questionnaires were obtained from 43 students (93.5%) and 47 students (97.9%), respectively. All 11 supervising instructors in each cohort completed the questionnaire. The mean final internship scores were 91.4 (development cohort) and 90.8 (reference cohort), both within the institution’s “good” grade band (≥ 85%). The total number of tooth extractions performed during the internship was 987 in the reference cohort and 943 in the development cohort. The corresponding mean number of tooth extractions per student was 20.6 (range 14–42) in the reference cohort and 20.5 (range 16–35) in the development cohort.

Student evaluations of the assessment system

Student ratings regarding the ability of the two systems to comprehensively reflect clinical competencies showed no significant inter-cohort difference (8.14 vs. 6.52, P = 0.086). Ratings for the ability to discriminate different levels of clinical competence were similar between the two systems (7.91 vs. 7.83, P = 0.924). Within the new system, mean scores for the theory examination and patient satisfaction components fell below 8. All components were perceived as fair except patient satisfaction. Although patient satisfaction received the lowest average score among all five components, the mean rating of 8.2 out of 10 still represented generally positive student feedback rather than dissatisfaction with this assessment module. All evaluation results are presented as mean values with corresponding 95% confidence intervals shown in Fig. 1.

Fig. 1.

Student evaluations of the new assessment system components: (a) mean objectivity ratings for each component; (b) perceived fairness of each component. Solid black circles represent mean ratings, horizontal whiskers indicate 95% confidence intervals

Teacher evaluations of the assessment system

Supervising instructors rated the new system significantly higher than the traditional system for comprehensiveness (9.64 vs. 6.28, P = 0.043). Ratings for discriminating different levels of competence also favored the new system (9.45 vs. 7.65), though the difference was not statistically significant (P = 0.314). All components of the new system received objectivity ratings above 9 from teachers. Consistent with student evaluations, patient satisfaction received the lowest fairness rating among all components. All evaluation results are presented as mean values with corresponding 95% confidence intervals shown in Fig. 2.

Fig. 2.

Teacher evaluations of the new assessment system components: (a) mean objectivity ratings for each component; (b) perceived fairness of each component. Solid black circles represent mean ratings, horizontal whiskers indicate 95% confidence intervals

Inter-rater reliability and system optimization

The overall ICC among the three independent raters was 0.78 (95% CI: 0.65–0.87), indicating good inter-rater consistency. According to questionnaire feedback, the weight ratios of theoretical examination and subjective patient satisfaction were appropriately reduced, and the indicator setting was optimized to increase objective items such as surgical complication incidence and medical normative performance (Table 1). Only the overall inter-rater reliability of the full assessment framework was calculated; item-level ICC for each individual assessment component was not analyzed in this pilot due to the limited panel of 11 raters.

Table 1.

Assessment blueprint for the development cohort

Assessment Item	Content	Competency Elements*	Weight (%)
End-of-rotation theory examination	Diagnosis and treatment of common OMFS diseases	③④	20
End-of-rotation integrated clinical operation assessment	Complete management of a clinical patient: history-taking, communication, diagnosis and treatment, procedural skills, and medical record documentation	①②③④⑤⑥	30
Phased operation skills assessment	Weekly skills assessment aligned with national licensure and residency examination standards: local anesthesia, tooth extraction and protective techniques, intraoral suturing, cross-chest bandaging, and submandibular gland bimanual palpation	②④	30
Patient satisfaction	Patient satisfaction and postoperative complication rates obtained through follow-up surveys	①④	5
Daily performance and attendance	Participation in daily internship activities, engagement, and attendance	①②③④⑤⑥	15

① Outpatient history-taking and doctor-patient communication

② Oral examination and outpatient basic procedures (aseptic techniques and patient care)

③ Diagnosis and treatment of common OMFS outpatient conditions

④ Prevention and management of common outpatient complications (patient care and minimally invasive principles)

⑤ Medical record documentation

⑥ Teamwork and collaboration skills

*Competency Elements

Adverse events

The overall incidence of adverse events was low in both cohorts, which limited the statistical power of intergroup comparative analysis. In the reference cohort, four cases of dry socket, three of retained root fragments, and two of intraoperative local anesthetic toxicity were recorded. In the development cohort, there was one retained root fragment, two cases of postoperative bleeding, and three of intraoperative local anesthetic toxicity; no dry socket occurred. Fisher’s exact test showed no statistically significant differences between cohorts for any adverse event (all P > 0.05, Table 2).

Table 2.

Comparison of adverse events between the reference and development cohorts

Adverse events	Reference cohort (987 extractions)	Development cohort (943 extractions)	P value*
Dry socket	4	0	0.125
Retained root fragments	3	1	0.625
Local anesthetic toxicity	2	3	0.684
Postoperative bleeding	0	2	0.245

*P values were calculated using Fisher’s exact test based on the total number of tooth extractions performed in each cohort

Discussion

This study developed and pilot-validated a competency-based assessment system for undergraduate OMFS outpatient internships. The system integrated six clinical competency domains into five assessment components. Supervising instructors rated the new system as more comprehensive than the traditional single-skill examination, and inter-rater reliability was good (ICC = 0.78). Although students gave numerically higher comprehensiveness scores to the new framework, this difference did not reach statistical significance, possibly due to limited sample size of returned student questionnaires. These findings suggest that a multi-component, competency-guided assessment framework is feasible and acceptable in a real undergraduate clinical teaching setting.

The present findings align with the broader shift toward competency-based assessment in health professions education. Traditional OMFS internship evaluation has relied heavily on procedure-count or single-skill examinations, which miss the communication, clinical reasoning, and documentation abilities emphasized in CBE frameworks [4–6]. Our study suggests that both students and teachers recognized the blueprint’s emphasis on these multidimensional abilities. The CBE framework constructed in this study fully covers the core competency requirements of OMFS outpatient practice, and the integrated clinical operation assessment simulates real clinical encounters, which is close to the concept of Entrustable Professional Activities (EPAs) advocated in current dental education [10, 11].

A major advantage of this study is the adoption of three independent raters from a fixed teacher panel, with an ICC of 0.78 confirming good scoring stability. It also proves that the unified scoring rubric formulated in reference to national training standards can effectively reduce individual subjective differences [12]. The rotating design also more closely reflects real-world clinical teaching, where students encounter multiple supervisors, and thus the reliability estimate carries ecological validity.

Patient satisfaction was incorporated into the system to reflect the patient-centered dimension of clinical competence, consistent with the broader trend toward patient-reported measures in healthcare quality assessment [13]. However, both students and teachers perceived patient satisfaction as the least objective component. Patient satisfaction is influenced by numerous factors beyond the student‘s control [14], including patients’ baseline expectations, communication style preferences, and the specific clinical context [15]. Notably, each student only completed a limited number of independent single-session outpatient consultations, so patient feedback cannot fully reflect students’ stable communication performance across diverse patient groups. Such one-off assessment conditions further weaken the fairness of patient satisfaction scoring. Based on this feedback, we revised the component weighting and scoring to emphasize objective indicators, such as medical errors or procedure-related complications, over subjective satisfaction ratings [16]. This adjustment is more in line with the practical needs of teaching assessment.

The adverse event data provide preliminary safety evidence for the new assessment framework. No statistically significant differences in complication rates were observed between cohorts, and adverse events were infrequent in both groups. This suggests that the new system did not compromise patient safety, an important consideration when implementing any educational change that involves direct student-patient contact. Adverse event records were only collected to help students reflect on their intraoperative aseptic technique and tissue protection awareness, acting merely as auxiliary references for overall teaching outcomes. The incidence of complications was not directly used to adjust or deduct formal evaluation scores. Future research will formulate refined criteria to build a reasonable linkage between different types of surgical complications and competency assessment scores.

Several limitations should be noted. This single-center sequential historical cohort design cannot eliminate time-varying confounders such as annual differences in case composition and faculty experience, thus restricting causal inference about the effectiveness of this assessment framework. We only draw preliminary conclusions on the feasibility and acceptability of the framework, and future parallel randomized controlled multi-center trials are needed to reduce such bias. In addition, the evaluation questionnaire only underwent content validity review by experts, without complete psychometric verification such as internal consistency analysis, which restricts the generalizability of relevant questionnaire feedback results. Full reliability and validity testing of the questionnaire will be conducted in subsequent multi-center validation research.

The absolute number of captured adverse events was small in both cohorts, which substantially reduced the statistical power of intergroup comparisons. Furthermore, collecting only common complications via routine telephone follow-up further limited the total event count. These data therefore provide only preliminary descriptive safety evidence and cannot support definitive conclusions about the association between the CBE framework and surgical complication risks. In addition, certain complications such as dry socket have multifactorial causes weakly correlated with students’ operative performance; thus complication records were used only to facilitate student self-reflection, not as deduction criteria for formal scores. Future work should establish refined grading standards to differentiate the linkage between distinct complication types and internship performance.

The number of participating instructors was small, which may affect the representativeness of teacher feedback. We only reported the overall ICC of the whole evaluation system without domain-specific reliability analysis. Future large-sample studies will calculate separate inter-rater consistency for each competency subdimension [17]. Rather than presenting a finalized tool, we offer a development process and an initial evidence base that other institutions can adapt.

Conclusions

This study developed and pilot-validated a feasible competency-based assessment system for undergraduate OMFS outpatient internships. The framework covers multi-dimensional clinical competency, has good inter-rater reliability and high recognition among teachers and students, and does not increase clinical adverse events. After iterative optimization based on practical feedback, it can provide a referable evaluation model for undergraduate dental internship teaching reform. Further multi-center studies are needed to verify its generalizability.

Authors’ contributions

YZ contributed to the study conception and design, data acquisition and analysis, and drafting of the manuscript. SY supervised the clinical teaching implementation, coordinated the assessment process, and critically revised the manuscript. WZ, NA, and CH provided overall supervision, contributed to the development of the competency framework, assisted in data interpretation, and reviewed the manuscript. All authors read and approved the final manuscript.

Funding

This study was supported by the Educational and Teaching Research Project of Peking University School of Stomatology (No.: 2024-ZC-05).

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to the inclusion of patient information and personal identity data of students and faculty, but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was reviewed and granted an exemption from full ethical review in accordance with the Declaration of Helsinki by the Institutional Ethics Committee of Peking University School and Hospital of Stomatology (Approval No. PKUSSIRB-2025MS116010). The research belonged to educational program evaluation using anonymized routine internship data and questionnaire information. All participating students and instructors provided verbal informed consent. Patient-related data were derived from routine follow-up without additional clinical intervention.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.