An 8-year retrospective survey of assessment in postgraduate dental training in complicated tooth extraction competency

Abstract

Background/purpose

Direct observation of procedural skills (DOPS) has been increasingly used in health education in recent years. This study evaluated the effect of education and trainees’ perception of assessment on the clinical skills of postgraduate dental trainees in complicated tooth extraction.

Materials and methods

This study was conducted as a retrospective survey among postgraduate dental trainees learning complicated tooth extraction in Taipei and Linkou Chang Gung Memorial Hospital from 2012 to 2019. Practical skills were assessed using DOPS by trainees and faculty members. Each clinical case included a complicated extraction of a permanent tooth.

Results

A total of 69 participants (26 men and 43 women, average age = 26.49 years, range = 24–34 years) were included in this study. Within the survey cohort, faculty assessments scored significantly higher than did trainees’ self-assessments in each complicated tooth extraction procedure, with no difference between both sexes. The higher-performing trainees tended to underrate their performance much more than did the lower-performing trainees. More than 40% of the trainees evaluated themselves as having “poor capability” in some invasive surgical procedures, even though their actual performance was not lower than that of those who evaluated themselves as having adequate or good capability.

Conclusion

Self-assessment skills should be developed with more practice and experience. We hope that these findings can guide the planning of faculty development programs for clinical instructors, particularly the new cohort of faculty who will succeed the rapidly retiring boomer generation in the next 10 years.

Introduction

Since the outbreak of severe acute respiratory syndrome (SARS) in Taiwan from May 5 to June 4 in 2003,¹ residency programs in the country have mostly ignored generalized medical and dental education. Students have long been perceived as disliking their school experiences; therefore, publications discussing student feedback have continually mentioned that change is necessary.² Recent graduates generally consider themselves unprepared for independent clinical practice despite having likely undergone adequate clinical training.^2,3 To resolve this problem in medical educational programs for postgraduates, the Taiwanese Department of Health (DOH) funded a nationwide project, “the Postgraduate Year Training Program” (PGY), in 2003. Moreover, the Taiwan Dental Association started a 2-year postgraduate training program in 2010. The training program implemented by the DOH comprises 2-year courses in operative dentistry, endodontics, prosthodontics, periodontics, oral and maxillofacial surgery, pediatric dentistry, and orthodontics. In addition, a 36-h basic curriculum, including medical ethics and related laws, evidence-based medicine, infection control, and dental care quality, was also enforced into this training program. The principal objective of the 2-year course is to improve postgraduate dental trainees’ psychomotor abilities by using a standard technique to render these abilities more germane to clinical practice. It is hoped that trainees will be able to achieve their desired goals due to their enhanced capacity to self-evaluate their psychomotor abilities. These goals include the abilities to perform lifelong learning and provide excellent patient care.⁴

There are great differences in the content, concepts and tools of the curriculum in dental education around the world. Traditionally, in Taiwan, trainees learn the tooth extraction technique during their internship in a hospital in the sixth year of dental school, and no phantom-head exercises on the tooth extraction training are conducted in a patient simulation laboratory at dental schools. In this environment, students tend to become passive learners without seeking refinement in clinical practice.⁵ Most of the PGY trainees have graduated from dental school within 1 or 2 years and are still somewhat unfamiliar with clinical skills and procedures. The clinical curriculum of the PGY training course generally comprises theory presentations, course manuals, and demonstrations of clinical procedures. During the course, clinical instructors evaluate pre-clinical assignments and provide verbal feedback or active instructions.

Dental education should comprise theoretical and practical learning for postgraduate dental trainees to develop the ability to treat patients clinically. For dental residents, the practical courses in the first 2 years of postgraduate study represent a novel experience in their education, and they must learn to self-assess.⁵ Self-assessment is a basic and indispensable skill for dental professionals to continually improve their competence during their careers.⁴

It is frequently the case that an individual's level of self-confidence is not justified by their ability level. Considerable academic effort has been exerted to understand why this is the case in diverse fields, including clinical, cognitive, personality, organizational, and social psychology.⁶ Education, practice, and experience are necessary to accurately and rationally undertake assessment.^7,8 Self-assessment ability improves and become more accurate with practice.

Numerous scholars assert that dental curricula require both clinical relevance and a scientific basis to enhance social responsibility.⁹ Scott opines that an ideal dental training program should produce dentists who are principled and who can perform general dentistry services in accordance with the reasonable expectations of the public that they work for.¹⁰ Additionally, they must commit to professional and educational development over their life-long careers. Manogue lists other desirable features of newly qualified dentists and determines that following qualification, trainees have to be both life-long researchers and competent practitioners.¹¹ Because dental education guides dental students through the transition into oral physicians, it should teach students to have a holistic vision and offer courses relevant to oral health, irrespective of any medical school affiliation.¹²

Direct observation of procedural skills (DOPS) has been increasingly used in health education in recent years.^13,14 In the oral surgery training course, DOPS was used to evaluate the preclinical and clinical skills of trainees.

This study aimed to evaluate trainees’ self-assessment of complicated tooth extraction skills and compared it with faculty assessment. In addition, to the best of our knowledge, this type of comparative investigation in postgraduate dental education has not been previously conducted in Taiwan.

Materials and methods

This study was approved by the Institutional Review Board (the “IRB”) of Chang Gung Medical Foundation on 2019/05/22 (Project No: 201801097B0C601) All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

This study was conducted as a retrospective survey among postgraduate dental trainees learning complicated tooth extraction in Taipei and Linkou Chang Gung Memorial Hospital from 2012 to 2019. Every trainee received a 2-month mandatory course in the first year in the department of oral and maxillofacial surgery. The practical skill assessment was performed using DOPS in accordance with the listed requirements of the 2-year postgraduate dental training program sponsored by the DOH in Taiwan since 2010. Each clinical case included a complicated extraction of a permanent tooth, with the involvement of advanced surgical procedures (e.g., flap reflection, removal of the covering bone, tooth sectioning, removal of the tooth fragment, and wound suturing).

Two DOPS checklists were modified for use in self-assessment and clinical faculty evaluation (Fig. 1). To set the qualification and standardization of grading outcomes, all of the 7 designated faculty members joined the cultivation camp held by the DOH for the clinical instructors of postgraduate dental training programs. All certified clinical instructors were required to meet the education and experience requirements established by the DOH. The faculty member to trainee ratio for supervision of complicated tooth extraction was 1:1. Trainees were also formally instructed on grading for each step of extraction procedures. Before the procedure started, trainees were asked to complete a form to assess their expected performance. Faculty members evaluated each item during each stage of tooth extraction. Following evaluations were recorded: 1) examination score (points awarded by the faculty) and 2) trainee self-assessment (self-assessed points).

Figure 1.

Modified clinical direct observation of procedural skills form used in oral and maxillofacial surgery. Trainees and faculty members were required to complete all 13 items of the surgical procedures at each encounter.

At the end of the 2-month training period in the department of oral and maxillofacial surgery in the first year, charting forms were collected, and these data were manually digitized into a spreadsheet. The relevance of all data was calculated and statistically analyzed using descriptive and inferential statistics. To analyze the differences between faculty assessments of trainees and trainees’ own self-assessments, paired t-testing was conducted; for identifying significant differences between genders, independent-samples t-testing was employed. A P value of <0.05 was considered statistically significant.

By using mean faculty assessments as a criterion for comparing trainees’ mean self-assessments, self-assessment accuracy was determined. Next, the trainee-faculty (T-F) gap was calculated as the difference between mean self-assessments and mean faculty assessments, with a T-F gap score of 0.0 indicating no difference between the trainee and faculty assessments and a positive gap score indicating more favorable trainee self-assessment compared with faculty assessment. However, because positive and negative gap scores may offset each other, leading the mean gap to approach 0.0, the absolute T-F gap scores were also calculated. When calculating absolute T-F gap scores, no positive or negative scores are considered; rather, only absolute differences, which can be considered the self-assessment accuracy, are reported.

The scores were graded at three levels (Fig. 1): poor capability (score: 1–3), adequate capability (score: 4–6), and good capability (score: 7–9). The relevance between each level of self-assessment and faculty assessment was also compared.

Results

A total of 69 participants (26 men and 43 women, average age = 26.49 years, range = 24–34 years) completed the 2-month training course in the first year in the department of oral and maxillofacial surgery from 2012 to 2019. Approximately 62.32% of the participants were women. The female dentists among the survey cohort tended to be younger than the male dentists, with respective mean ages of 25.84 and 27.35 years. Mean scores and standard deviations for each group (trainees' self-assessments and faculty assessments) are presented in Table 1. Within the survey cohort, the faculty assessments scored significantly higher than did the trainees’ self-assessments in each procedure. For absolute T-F gap scores, overestimates and underestimates did not average out as they did for the T-F gap scores; thus, absolute T-F gap scores were higher than their corresponding T-F gap scores. Here, negative gap scores indicated lower trainee self-assessment compared with faculty assessment in the exercise.

Table 1.

Descriptive statistical measures for variables and t-tests comparing mean differences between scores of trainee self-assessments and faculty assessments (N = 69). The statistical significance of the difference between trainee self-assessments and faculty assessments in all 13 DOPS items is also presented. Differences between faculty score and self-assessment (T-F gap) are statistically significant in all items.

	Item	Trainee self-assessments (Mean ± SD)	Faculty assessments (Mean ± SD)	P value of difference	Mean T-F Gap	Mean absolute T-F Gap
1	Pre-surgery explanation	4.32 ± 1.30	6.41 ± 1.29	<0.0001∗	−2.09∗	2.26∗
2	Pre-surgery preparation	4.30 ± 1.34	6.42 ± 1.18	<0.0001∗	−2.13∗	2.19∗
3	Local anesthesia/Analgesic management	4.36 ± 1.40	6.45 ± 1.18	<0.0001∗	−2.09∗	2.20∗
4	Instrumentation	4.23 ± 1.34	6.46 ± 1.11	<0.0001∗	−2.23∗	2.32∗
5	Protection of neighboring tissue	4.10 ± 1.35	6.32 ± 1.29	<0.0001∗	−2.22∗	2.36∗
6	Flap reflection	3.76 ± 1.26	6.36 ± 1.13	<0.0001∗	−2.6∗	2.6∗
7	Removal of covering bone	3.70 ± 1.37	6.34 ± 0.98	<0.0001∗	−2.64∗	2.64∗
8	Tooth sectioning	3.68 ± 1.28	6.18 ± 1.12	<0.0001∗	−2.5∗	2.54∗
9	Removal of tooth fragment	3.96 ± 1.34	6.33 ± 1.30	<0.0001∗	−2.38∗	2.46∗
10	Wound suturing	3.92 ± 1.35	6.34 ± 1.06	<0.0001∗	−2.42∗	2.42∗
11	Gauze packing	4.55 ± 1.49	6.52 ± 1.20	<0.0001∗	−1.97∗	2.12∗
12	Infection control	4.48 ± 1.50	6.42 ± 1.24	<0.0001∗	−1.94∗	2.14∗
13	Post-operative wound management	4.55 ± 1.45	6.40 ± 1.30	<0.0001∗	−1.86∗	2.09∗

∗Statistically significant (p < 0.05).

Table 2 presents the correlation between the aforementioned scores and both sexes of the trainees. The faculty assessments scored significantly higher than did the trainees’ self-assessments in each procedure in both sexes. However, no significant difference was observed between both sexes in each procedure.

Table 2.

Descriptive statistical measures presented the correlation between the aforementioned scores and both sexes of the trainees. No significant difference was observed between both sexes in each procedure.

	Item	Trainee self-assessments (male) (Mean ± SD)	Faculty assessments (male) (Mean ± SD)	P value of difference (male)	Trainee self-assessments (female) (Mean ± SD)	Faculty assessments (female) (Mean ± SD)	P value of difference (female)	P value of difference between male and female (self-assessments)	P value of difference between male and female (faculty assessments)
1	Pre-surgery explanation	4.27 ± 1.19	6.46 ± 1.21	<0.0001∗	4.35 ± 1.40	6.37 ± 1.35	<0.0001∗	0.401	0.388
2	Pre-surgery preparation	4.31 ± 1.29	6.54 ± 1.10	<0.0001∗	4.30 ± 1.39	6.35 ± 1.23	<0.0001∗	0.494	0.255
3	Local anesthesia/Analgesic management	4.54 ± 1.36	6.58 ± 1.06	<0.0001∗	4.26 ± 1.43	6.37 ± 1.25	<0.0001∗	0.208	0.236
4	Instrumentation	4.23 ± 1.31	6.54 ± 1.03	<0.0001∗	4.24 ± 1.39	5.98 ± 0.27	<0.0001∗	0.491	0.006
5	Protection of neighboring tissue	4.19 ± 1.33	6.42 ± 1.17	<0.0001∗	4.05 ± 1.38	6.26 ± 1.36	<0.0001∗	0.332	0.296
6	Flap reflection	3.79 ± 1.25	6.26 ± 1.09	<0.0001∗	3.74 ± 1.29	6.19 ± 1.09	<0.0001∗	0.283	0.386
7	Removal of covering bone	3.74 ± 1.16	6.32 ± 1.10	<0.0001∗	3.68 ± 1.50	6.35 ± 0.90	<0.0001∗	0.342	0.408
8	Tooth sectioning	3.79 ± 1.16	6.37 ± 1.06	<0.0001∗	3.61 ± 1.40	6.29 ± 0.96	<0.0001∗	0.280	0.275
9	Removal of tooth fragment	4.04 ± 1.15	6.38 ± 1.17	<0.0001∗	3.93 ± 1.47	6.5 ± 1.39	<0.0001∗	0.367	0.435
10	Wound suturing	3.95 ± 1.27	6.37 ± 1.06	<0.0001∗	3.90 ± 1.42	6.40 ± 1.06	<0.0001∗	0.351	0.365
11	Gauze packing	4.73 ± 1.66	6.54 ± 1.03	<0.0001∗	4.47 ± 1.39	6.51 ± 1.30	<0.0001∗	0.249	0.462
12	Infection control	4.69 ± 1.67	6.46 ± 1.14	0.000106∗	4.40 ± 1.38	6.40 ± 1.31	<0.0001∗	0.225	0.413
13	Post-operative wound management	4.88 ± 1.66	6.5 ± 1.14	0.000273∗	4.40 ± 1.28	6.35 ± 1.40	<0.0001∗	0.124	0.313

∗Statistically significant (p < 0.05).

Because good self-evaluated trainees were among the minority in all training items (<6%), trainees were divided into group 1 (poor self-evaluation) and group 2 (adequate or good self-evaluation) for comparison. More than 40% of the trainees evaluated themselves as having “poor capability” in some invasive surgical procedures (flap reflection, removal of the covering bone, and tooth sectioning) and also obtained lower average self-evaluation scores. This finding indicated that many trainees were unconfident in these procedures. However, the actual performance of the trainees with poor self-evaluation was not lower than that of the trainees with adequate or good self-evaluation. Although no statistical significance was observed in faculty assessments between the two groups, average scores were slightly higher in the poor self-evaluation trainee group. On the other hand, some steps in the procedure (e.g., gauze packing and postoperative wound management) had a lower percentage of the trainees evaluating themselves as having poor capability, indicating higher average self-evaluation scores (Fig. 2).

Figure 2.

Trainees were divided into group 1 (poor self-evaluated) and group 2 (adequate or good self-evaluated) for comparison. The number in the grid represents the ratio of the total number of trainees in each group. Graphical representation of DOPS marks for assessment scores (mean value), showing that the actual performance of poor self-evaluated trainees was not lower than that of adequate or good self-evaluated trainees. No statistical significance in faculty assessments was observed between the two groups in each procedure.

Discussion

Teaching is considered inseparable from learning. Teaching generally involves traditional lectures, which provide students with information while allowing few chances to interact. The optimal dental educational facility would empower students to obtain the required clinical, interpersonal, and theoretical competencies and provide them with an insight into the “clinical experiences” that they will likely encounter following graduation. Trainees’ physical and psychological health and social and mental well-being must be maintained and enhanced, and their moral principles and cultural values should be fortified. Personal and occupational development should be provided in all higher education environments. Furthermore, extraordinary opportunities to volunteer, collaborate, and perform community service should be provided to develop desirable values. In addition, to avoid the personal and professional isolation that frequently affects dental practitioners, dental trainees should learn to collaborate and communicate effectively with health professionals as well as their colleagues.²

The PGY dental residents in this study revealed they can still greatly improve their accuracy of self-assessment. According to a review article in 2016,¹⁵ some studies by other scholars have reported similar conclusions. The reason that the trainees were prone to perform an incorrect evaluation warrants investigation. Low self-assessment may be related to self-confidence of the trainees.¹⁶ Notably, overall, the trainees tended to provide lower self-assessment scores than faculty assessment in our present study. However, this result is just the opposite of the results of other studies.^15,17 Experimental results were probably affected by cultural differences. However, because surgical extraction-related research remains rare, this potential effect of cultural differences on the current experiment design cannot be determined. Surgical extractions are one of the more advanced and aggressive dental treatments. Students relatively lacked relevant experience in dental school;^{18, 19, 20} therefore, they were less confident about it.^18,21,22 Trainees' preclinical performance during their undergraduate years may be correlated with self-assessment accuracy because of many reasons. With practice, trainees’ self-assessment accuracy may improve alongside their hand skills.^5,15

At all stages of dental education, the confidence of students is crucial. Self-confidence strongly influences graduate education selection.²³ Because clinical exposure helps build confidence,^{24, 25, 26} the disparate reports of confidence in the literature may reflect differences in experience working with patients. Self-assessment is often an external expression of self-confidence.^{6, 7, 8,15} Assisting the trainees understand their shortcomings and enhancing their self-confidence will indirectly enhance their self-assessment ability. Because ‘feedback’ is a major component of DOPS, the trainees can understand their shortcomings in surgical procedures immediately after teachers' timely feedback. This also helps improve self-confidence of the trainees.^13,27

Our findings could help the dental trainees’ scope for improving their self-assessment accuracy based on the statistical significance between the trainees’ self-assessments and faculty assessments in each procedure. In addition, high performers tended to underestimate their expected performance much more than low performers. Personality traits may explain this finding. High-performance students may perform well because they are highly critical and hold themselves to high standards, whereas low-performance students may be lackadaisical and not focus painstakingly on minutiae. Successful students have been discovered to assess themselves in comparison with their past performance or their potential and thus underestimate themselves.^{5,28, 29, 30} Such results are similar to that obtained in the present study.

The self-assessment ability correlated with clinical performance; as trainees perform with more confidence, their self-assessment accuracy improves.^5,15,17 Early operating experience and clinical exposure to some surgical steps help foster trainees’ clinical confidence.^4,17,19,25 In this study, some steps (e.g., gauze packing and postoperative wound management, etc.) had a lower percentage of the trainees evaluating themselves as having “poor capability”, indicating higher average self-evaluation scores. These steps were usually considered non-invasive surgical procedures, and trainees mostly had some experiences of early clinical exposure during their clerkship and internship in dental school. On the other hand, more than 40% of the trainees evaluated themselves as having “poor capability” in some invasive surgical procedures, including flap reflection, removal of the covering bone and tooth sectioning. New graduates lack confidence in these surgical procedures. Further developments in the areas of clinical supervision and teaching in these surgical procedures were considered. Oral surgery has some of the most serious complications in dentistry; this strains faculty members and trainees involved. However, some dental programs struggle to provide students with an adequate number of surgical extraction cases. Such inadequacy can hinder the output of graduates confident in their dental extraction abilities.

The application of the DOPS method appeared to help to improve clinical skills among the trainees, and it is recommended that dental education centers employ the DOPS method to assess trainees' clinical skills in conjunction with other methods to help promote their learning.^13,14 Traditionally, phantom head mannequins or pig's heads was utilized for pre-clinical training.²⁴ With advances in technology, three-dimensional (3D) and virtual reality (VR) preclinical training can be the next-generation training methods. All these teaching aids will help dental trainees augment and enrich their learning experience, thereby developing self-confidence in clinical practice.³¹

As the requirement for trainees’ self-assessment becomes more common, trainees will become more skilled in assessing their performance. However, our challenge as dental educators is not to apply self-assessment as an endpoint but rather to encourage dental trainees to be self-directed in the self-assessment process so that they can evaluate their knowledge against valid measures within their area of expertise.

Other crucial concerns that must be emphasized in future studies are as follows: (1) the necessity of assessment and feedback in DOPS training programs; (2) the clinical relevance and importance of communication and empathetic abilities and dental training; and (3) requesting that hospitals allocate more equipment and human and financial resources to dental training.

This study is one of the few studies to evaluate the competency of complicated tooth extraction using DOPS. A limitation of this study may be the low number of the trainees who participated in the dental PGY courses during 2012–2019. In addition, complete standardization is difficult because complicated tooth extraction is dependent on a number of unknowns, such as the cooperation and anxiety levels and status of the patient, experience of the trainee, local anatomical parameters, and labor factors.^18,19,24 Competency training must ensure trainees can perform any operation required in dentist–patient interaction as well as educate them holistically. In future studies, larger samples should be evaluated, and a prospective cohort design should be applied for reducing error and bias.

We will continue to contribute to improve the 2-year postgraduate dental training program, and we believe that this project will drastically change the behavior of trainees and substantially improve the quality of dental care in Taiwan.

In conclusion, within the survey cohort, these postgraduate dental trainees had room for improving their self-assessment accuracy in complicated tooth extractions. Furthermore, this survey highlighted that more than 40% of the trainees were unconfident with some invasive surgical procedures, including flap reflection, covering bone removal, and tooth sectioning.

Self-assessment skills should be developed with more practice and experience. We hope these findings can guide the planning of faculty development programmes for clinical instructors, particularly the new cohort of faculty who will succeed the rapidly retiring boomer generation in the next 10 years.

Declaration of competing interest

The authors have no conflicts of interest relevant to this article.