Rating criteria to evaluate student performance in digital wax-up training using multi-purpose software

Abstract

PURPOSE

The aim of this study was to introduce rating criteria to evaluate student performance in a newly developed, digital wax-up preclinical program for computer-aided design (CAD) of full-coverage crowns and preliminarily investigate the reliability and internal consistency of the rating system.

MATERIALS AND METHODS

This study, conducted in 2017, enrolled 47 fifth-year dental students of Okayama University Dental School. Digital wax-up training included a fundamental practice using computer graphics (CG), multipurpose CAD software programs, and an advanced practice to execute a digital wax-up of the right mandibular second molar (#47). Each student’s digital wax-up work (stereolithography data) was evaluated by two instructors using seven qualitative criteria. The total qualitative score (0-90) of the criteria was calculated. The total volumetric discrepancy between each student’s digital wax-up work and a reference prepared by an instructor was automatically measured by the CAD software. The inter-rater reliability of each criterion was analyzed using a weighted kappa index. The relationship between the total volume discrepancy and the total qualitative score was analyzed using Spearman’s correlation.

RESULTS

The weighted kappa values for the seven qualitative criteria ranged from 0.62 - 0.93. The total qualitative score and the total volumetric discrepancy were negatively correlated (ρ = -0.27, P = .09, respectively); however, this was not statistically significant.

CONCLUSION

The established qualitative criteria to evaluate students’ work showed sufficiently high inter-rater reliability; however, the digitally measured volumetric discrepancy could not sufficiently predict the total qualitative score.

INTRODUCTION

Remarkable developments in digital technology have increased the popularity of dental prostheses fabricated using computer-aided design-computer-aided manufacturing (CAD-CAM) technology in clinical settings.¹,²,³,⁴,⁵ Accordingly, there is an increased need to provide technical skill training using CAD-CAM technology in undergraduate preclinical practice in dental schools.¹,⁴,⁶,⁷ However, overcoming the high costs of preparing and maintaining CAD-CAM systems and CAD software programs is a major challenge since hands-on training exclusively uses commercially available CAD-CAM systems for the fabrication of crown and bridge prostheses.¹ Furthermore, a validated and reliable evaluation system is needed to assess students’ achievements in undergraduate hands-on training. In conventional wax-up training for the fabrication of crown-bridge restorations or tooth sculpting training with plaster, student products are only visually evaluated by instructors.⁸,⁹ Evaluation bias and variability in grade evaluation still exist despite sufficient calibration of the instructors.¹⁰,¹¹,¹²,¹³,¹⁴,¹⁵,¹⁶ Therefore, recent studies have indicated the necessity of an objective evaluation system based on digital technology.¹⁰,¹⁵,¹⁷,¹⁸,¹⁹,²⁰

Low-cost multipurpose CAD software was utilized for several years in our institution to train students in designing dental prostheses. As this software is inexpensive and easily available online, students can install the software onto their computers. Moreover, the software is not restricted to designing dental prostheses; therefore, it can be used for three-dimensional (3D) design and assessment of certain digital products. These advantages led us to develop a new 3D digital designing education system for crown restorations (a digital wax-up training system) capable of evaluating students’ works.

The 3D volumetric difference between the students’ work and a reference prepared by an instructor as an evaluation tool was utilized.²¹ This evaluation method aids in depicting the convex and concave portions on the 3D surface data derived from the reference in the 3D cyberspace. Kwon et al. ¹⁵ reported that the 3D volumetric difference comparing the digitally scanned surface data of the wax-up product to the reference data showed high reliability. However, Garrett et al. ¹⁹ reported that estimations based on conventional visual evaluation, which is crucial in a clinical setting, were not significantly correlated to estimations based on the digital quantitative evaluation. This incongruence was partly attributed to lower reliability in the conventional visual evaluation criteria and partly to the lack of essential evaluation items relevant to clinical significance in the digital quantitative evaluation. Therefore, a qualitative evaluation system was developed to evaluate students’ digital wax-up performance with essential tools relevant to clinical significance with improved reliability, using the multidirectional visibility of the digital quantitative evaluation system. To the best of our knowledge, no previous study has successfully evaluated the reliability of qualitative morphological data of digital wax-up work using CAD software.

This study aimed to introduce rating criteria to evaluate student performance in a newly developed preclinical hands-on student training protocol of designing crown restorations using multi-purpose CAD and computer graphics (CG) software programs and to perform a preliminary estimation of the reliability and internal consistency of the protocol’s criteria for evaluating students’ designing performance. The inter-rater reliability (reproducibility) of the rating criteria proposed in this study was evaluated according to a reliability-estimating standard by Landis and Koch.²² The null hypothesis tested in this study was that there would be no statistical correlation between the total volumetric discrepancy between each student’s work and an instructor’s reference and the total qualitative score of the rating criteria proposed.

MATERIALS AND METHODS

This study was approved by the institutional review board for clinical research at Okayama University (approval no. K2007-031). All students provided informed consent for the publication of this study.

Forty-seven students in the fifth year of Okayama University Dental School participated in this newly developed digital hands-on training in 2017. All students installed the multipurpose CAD (Rhinoceros 3D; Robert McNeel & Associates, Seattle, WA, USA) and free CG (Sculptris; Pixologic, Los Angeles, CA, USA) software programs onto their laptop computers.

The general instructional objective of this training system was to understand how and what type of digital commands are performed during the digital wax-up procedure to fabricate a full-coverage crown restoration to develop and utilize new digital workflows in the upcoming “general format and multipurpose” era. The specific behavioral objectives were: to obtain basic competence in operating digital commands in CG and CAD software programs, to create a practical training protocol for the designing of a full-coverage crown restoration using these digital commands, and to fabricate the restoration using CAD-CAM technology.

The preclinical hands-on training consisted of seven three-hour long sessions titled: fundamental practice using a CG software program (volumes 1 and 2), fundamental practice using a multipurpose CAD software program, advanced practice of digital wax-up of the right second molar (#47) using CAD and CG software programs (volumes 1, 2, and 3), and try-in and adjustment of the prefabricated CAD-CAM crown.

The fundamental practice using a CG software program sessions (volumes 1 and 2) included information regarding import and export of the 3D data (.obj as a Wavefront OBJ format) and moving and orienting the object (rotate, zoom in, and zoom out). Three practical cases were prepared for students to enable recovering the form of the artificially deformed #47 without occlusal surface undulations and with an ill-fitted marginal line.

The fundamental practice using multipurpose CAD software program session included information regarding how to rotate, move, split, and mirror the entire 3D mandibular dentition data. In this practice, the methods for importing and exporting the 3D data were also included (.obj and .stl in stereolithography [STL] format).

The advanced practice of digital wax-up of #47 using CAD and CG software programs sessions (volumes 1, 2, and 3) utilized a set of original 3D data of the whole maxilla and mandible. An instructor made an oral impression of a clinical case and created stone casts of the maxilla and mandible. Abutment tooth preparation of #47 for a full-coverage composite resin CAD-CAM crown was performed in the full mandible plaster model by the chief instructor. Digital scanning of each model in its occluded position in the centric occlusion was performed using a 3D scanner (3D scanner D810; 3Shape, Copenhagen, Denmark). Subsequently, the digital data encoding the 3D morphology and position of the entire maxilla and mandible were restored as STL data (.stl). The abutment tooth 3D data (.stl) were also prepared. The digital data encoding the 3D morphology and position of the maxilla and mandible (main 3D data) were imported into the CAD software. The coronal aspect of the opposite corresponding tooth (left second molar #37) was cut and split from the 3D data of the whole mandible. The cut object was mirrored and pasted onto the missing occlusal space of the abutment tooth #47. The position of the object was optimized in terms of the cusp and marginal ridge positions (Fig. 1A). To modify the occlusal morphology, the file format of the coronal part of #47 was converted from STL (.stl) to OBJ data (.obj) by the CAD software and subsequently transferred to the CG software. In the CG software, the occlusal surface morphology of #47 was designed precisely, paying close attention to the occlusal contact (Fig. 1B). The occlusal surface data (.obj) were converted to STL data and transferred back into the CAD software at the pre-set location. The margin line in the 3D data of #47 was drawn manually by each student using the 3D data from the abutment tooth (.stl). The obtained margin line data were then merged with the main 3D data (.stl).

Fig. 1. Advanced practice of digital wax-up of the right second molar (#47) using the computer-aided design (CAD) and computer graphics (CG) software programs. (A) Preparing the prototype of #47 crown morphology using the coronal part of #37 on the CAD software. (B) Crown modeling of occlusal morphology on the CG software. (C) Steps from settings of the margin to adjusting the axial crown contour and finished #47 crown.

The surface data of the abutment were digitally relieved to allow appropriate cement space in the main 3D data (.stl) to build the inner surface of the restoration. The occlusal surface 3D data (.stl) was integrated into the margin line with an axial wall surface to build the outer surface. The inner and outer surfaces were automatically converged at the margin line of the abutment tooth. Lastly, the line angle, subgingival contour, gingival contour crest, and proximal contact shape were optimized according to the morphology of the opposite corresponding tooth (#37) (Fig. 1C). The integrated inner and outer surface data of each student were exported to STL data to fabricate a composite resin crown based on the student’s design.

Each composite resin crown was fabricated in a dental technician laboratory from a polymethyl methacrylate disk (RESIN DISK; Yamahachi Dental MFG, Co., Gamagori, Japan), according to the digital wax-up designed by the student. In try-in and adjustment session, the students adjusted the occlusal and proximal contacts and, if necessary, the inner surface. The surface of the crown was then polished by the student.

The instructors developed a new set of qualitative assessment criteria specialized for the students’ work shown on a liquid crystal display, according to clinical significance (Table 1). The assessment criteria were classified according to the clinical significance as follows: crown margin integrity (Fig. 2), occlusal contact (Fig. 3), proximal contact, cusp position and morphology, axial crown contour, marginal ridge position, and accessory ridge and groove.

Table 1. Qualitative and objective evaluation criteria of the digital wax-up training protocol.

Evaluation items	Score					Remarks
	20	15	10	5	0
Crown margin integrity	All points are on the finish line	One point is not on the finish line	Two points are not on the finish line	Three points are not on the finish line	More than four points are not on the finish line	Evaluated using the 20x enlarged view
Occlusal contacts	There are two occlusal contacts around the marginal ridge and distal fossa of #47, and both satisfy B+C contact	There are two occlusal contacts around the marginal ridge and distal fossa of #47, and one satisfies B+C contact	There is only one occlusal contact around the marginal ridge or distal fossa	/	Contact points ar greatly entered to the opposing teeth or hardly contact to opposite teeth	B contact: between the inner inclination of the #17 lingual cusp and the inner inclination of the #47 buccal cusp C contact: between the outer inclination of the #17 lingual cusp and the inner inclination of the #47 lingual cusp Evaluated using the 8x enlarged occlusal view
Proximal contacts	/	Proximal contact is a smooth convex surface, in the appropriate position, and the appropriate shape and size	Proximal contact is a smooth convex surface but the position is not appropriate, or axial contact is a smooth convex surface but the size is not appropriate	Proximal contact is a smooth convex surface but the position or the size is not appropriate, or axial contact is not a smooth convex surface and the position or the size is not appropriate	Proximal contact is greatly entering axial teeth or hardly contact to it	Evaluated using the 5x enlarged occlusal view. Appropriate position: one-third occlusal side and one-third buccal side. Appropriate shape and size: oval of approximately 1 x 2 mm)
Cusp positions and morphologies	/	/	Both buccal cusps and lingual cusps are continuously in line with the cusps of the axial teeth	Either buccal cusps or lingual cusps are not continuously in line with the cusps of the axial teeth	Neither buccal cusps nor lingual cusps are continuously in line with the cusps of the axial teeth	Evaluated using the 5x enlarged occlusal view.
Axial contour	/	/	Both appropriate	Either is not appropriate	Neither is not appropriate	Evaluated using the 2x enlarged occlusal view. Buccal and lingual surface contour was evaluated from the distal view. Mesial and distal surface contour was evaluated from the buccal view.
Marginal ridge position	/	/	Mesial marginal ridge height of #47 is mostly the same as the distal marginal ridge height of #46	/	Mesial marginal ridge height of #47 is not the same as the distal marginal ridge height of #46	Evaluated using the 5x enlarged occlusal view.
Accessory ridge and groove	/	/	/	Secondary ridge and groove are added	Secondary ridge and groove are not added	Evaluated using the 5x enlarged occlusal view.

Fig. 2. Assessment of crown-margin integrity. (A) Score of twenty (all points are on the finish line) and (B) score of zero (more than four points are out of the finish line).

Fig. 3. Assessment of occlusal contacts. (A) Score of twenty was awarded when two occlusal contacts could be observed around both the marginal ridge (arrow) and the distal fossa (arrowheads) of the right second molar (#47), satisfying at least one B contact (between the inner inclination of the #17 lingual cusp and the inner inclination of the #47 buccal cusp) and at least one C contact (between the outer inclination of the #17 lingual cusp and the inner inclination of the #47 lingual cusp). (B) Score of zero was awarded when the contact points intruded substantially into the opposing teeth.

Each finished STL datum that students designed was used for their work evaluation. Two instructors separately evaluated the digital work on a high-resolution liquid crystal display monitor (1366 × 768) using the qualitative assessment criteria with the same magnification set for each evaluation item (Table 1). The evaluation time for each criterion for a single student’s work was less than one minute. The inter-rater consistency of the two assessment scores for each criterion was calculated to investigate the inter-rater reliability of the qualitative assessment criteria. The average of the two instructors’ assessment scores was used as the final score of each assessment criterion. The total score of all assessment criteria for a student’s digital work was calculated as the sum of the total qualitative score of their digital wax-up work.

The total 3D volume of each student’s digital wax-up work was digitally measured and compared to that of the instructor’s reference restoration using commercially available CAD software (Power Shape; Autodesk, San Rafael, CA, USA) to quantitatively evaluate the students’ works. STL data that students and the instructor designed were used for the evaluation. Specifically, the total volumetric discrepancy between each student’s work and the instructor’s reference was quantified as the total volumetric difference (mm³), including the concave and convex volumes relative to the surface of the reference.²¹ The relationship between the total volumetric discrepancy and the total qualitative score was analyzed.

All students were requested to fill out a self-administered questionnaire immediately after the completion of the digital wax-up training. The questionnaire consisted of eight items, including the evaluation of achievement regarding specific behavioral objectives and the student satisfaction levels in this educational practice. A five-point Likert scale was utilized for this evaluation.

For the statistical analysis, the inter-rater reliability of the qualitative assessment criteria was analyzed using a weighted kappa index. The achievement rate of each qualitative criterion was calculated using the following formula: score achieved by student/the highest score in each criterion × 100 (%). The relationship between the total volumetric discrepancy (mm³) and the total qualitative score of the students’ digital wax-up products was analyzed using Spearman’s correlation.

RESULTS

All students successfully completed the training series with their personal laptop computers and emailed their digital wax-up work to the chief instructor. The final scores and achievement rates (%) of the qualitative assessment criteria and their weighted kappa values are shown in Table 2. The mean achievement rates of the crown margin integrity, accessory ridge, and groove criteria were > 80%; however, the mean rates of the occlusal contact and axial crown contour criteria were < 50%. The mean total achievement rate of the students was 59.4% ± 17.5% (highest, 88.9%; lowest, 33.3%).

Table 2. Final score, achievement rate, and weighted kappa value of each evaluation item (n = 47).

Evaluation items (allocation)	Final score (mean ± SD)	Achievement rate (mean ± SD)	Weighted kappa value
Crown margin integrity (20)	16.5 ± 4.6	82.7 ± 23.0	0.89
Occlusal contacts (20)	9.5 ± 5.5	47.6 ± 27.5	0.93
Proximal contacts (15)	8.4 ± 4.9	56.0 ± 27.5	0.93
Cusp positions and morphologies (10)	5.8 ± 3.2	58.0 ± 31.8	0.81
Axial contour (10)	3.7 ± 2.6	36.9 ± 26.0	0.84
Marginal ridge position (10)	5.4 ± 4.5	54.3 ± 45.2	0.62
Accessory ridge and groove (5)	4.1 ± 1.8	81.9 ± 35.2	0.66
Total	53.5 ± 15.8	59.4 ± 17.5	-

SD, standard deviation

The weighted kappa values of the seven qualitative criteria ranged from 0.62 to 0.93. According to Landis and Koch, the reliability level of the qualitative assessment criteria can be estimated as more than “substantial”.²²

The total volumetric discrepancy seemed to be negatively correlated with the total qualitative score of the students’ digital wax-up work. However, no statistically significant correlation was observed between the total volumetric discrepancy and the total qualitative score of their digital wax-up work (ρ = -0.27, P = .09).

All students answered the self-administered questionnaire, with the statements and results of the questionnaire shown in Table 3.

Table 3. Student responses to the self-administered questionnaire (n = 47).

	1	2	3	4	5
Statement	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1. Did you master basic operation procedures of the CG software?	4%	11%	19%	51%	15%
2. Did you master basic operation procedures of the CAD software?	4%	17%	23%	43%	13%
3. Did you understand the process of digital wax-up using the CG and CAD software?	0%	0%	4%	43%	53%
4. Did you master the procedures of trial and adjustment using the crown made with CAD/CAM technology?	2%	2%	21%	51%	23%
5. Was digital wax-up education meaningful to you?	0%	0%	19%	45%	36%
6. Do you want to use CG and CAD software programs in your future education?	2%	26%	26%	36%	11%
7. Do you want to use CG and CAD software programs in your future as a dentist?	2%	11%	19%	45%	23%
8. Do you think it useful to educate dental students with CG and CAD software?	2%	4%	9%	49%	36%

DISCUSSION

This study introduced a set of qualitative evaluation criteria for estimating students’ digital work on newly developed, hands-on, undergraduate preclinical training protocol for designing, manufacturing, and installing a full-coverage crown restoration using multipurpose CAD and free CG software programs. The inter-rater reliability of the new qualitative evaluation criteria was examined.

Regarding actual student performance in this preclinical practice, the mean crown margin integrity and accessory ridge and groove achievement rates were > 80%, and those of the occlusal contact and axial crown contour were < 50%. This difference may be because the students are able to understand the requisites for the margin and accessory ridge and groove more easily than those for occlusal contact and axial crown contour. To rectify this, adding a revision step with the reference digital restoration may allow a better understanding of the requisites for appropriate occlusal contact and axial crown contour. Specifically, a variation of the axial and subgingival contour (such as convex, normal, less) should be understood in terms of aesthetics and cleanability. Combining clinical instruction with the practice and examination cases may be beneficial.

The weighted kappa values for the seven qualitative criteria ranged from 0.62 - 0.93. According to the criteria by Landis and Koch, the inter-rater reliability for all the evaluation criteria was evaluated as “substantial agreement” or “almost perfect or perfect agreement.”²² In a previous study in which instructors visually evaluated the reliability of students’ conventional crown wax-up products, the inter-rater reliability (measured using the intraclass correlation coefficient) was < 0.4.¹⁵ Therefore, the criteria for the qualitative evaluation established in the digital wax-up practice seemed sufficiently operational and showed sufficient inter-rater reliability.

A substantially negative but not statistically significant correlation between the total volumetric discrepancy and the total qualitative score was observed, indicating that the standard evaluation under digital circumstances seemed to be introduced by qualitative assessment based on clinical requisites. The same statistical trend of a weak correlation between the rubric evaluation and the digital quantitative evaluation of some abutment tooth preparation practices was reported in two previous studies, with varying statistical significances due to diverse sample sizes.²⁰,²³ Thus, the total volumetric discrepancy measure can describe the overall estimation of the restorations, but using this measure solely may not be sufficient for evaluating the criteria shown in Table 1. However, we believe artificial intelligence technology will assist in creating an automatic qualitative evaluation methodology to easily estimate the students’ work and the process of the practice.

Regarding the self-administered student questionnaire, approximately 60% of the students strongly agreed or agreed to the question “Did you master basic operation procedures of the CG and CAD software programs?” Basic operational practice should be assigned as homework, considering the short time allowance for mastering the numerous commands of the CG and CAD software programs in a classroom setting. More than 90% of students strongly agreed or agreed to the question, “Did you understand the process of digital wax-up with the CG and CAD software programs?” Furthermore, > 80% of students strongly agreed or agreed to the question, “Was digital wax-up education meaningful to you?” Based on these answers, we believe this training worked well in satisfying the specific behavioral objectives of the preclinical practice.

CONCLUSION

A newly developed hands-on practice for digital design, digital manufacture of crown restoration using multipurpose CAD and free CG software programs (digital wax-up training) was introduced for undergraduate dental preclinical education. In addition, a set of qualitative evaluation criteria that displayed sufficient inter-rater reliability in estimating the students’ overall performance in designing digital wax-up work was introduced. A substantially negative but not statistically significant correlation of the total volumetric discrepancy between each student’s digital wax-up work and an instructor’s digital wax-up reference with the total qualitative score calculated by the set of qualitative evaluation criteria was observed.