Correlation Between Student Performances on Case-Based Constructed-Response Formative Assessment and Summative Assessment

Abstract

OBJECTIVES

This study aimed to evaluate the impact of formative assessment with case-based constructed-response question (CRQ) formats on student performance on the final summative assessment in the second-year periodontics course.

METHODS

Classroom quizzes with case-based CRQs were implemented as the formative assessment during the course. Each student received feedback on their responses from the course director. After all students (N = 128) took the second-year final examination, the Friedman test was conducted to compare student performances in each assessment over time. The multiple linear regression (MLR) model was used to evaluate the association between the second-year final examination score and plausible predictors—student gender, the second-year formative and midterm examination scores, and time spent on the final examination.

RESULTS

The mean % scores in the formative assessment (51) and midterm (84) examination were significantly lower than that of the final (87) examination (P < .01). The number of students who failed the final (6) examination was significantly lower than the midterm (16) examination (P = .03). The midterm (P < .0001) and the formative assessment (P = .0009) scores significantly affected the second-year final examination score while student gender (P = .59) and time spending (P = .83) showed no correlations.

CONCLUSION

Within the limitations of the study, student performance on case-based CRQs was correlated with student performance on the summative assessment.

Introduction

While summative assessment at the end of the course is critical to measure student learning, ¹ predicting student performance in summative assessment is challenging because various known or unknown factors may affect student performance.^2–4 Identifying deficiencies significantly affecting student performance in summative assessment would be helpful to improve student academic achievement. Therefore, low-stakes formative assessment may be useful to collect information and feedback from students with respect to the ongoing teaching and learning context, ⁵ and predict student performance in high-stakes summative assessment.^6,7 However, the validity of scores from such low-stake assessments is potentially susceptible to examinees’ effort because they may not devote themselves to considerable work. ⁸ Providing high-quality feedback following the formative assessment is challenging especially for the large-sized class. ⁹

Classroom quizzes with multiple-choice questions (MCQs) can be implemented during the course as formative assessment prior to the midterm and final examinations. While MCQ tests can be graded quickly and accurately, one of the most common critiques for MCQs is their inability to assess in-depth comprehension on the subject matter. ¹⁰ In fact, most students performed well on the MCQ quizzes in the second-year periodontics course; student performance on the quizzes did not provide any estimations for their performance on the summative assessment. In contrast, tests with constructed-response questions (CRQs) are considered to be superior to measuring critical reasoning skills. ¹⁰ While MCQs are a closed form, in which students need to select the correct answer, CRQs are open-ended formats; students must provide their own answers. CRQs demand different skills from students to answer them correctly. The different skills range from factual recalling for fill-in-blank or short-answer questions (SAQs) to integrating multiple facts into a rational context for essay-type questions. ¹¹

The most preferred aspect of CRQs may be the capability to evaluate students’ problem-solving skills by presenting real-life or similar situations which students may encounter in the clinic via case-based formats. ¹⁰ Case-based questions are based on the use of condition and/or disease scenarios. ¹² Since the integrated National Board Dental Examination intends to test student problem-solving skills on various cases, ¹³ case-based CRQs have been incorporated in the preclinical second-year periodontics course for students to train problem-solving skills on simulated cases. However, students have shown poor performance in case-based CRQs on the second-year periodontics examinations. ³ Their poor performance in the CRQs negatively affected their scores in the final examinations that included MCQs, SAQs, and essay-type questions. The second-year final examination scores had been considerably lower than the first-year final examination scores within the classes. Therefore, two classroom quizzes with case-based CRQs have been implemented as the formative assessment in preparation for the second-year final examination.

The purpose of this interventional educational action research was to evaluate the impact of formative assessment with case-based CRQ formats on student performance on the final summative assessment in the second-year periodontics course. The null hypothesis was that there is no association between student performances in the implemented CRQ formative assessment and the final examination.

Methods

Participants and study outline

This study was conducted under the deemed exemption from the Institutional Review Board (IRB) at the University of Maryland, Baltimore (HP-00082574). The study was conducted with one cohort via a retrospective longitudinal assessment from September 2022 to May 2023. The second-year periodontics course is a year-long course, which takes place over the combined fall/spring semester. The fall semester included 10 lectures and the midterm examination, and the spring semester included 13 lectures and the course final examination. The course final examination is cumulative, covering all 23 lectures in the course. Faculty delivered the lectures in the classroom; the classroom lectures were recorded and posted on the online learning management system ¹⁴ after the completion of each lecture. All formative and summative assessments in the course were delivered via an online assessment platform ¹⁵ in the classroom under the surveillance of faculty proctor. Achieving <70% in each summative assessment was marked as failure.

The first classroom quiz with four case-based CRQs was administered 4 weeks before the midterm examination; the second one with two case-based CRQs was administered 6 weeks before the final examination. The number of CRQs and the timing of administration were determined based on lecture contents and course lecture and examination schedules. Due to spring break, the interval between the second formative assessment and the course final examination was increased to 6 weeks. A weight of 3% was assigned to the results of the formative assessments in calculating the final course grade.

Case-based CRQs covered the following topics: diagnosis and etiology for a gingivitis case, local contributing and risk factors, and stage and grade for a periodontitis case. These topics are essential to make a diagnosis and to further formulate appropriate treatment plans for patients. All case-based CRQs were written by the course director (SO) with another faculty member (DJ). Figure 1 presents examples of case-based CRQs. Three faculty members (SO, DJ, and GS) reviewed the questions and agreed on the correct answers. The grading of the CRQs in the quizzes was divided among three faculty (SO, DJ, and GS) to lessen the burden of grading CRQs for the large-sized class. Using the scoring rubric for CRQs (Table 1), the faculty members graded the same assigned questions for all 128 students to minimize potential variations. After grading, each student received feedback on their answers from the course director via emails. Table 2 presents examples of student responses and feedback from the course director. The CRQs in the quizzes were also reviewed during the subsequent classroom lectures with the best exemplars from the class.

Figure 1.

Examples of case-based CRQs in formative assessment.

Table 1.

A scoring rubric.

Score	Criteria
4	Correct answer with listing all key supporting evidence, using proper terms
3	Correct answer with missing key supporting evidence, using proper terms
2	Correct answer with mixing correct and incorrect supporting evidence, using proper terms
1	Correct answer with listing only incorrect supporting evidence, not using proper terms
0	Incorrect answer to the question

Table 2.

Examples of student answers and feedback from the course director.

Question	Student Answer	Feedback
Etiology for a gingivitis case	Crohn's disease and smoking	The etiology for periodontal disease is bacterial plaque with ensuing host responses. Smoking is the risk factor. There is no evidence that Crohn's disease directly causes gingival inflammation.
Risk factor for periodontal disease	Risk factors with this patient are high with gingival recession and bone resorption.	Gingival recession and bone resorption are the results of periodontal disease. Risk factors for periodontal disease, such as smoking and diabetes, increase the probability of developing periodontal disease.
Local contributing factor for periodontal disease	The local contributing factor is the factor which can contribute to the likelihood of periodontal disease occurring.	The local contributing factors (LCFs) for periodontal disease maintain the disease by keeping plaque close to the gingiva. The LCFs do not directly induce gingival inflammation. Your statement is for the risk factors of periodontal disease.
Grade assignment for a periodontitis case	Grade B because her HbA1c level is 6%.	The primary determinant for assigning a grade is % bone loss (BL)/age as indirect evidence of progression. The ratio of % BL/age was approximately 1.3. Therefore, Grade C should be assigned regardless of the HbA1c level.

The course director (SO) and the course co-director (GS) wrote all the midterm and final examination questions, reviewed the questions, and agreed on the correct answers. The midterm examination contained 24 MCQs and 12 SAQs. The final examination was cumulative, covering all 23 lectures, and contained 45 MCQs, 7 SAQs, and 6 essay-type CRQs. The students had the opportunity to write comments on the final examination after they submitted their answers. Grading for SAQs and CRQs in the midterm and final examinations was conducted only by the course director (SO) using the same scoring rubric (Table 1). Item analyses were conducted for the midterm and final examinations to make sure that their question difficulty level was over 0.6 and discrimination power was ≥0.2. Two MCQs from the midterm and final examinations were dropped respectively because their item difficulty index was 0.2.

Statistical analysis

Given the nature of this coursework, we included the entire class. Because the sample size of 128 was robust for various analyses especially within educational settings, a formal sample size calculation was not performed.

The scores from the two quizzes were combined as the formative assessment score in statistical analysis because the topics covered by the formative assessments were from the lectures in fall semester. Student performances within the class in each assessment were compared with the Friedman test because the distributions of % scores did not pass the normality test conducted by the Kolmogorov-Smirnov normality and the Shapiro-Wilk's tests. Multiple comparisons were conducted by Dunn's test. The number of students who failed the second-year midterm and final examinations was compared with a chi-square test.

Descriptive statistics were used to identify deficient areas from the formative assessment. Gender, the second-year midterm and formative assessment scores, and time spent in taking the second-year final examination were selected as predictors. Multiple linear regression (MLR) model was built to investigate the associations between the four predictors and the second-year final examination score. All analyses were performed by tracking individual students on all variables (matched groups). The analyses were performed with GraphPad Prism (version 9.4.1; GraphPad Software, Inc. CA); P < .05 was considered significant.

Results

The mean final examination score of this cohort in the first-year periodontics course was 86 ± 6. The study class included 74 (57.4%) female and 55 (42.6%) male students. The mean % examination scores with standard deviations were 84 ± 10 (the second-year midterm examination), 51 ± 19 (formative assessment), and 87 ± 9 (the second-year final examination), showing a significant variation over time (Friedman test, P < .0001). In multiple comparisons with Dunn's test, the mean % scores of the second-year midterm examination (P = .0099) and formative assessment (P < .0001) were significantly lower than the second-year final examination (Figure 2). However, no significant difference (P > .9999) was observed between the first- and second-year final examination scores (Figure 2).

Figure 2.

Comparisons for student performances in each assessment. Dunn's test; ns not significant, ** P < .01, **** P < .0001.

Sixteen students (13%) failed the second-year midterm examination while six students (5%) failed the second-year final examination. The number of students who failed the final examination was significantly reduced compared to that in the midterm examination (chi-square test, P = .03). Among the six students who failed the final examination, two students failed both midterm and final examinations although their scores improved from 60 to 66. The other four students achieved scores between 79 and 90 in the midterm examination, but their final examination scores ranged from 63 to 68.

Table 3 presents student performances in each question in the formative assessment. Less than 50% of the second-year students answered correctly on four of the six questions: diagnosis of gingivitis (6.2%), defining and recognizing risk factors (32.8%) and local contributing factors (24.2%), and assigning grade for a periodontitis case (6.2%). Seventy-eight students (61%) were assigned the correct stage for a periodontitis case and 99 students (77.3%) correctly answered the etiology for a periodontal disease. The correct answer rates in assigning stage (93%) and grade (73.6%) for a different periodontitis case on the final examination were improved.

Table 3.

Student performances in the formative assessment with CRQs.

Question		The number of students with the correct answers (N = 128)
Quiz 1	Diagnosis for a gingivitis case	8 (6.2%)
	Etiology for a gingivitis case	99 (77.3%)
	Risk factor for periodontal disease	42 (32.8%)
	Local contributing factor for periodontal disease	31 (24.2%)
Quiz 2	Stage assignment for a periodontitis case	78 (61%)
	Grade assignment for a periodontitis case	8 (6.2%)

Ten students left comments after the final examination. Students thought the second-year final examination was “difficult” and “tricky” although a few students, who left such comments, achieved over 90% scores. One student left a comment that it can be very challenging to see radiographs and determine the exact % of bone loss in the examination room with all the lights on. One student stated that too many questions related to the prognosis were included in the examination. This student answered “fair” or “questionable” to the classification questions; the answers were supposed to be Stage II or Grade B.

The MLR model was used to evaluate the associations between the selected predictors and the final examination score (Table 4). Based on the MLR model, the second-year midterm (P < .0001) and formative assessment (P = .0009) scores were significantly correlated with the second-year final examination score. However, student gender (P = .59) and time spent in taking the second-year final examination (P = .83) did not affect student performance in the second-year final examination.

Table 4.

MLR model.

Outcome	second-year final examination score
Regression type	Least squares
Model Diagnostics	Degrees of Freedom = 4, F = 16.43, P < .0001
Goodness of Fit	Degrees of Freedom = 123, R2 = 0.35, Adjusted R2 = 0.33
Parameter estimates
Parameter	Estimate	Standard Error	95% Confidence Interval	|t|	P value
Intercept	49.31	6.63	36.20 to 62.42	7.443	< .0001
Second-year Midterm	0.36	0.07	0.23 to 0.49	5.447	< .0001
Formative assessment	0.12	0.04	0.05 to 0.19	3.399	.0009
Gender [male]	0.71	1.31	−1.88 to 3.29	0.542	.59
Time spent on the second-year final examination	0	0.05	−0.1 to 0.1	0.215	.83

Discussion

This study examined the correlation between student performances on the formative assessment and the final examination in the second-year preclinical periodontics course. The regression model (Table 4) demonstrated that the case-based CRQ formative assessment significantly affected student performances on the summative assessment (P < .0009), by rejecting the null hypothesis.

Periodontics primarily deals with the prevention, diagnosis, and treatment of periodontal diseases. Dental students must attain sufficient knowledge in periodontics during preclinical education. While assessment in dental education embraces a wide range of concepts, preclinical assessment should include a diagnostic tool to identify learning deficiencies among students. ¹⁶ Therefore, formative assessments with case-based CRQs were administered as assessment for learning. ⁵

Attention to low-stakes assessment without grading has been increased in medical education. ¹⁷ However, students may not give their best efforts to take those tests, especially with no grade assigned. ¹⁷ Therefore, low weight (3% in calculating the final course grade) has been assigned to the formative assessment in this study to provide incentive to the students who performed well.

The results of the formative assessment revealed that the most difficult tasks for the students were diagnosing gingivitis and determining a grade for a periodontitis case (Table 3). This may be due to clinical reasoning errors in interpretating clinical data. ¹⁸ Most students seemed not to grasp the meaning of the probing pocket depth. Therefore, they used the pocket depths as the primary diagnostic parameter. This also indicates that students had difficulty in evaluating radiographic images with respect to the normal crestal bone level. Students knew and described the basic information, but they could not apply their knowledge to solve clinical problems. Students paid more attention to medical history rather than dental clinical data because they were confused among the etiologic, risk, and local contributing factors for periodontal disease.^3,19 This led to students’ poor performance in assigning the grade for a periodontitis case. After reviewing students’ responses, the course director identified these deficiencies and tried to provide constructive feedback rather than simple answers (Table 2).

Reviewing the results of the formative assessments influenced the following lecture contents and teaching focuses. Lecturers modified subsequent lectures to review the concepts which the students were struggling to understand. This adaptation allowed for a more tailored approach, aiming to enhance student comprehension.

Feedback is the information provided by instructors regarding aspects of student's understanding or performance. ²⁰ While either negative or positive feedback can be delivered, feedback is supposed to be constructive. ²¹ Constructive feedback should be clear, tailored, factual, and descriptive based on the direct observation, provide a reasonable amount of information, and be delivered in a timely manner. ²² The course director tried to provide feedback to individual students via online (emails) and to the entire class via offline (classroom lecture) platforms. ²³

While all 128 students received feedback and the correct answer rates for assigning stage and grade were improved on the final examination, it was not clear how they utilized feedback to enhance their performance on the summative assessment. Students’ attitude toward learning, as the receiving end, affects the effectiveness of constructive feedback. ²⁴ While deep learners are more motivated to learn and enhance their performance, strategic learners may adopt tactics aiming toward academic success. ²⁵ Furthermore, surface learners focus on the superficial features of the subjects and may skip over any content that they think is not related to their final goal. ²⁵ Surface learners are often motivated by extrinsic factors and their engagement and commitment are not solid. Educational tactics to engage surface learners with feedback should be further explored.

Student performances on examinations are influenced by various factors, which may be controllable or not. ²⁶ Uncontrollable factors include students’ health and personal issues, personal learning strategies, levels of test anxiety, and cultural or socioeconomic backgrounds. ²⁶ This study maintained the same environments of examination classroom and the same style via online assessment. Student gender and time spent on the examination did not affect the final examination outcome (Table 4). Four students, who passed the midterm examination with scores between 79 and 90, failed the final examination. While students’ personal factors, such as family problems, over confidence/lack of confidence, or use of feedback, may affect their performances on the final examination, this study could not evaluate those factors.

External validity of the study findings is limited due to several factors. First, the formal sample size calculation was not performed. The decision to include the entire class was based on practical constraints and the unique context of educational research. The absence of a predetermined sample size calculation may raise questions regarding the generalizability or statistical power of the study findings. However, the inclusion of the entire class enabled us to collect comprehensive data. Second, the students in this study received the same experience with no control group. Although the MLR model tracked individual students to evaluate the correlation between the formative and summative assessments, we did not know whether the students would have performed more poorly without implementing CRQ formative assessments. Third, this study did not obtain students’ perceptions on case-based CRQ formative assessment and feedback from the course director. As students’ perceptions may affect their motivation and learning behavior, ²⁷ obtaining students’ perceptions may be helpful to understand how they utilize formative assessment and feedback. ²⁸ Finally, preclinical periodontics courses heavily focus on delivering and testing explicit knowledge as “know what”, although there is a component for manual skill development via simulation-based learning. ²⁹ Therefore, the format of formative assessment in this study may not be applicable for disciplines in which attaining manual dexterity and testing manual skill levels are a major part of courses.^30,31

Conclusions

Implementing case-based CRQs as the low-stakes formative assessment assisted the course director in identifying areas of deficiency on ongoing teaching. Within the limitations of the study, student performance on case-based CRQs was correlated with that on the summative assessment. Further study is needed to include more cohorts and to determine specific effects.