Effect of diagnostic score reporting following a structured clinical assessment of dental hygiene student performance

Abstract

Background

Diagnostic score reporting is one method of providing feedback to all students following a structured clinical assessment but its effect on learning has not been studied. The objective of this study was to assess the impact of this feedback on student reflection and performance following a dental hygiene assessment.

Methods

In 2016, dental hygiene students at the University of Alberta participated in a mock structured clinical assessment during which they were randomly assigned to receive a diagnostic score report (intervention group) or an overall percentage grade of performance (control group). The students later reflected upon their performance and took their regularly scheduled structured clinical assessment. Reflections underwent content analysis by diagnostic domains (eliciting essential information, effective communication, client-centred care, and interpreting findings). Results were analysed for group differences.

Results

Students performed best on eliciting essential information (92%) and poorest on interpreting findings (42%). The intervention group was more likely to view interpreting findings as a weakness, p = 0.007, while the control group was more likely to view eliciting essential information as a weakness, p = 0.04. No differences were found on the actual assessment scores, p > 0.05.

Discussion

Students who received diagnostic score reporting appeared to reflect more accurately upon their weaknesses. However, this knowledge did not translate into improved performance. Modifications and enhancements to the report may be necessary before an effect on performance will be seen.

Conclusion

Diagnostic score reporting is a promising feedback method that may aid student reflection. More research is needed to determine if these reports can improve performance.

PRACTICAL IMPLICATIONS OF THIS RESEARCH.

• The ability to accurately self-assess is an essential skill for practising dental hygienists.

• Diagnostic score reporting may help develop this skill during dental hygiene education so it can carry over into professional practice.

INTRODUCTION

This article is 1 of 2 papers published in this issue that report on the development, implementation, and evaluation of a diagnostic score reporting framework for structured clinical assessments in dental hygiene. A literature review describes an evidence-based methodology for developing valid reports that summarize student performance (based on diagnostic domains) and for implementing that framework within a dental hygiene skills-based assessment.¹ This article reports the evaluation component, specifically how students responded to the feedback, as feedback must be both valid and useful to be considered of quality.

Feedback is considered an essential pedagogical tool in higher education.^{2, 3} While assessments evaluate student performance, feedback provides students with detailed information to improve future performances.^{2, 4, 5} Many educators believe students must receive feedback throughout their education for effective learning to take place.

Kolb⁶ describes learning as a cycle. The process begins with experience, which leads to reflection, observations of others, analysis, conceptualization, and finally modified behaviours. Feedback can play an integral role in facilitating this cycle.^3,7,8 Feedback encourages individuals to use their experiences to make improvements, by identifying when performance has deviated from expectations (i.e., starts the cycle).^7,8 Feedback also guides students through the learning process by improving the accuracy of self-assessments and suggesting where reflection and analysis should be focused (i.e., accelerates the cycle).^3,9,10 Reviews have found that externally provided feedback (such as written or verbal information on, or assessment of, a student’s performance or technique) generally leads to improved performance.^11-13 However, not all feedback will have a positive impact, with guidelines recommending timely, clear and concise, task-specific information from an authoritative figure, given with the intent to help the student improve.^4,5,11

Despite its established importance, feedback is frequently reported as inadequate across the health disciplines.^2,14 Structured clinical assessments (SCAs) capture detailed information on students’ knowledge, skills, and abilities (i.e., their competence), presenting key opportunities to provide feedback.^14-17 The most common SCA is the multistation objective structured clinical examination (OSCE) seen in medical education^18,19 ; single client assessments, which often focus on interpersonal communication, are also common in nursing and dental hygiene.^20,21 These assessments are typically high-stakes, and they attempt to mirror authentic clinical practice in a standardized manner, such as by using trained actors and predetermined grading checklists.^18-21

Research suggests that the majority of students will review feedback offered following an SCA.²² However, feedback is often limited in these types of assessments. Traditional methods, such as providing the student with their results by test item, may be precluded by concerns over test security.^23,24 Other methods, such as having the examiner provide immediate verbal or written comments within the context of the assessment, significantly increase the time demands placed on instructors and often lead to rushed, incomplete feedback.^14,25 A lack of instructor time is frequently cited as a key issue precluding the provision of feedback.^9,26,27

Diagnostic score reporting (DSR) is a possible means of providing feedback to students following an SCA. DSR provides test-takers with information on global performance, performance by domains, relative standings to professional standards and/or peers, and specific suggestions for improving performance.^28-30 Domains reflect the specific areas of knowledge, skill or ability that the examination intends to capture, defined by professional standards and competencies, so that student strengths and weaknesses can be readily identified.^28,29,31 Domain scores are determined from a subset of relevant test items and provide additional information to the student beyond a single summary score, including information on how to improve within those areas.^28,32,33 DSR does not reveal the actual test items to students, and the content of the reports can be pre-established and delivered efficiently online, overcoming the major barriers to feedback for SCAs. DSR has been largely confined to nationwide testing programs, and much of the literature has focused on reporting features such as usability and interpretability.^28,29,31-34 The effect of DSR on student outcomes has not been well studied.

The aim of this study was to assess the impact of DSR following an SCA on student-level outcomes, specifically reflection and performance.

METHODS

Ethics approval for this study was obtained from the University of Alberta Research Ethics Office (Pro00062297).

Our companion paper describes the literature-based development and validation process for implementing DSR within a dental hygiene SCA at the University of Alberta in 2016.¹

The dental hygiene structured clinical assessment

The dental hygiene SCA was a comprehensive single-client assessment requiring students to establish a rapport, conduct a full health and dental history, and identify any risk factors contraindicating or requiring modifications to dental hygiene therapy.³⁵ The client was portrayed by a standardized patient (a trained actor), who received prior information on the client’s demographics, medical health and background, and oral health status and beliefs. The SCA was graded by clinical instructors using a checklist of observable items, such as questions the student must ask and conclusions they should make during the encounter—marked yes or no—and rating scales to assess global skills such as organization and communication. There were 5 clinical instructors who marked an average of 8 students. No formal calibration was conducted, although instructors had previous SCA experience and met with the clinical professor before and after the assessment to familiarize themselves with the checklist and address any questions or concerns.

The diagnostic score report

DSR for the dental hygiene SCA included overall scores on the assessment and a breakdown of scores by 4 skill-based domains: 1) effective communication; 2) client-centred care; 3) eliciting essential information; and 4) interpreting findings. The 4 domains were established using the Canadian dental hygiene entry-to-practice competencies³⁶ for guidance. Test items were then mapped to an appropriate domain so students could receive a domain score. The reports also contained peer comparisons and information on how to improve performance within each of the 4 skills. Careful consideration was given to the reporting features (e.g., esthetics and language) in keeping with best practice recommendations.¹ Reports were provided to students through an online portal.

An assessment blueprint, which shows how test items and competencies were mapped to domains, as well as screenshots of the actual report, is presented in the literature review.¹

Study design

An experimental design was used to evaluate the effect of DSR on student-level outcomes. Thirty-nine dental hygiene students scheduled to take the SCA in 2016 were invited to participate in a mock SCA, a practice assessment prior to their final SCA, where they were randomly assigned to either the intervention or control group. The intervention group received DSR after the mock SCA, while the control group received one overall percentage grade of performance. After receiving their results, all students were asked to reflect upon their mock SCA performance and later completed their regularly scheduled SCA. Thus the effect of DSR on reflection and performance could be ascertained. Figure 1 illustrates this design.

The mock structured clinical assessment

The mock SCA covered the same diagnostic domains as the actual SCA, but with different content. The client’s demographics, health history, and oral health perceptions and concerns were thus unique, but the skills required for the assessment were identical. Two content experts (AC & AS) developed the mock SCA using subject matter expertise and course resources such as the clinic manual and course textbook.³⁷ An assessment blueprint was established linking the test items to the diagnostic domains. The reliability of these links was further validated by clinical instructors using a modified Delphi approach to reaching expert consensus.³⁸

The mock SCA ran identically to the regularly scheduled SCA (as described above), with standardized patients given their client history prior to the assessment, and 5 clinical instructors grading using a predetermined checklist (although standardized clients and instructors were not necessarily the same for each SCA).

Figure 1.

Experimental design for evaluation of student-level outcomes

Experimental procedure

The mock SCA took place 12 days before the actual SCA. Both students and graders were blinded to group assignment (DSR or control), and verbal feedback from clinical instructors was prohibited. One day after the mock SCA, students received an email link to either their DSR (intervention group) or their overall percentage of performance (control group). Five days after the mock SCA, students were prompted to reflect upon their performance. Specifically, students were asked to describe what they did well during the mock SCA, recognizing their strengths, and what they could improve upon, identifying their weaknesses. Reflections were coded for their quality and content. The regularly scheduled SCA took place one week later and results were collected for analysis. All students received DSR for the actual SCA once the study data were collected.

Data coding and analysis

All data analysis was conducted using statistical software STATA 14.³⁹ Descriptive statistics were reported using means or percentages, where appropriate. For inferential statistics, in cases of violations of assumptions, equivalent non-parametric tests were used in place of parametric ones.

Analysis of performance

The SCA results were analysed for group differences using linear regression controlling for mock SCA results, according to best practices in test-retest designs.⁴⁰

Analysis of reflection quality

To determine the quality of the students’ reflections, a grading rubric was developed using the University of Alberta Heath Sciences Education and Research Commons (HSERC) Interprofessional Reflection Guide.⁴¹ Through an iterative process, the rubric was adjusted until 2 trained raters (AC & MY) could reliably evaluate the reflections; raters were blinded to group assignment. The final rubric dichotomized reflective statements as either low- or high-quality (Appendix A). Exact inter-rater agreement was 84% with a Cohen’s Kappa of 0.64, indicating substantial agreement,⁴² and any remaining discrepancies were reviewed and coded via consensus. Results were analysed using independent t-tests.

Analysis of reflection content

Each reflection underwent content analysis.⁴³ The units of analysis were the diagnostic domains (effective communication, client-centred care, eliciting essential information, and interpreting findings). The rubric for coding reflective statements into the appropriate domains was based on the assessment blueprint (Appendix B). One statement could have represented multiple domains and an “other” category captured reflections that did not fit within the domains. Two blinded researchers (AC & MY) independently coded each reflective statement. Exact inter-rater agreement was 70% with another substantial Cohen’s Kappa of 0.64. Discrepancies were reviewed and coded via consensus. Reflection content was analysed using Poisson regression.

Table 1.

Structured clinical assessment results: Mean % (SD)

	Control group (n = 20)	DSR group (n = 18)	Total (N = 38)
Mock SCA
Total	73.67 (7.69)	76.30 (7.89)	74.92 (7.80)
Effective communication	75.28 (13.55)	79.01 (10.18)	77.05 (12.06)
Client-centred care	76 (10.01)	76.67 (10.54)	76.32 (10.13)
Eliciting essential information	91.67 (9.45)	92.59 (10.08)	92.11 (9.63)
Interpreting findings	37.86 (10.65)	46.83 (22.35)	42.11 (17.56)
Actual SCA results
Total	82.96 (6.60)	82.20 (7.69)	82.60 (7.05)
Effective communication	88.16 (10.23)	82.16 (9.40)	85.32 (10.18)
Client-centred care	83.00 (10.81)	83.33 (9.70)	83.16 (10.16)
Eliciting essential information	92.31 (8.65)	93.59 (8.02)	92.91 (8.27)
Interpreting findings	45.00 (26.72)	52.08 (25.36)	48.36 (25.36)
Note: nothing significant at p< 0.05

RESULTS

Thirty-eight (38) of the thirty-nine (39) students (97%) participated in the mock SCA, and 37 (95%) submitted a reflection.

The effect of diagnostic score reporting on performance

The results of the mock SCA and actual SCA are found in Table 1. The average score on the mock SCA was 75%. Students scored best on the domain eliciting essential information, averaging 92%. Students performed worst on interpreting findings, averaging 42%. No significant differences were found between the control group and the DSR group on the mock SCA, p>0.05.

The average score on the actual SCA was 83%. Similar trends were found on domain scores, with students performing best on eliciting essential information (93%) and worst on interpreting findings (48%). Controlling for mock SCA scores, there was no significant difference on overall SCA scores between the DSR group and control group, p>0.05. There was a non-significant trend that the DSR group performed better than the control group on interpreting findings by an average of 10%, p= 0.24. There was a borderline significant difference on effective communication scores, p= 0.05, where the DSR group performed poorer than the control group by 7%. Overall, there was limited evidence to support that DSR improved student performance.

The effect of diagnostic score reporting on reflection

The quality of the students’ reflections is described in Table 2. The average score was 0.40, falling below the midpoint of the scoring rubric, which ranged from 0 to 1. No significant differences between the DSR and control group were found for reflection quality overall or by question (i.e., strengths vs weaknesses), p>0.05.

The result of the content analysis is presented in Table 3. The students’ reflections fit well within the 4 diagnostic domains, with only 7% of the comments allocated to the “other” category. The majority of the comments focused on eliciting information (33%) and effective communication (29%), with fewer reflective statements on client-centred care (18%) and interpreting findings (13%). The 2 prompts on strengths and weaknesses were analysed separately for group differences.

Table 2.

Quality of student reflections: Mean (SD)

	Control (n = 20)	DSR (n = 17)	Total (N = 37)
Strengths	0.5 (0.43)	0.37 (0.45)	0.44 (0.44)
Improvements	0.38 (0.46)	0.34 (0.42)	0.36 (0.44)
Total	0.44 (0.34)	0.36 (0.27)	0.40 (0.31)
Note: nothing significant at p < 0.05

Table 3.

Content of student reflections: Frequency (%)

		Effective communication	Client-centred care	Eliciting essential information	Interpreting findings	Other
Strengths	Control	11 (22%)	8 (16%)	21 (40%)	6 (12%)	4 (8%)
	DSR	13 (35%)	7 (19%)	14 (38%)	1 (3%)	2 (5%)
	Combined	24 (28%)	15 (17%)	35 (40%)	7 (8%)	6 (7%)
Weaknesses	Control	15 (35%)	8 (19%)	15 (35%)a	2 (5%)a	3 (7%)
	DSR	8 (24%)	7 (21%)	4 (12%)a	13 (38%)a	2 (6%)
	Combined	23 (30%)	15 (19%)	19 (25%)	15 (19%)	5 (6%)
Total	Control	26 (28%)	16 (17%)	36 (39%)	8 (9%)	7 (8%)
	DSR	21 (30%)	14 (20%)	18 (25%)	14 (20%)	4 (6%)
	Combined	47 (29%)	30 (18%)	54 (33%)	22 (13%)	11 (7%)
a significant at p < 0.05 Percentages reflect row totals and may not sum to 100% due to rounding.

Significant differences were found in the identification of weaknesses. The DSR group had 7.65 times the average number of comments on interpreting findings than the control group, p = 0.007, and 0.31 times (or 70% fewer) comments on eliciting essential information, p = 0.04. Figure 2 illustrates that students within both groups performed best on the domain eliciting essential information and worst on interpreting findings. Therefore, the DSR group appeared to reflect more accurately upon their weaknesses.

In regard to identifying strengths, the DSR group was more likely to reflect upon eliciting essential information as a strength compared to the control group (35% vs 22%), and only 1 student in the DSR group reflected upon interpreting findings as a strength compared to 6 students in the control group (3% vs 12%). While these differences were not significant, p > 0.05, these trends support the claim that DSR resulted in more accurate self-assessments.

Figure 2.

Mock structured clinical assessment percentages by domain

DISCUSSION

DSR is a viable framework for providing feedback to all students following an SCA. DSR describes test results by the underlying domains the test intends to measure and includes resources for making individual-level improvements.^28,29,32,33 Despite the potential of DSR, its effect on student-level outcomes has not been well investigated.

Reflection is a key component of learning.^6,8,44 Through the process of experiencing, reflecting, thinking, and acting,⁶ each SCA provides a learning opportunity that could improve future clinical practice. Accurate self-assessment aids this process by helping students to notice when their performance needs improvement, prompting the learning cycle to start,^7,8 and may speed up learning by directing students to focus on key issues, eliminating uncertainty or misdirection.³ The ability to correctly identify problem areas is one important element of reflection-in-action, a life-long skill essential for health care professionals.⁸ Feedback is believed to facilitate this type of reflection^9,45 and yet feedback is often limited following an SCA due to time limitations and concerns over test security. DSR overcomes these issues and this study showed that this feedback mechanism did appear to encourage student reflective capabilities in a positive direction.

Nevertheless, this study did not find that DSR facilitated improved performance. While students were better able to identify their strengths and weaknesses, this awareness did not translate into a behaviour change. These findings suggest a breakdown of the learning cycle between the stages of reflection and action. Ultimately, the goal of feedback is to improve performance,^2,5,8 and reasons for DSR not achieving this goal should be examined.

One possibility is that DSR may need to be provided consistently over a longer time span before it will have any impact. A review of feedback and physician performance showed that studies with longer durations were more likely to find significant effects.¹¹ Similarly, a review of audit and feedback found that feedback was more effective when provided more than once.¹² This current study took place over a few weeks and only provided DSR a single time.

Another explanation could be that simply providing resources to students is insufficient to encourage meaningful interactions with that information. Even if students are aware of their weaknesses, they may lack the motivation to improve. Characteristics such as confidence, self-esteem, self-efficacy, personal beliefs, and intrinsic interests can all affect a student’s ability to self-regulate and become accountable for their own learning.^46-48 The design of this study made it difficult to determine exactly howstudents interacted with their feedback, and more qualitative research in this area may help pinpoint key motivational factors.

Finally, performance may not have improved because of issues with the feedback itself. Quality feedback should be task specific,^2-5,9,27 and the information provided through DSR may not have been specific enough to alter behaviour. Providing students with the actual test items would be more precise, but for high-stakes SCAs, where test security is a chief concern, this option is not feasible. When actual test items cannot be revealed, providing example test items is a practical alternative.³² Other options include providing web links to specific sections of textbooks or other online documents,^28,49 or video-based feedback where students view their own performance or exemplars of professional dental hygienists interacting with clients.^3,50,51 The degree of report individualization could also be improved.^28,33,49 Different reporting components could be displayed based on different SCA checklist response patterns or different levels of achievement. Low achievers might benefit from information on how to interpret their scores, while advanced students might benefit from supplemental learning activities.⁴⁹ Online DSR can be expanded and modified to better encourage student-level improvements.

A limitation of this study was its small sample size, which limited statistical power and precluded more in-depth analyses., Results may not be generalizable to other health professions. This research focused on a select dental hygiene student population at a single school within a single term. Furthermore, each dental hygiene student interacted with only a single standardized patient and was graded by one clinical instructor (as compared to an OSCE that uses multiple stations and graders), and while the reliability of examiner grading was outside the considerations of this project, it may have influenced the results. This project attempted to overcome this issue by randomizing students (and thus their clients and instructors) to the control and intervention groups. Finally, there may also have been contamination between the intervention and control groups, where score reports and information on how to improve were shared between classmates. A next step in this area of research would be to replicate this study with a larger, more diverse group of students.

CONCLUSION

Providing DSR to students following a dental hygiene SCA resulted in more accurate self-assessments but did not improve performance. Online DSR offers a promising feedback framework and enhancing the reports may facilitate behaviour change. Suggestions include providing links to relevant references, incorporating video feedback, and developing more personalized/individualized reports.

CONFLICT OF INTEREST

The authors declare they have no financial, economic or professional interests that may have influenced the design, execution or presentation of this scholarly work.