Development of diagnostic score reporting for a dental hygiene structured clinical assessment

Abstract

Background

Structured clinical assessments capture key information about performance that is rarely shared with the student as feedback. The purpose of this review is to describe a general framework for applying diagnostic score reporting within the context of a structured clinical assessment and to demonstrate that framework within dental hygiene.

Methods

The framework was developed using current research in the areas of structured clinical assessments, test development, feedback in higher education, and diagnostic score reporting. An assessment blueprint establishes valid diagnostic domains by linking clinical competencies and test items to the domains (e.g., knowledge or skills) the assessment intends to measure. Domain scores can be given to students as reports that identify strengths and weaknesses and provide information on how to improve.

Results

The framework for diagnostic score reporting was applied to a dental hygiene structured clinical assessment at the University of Alberta in 2016. Canadian dental hygiene entry-to-practice competencies guided the assessment blueprinting process, and a modified Delphi technique was used to validate the blueprint. The final report identified 4 competency-based skills relevant to the examination: effective communication, client-centred care, eliciting essential information, and interpreting findings. Students received reports on their performance within each domain.

Discussion

Diagnostic score reporting has the potential to solve many of the issues faced by administrators, such as item confidentiality and the time-consuming nature of providing individual feedback.

Conclusion

Diagnostic score reporting offers a promising framework for providing valid and timely feedback to all students following a structured clinical assessment.

PRACTICAL IMPLICATIONS OF THIS RESEARCH.

• Diagnostic score reporting helps students make important connections between dental hygiene competencies, their education, and their clinical practice.

• Improving the feedback received during training may produce more mindful clinicians.

INTRODUCTION

This article is the second of 2 papers published in this issue that report on the development, implementation, and evaluation of a diagnostic score reporting framework for structured clinical assessments in dental hygiene. This review article examines the development and implementation components; an original research article¹ presents the evaluation component, specifically how students responded to this novel feedback approach.

Structured clinical assessments (SCAs) are a necessary part of health education to ensure that students not only have knowledge but are able to apply this knowledge competently.² The most common form of SCAs is the objective structured clinical examination (OSCE), which involves multiple stations, each with a structured scenario portrayed by a standardized patient (a trained actor), and graded by qualified examiners who use a predetermined checklist of yes/no criteria.^{3- 6} Other SCAs, such as those seen in nursing or dental hygiene education, may involve a comprehensive single client interaction with a focus on interpersonal communication and health promotion.^{7- 10} Regardless of form or content, an SCA captures information on the achievement of competence in an objective and realistic manner.^2-4

Despite the detailed information captured about the student in an SCA, this information is rarely communicated to the student as feedback.¹¹ Time restrictions and test security are key issues. Providing feedback is a time-consuming endeavour requiring extensive faculty commitment,^12-15 and feedback must be provided to students in a timely manner to be effective.^13,16-19 Therefore feedback is either not provided¹¹ or provided too late to have an impact.²⁰ The development of a reliable and valid SCA is also labour intensive, thus similar examinations are reused across administrations, and these assessments are frequently high-stakes (i.e., they determine the ability to progress within the program).¹² As such, there is a reasonable security concern that providing students with certain types of feedback, such as the assessment checklist itself, may contaminate future results and the validity of subsequent decision making.^21,22 Novel approaches to providing feedback are required.

Diagnostic score reporting (DSR) presents a possible feedback framework for SCAs. DSR summarizes student performance on key domains of learning captured throughout the assessment, with an emphasis on encouraging improvements at the individual level.^23,24

Domains reflect specific areas of knowledge, skill, and/or ability, so that student strengths and weaknesses can be readily identified.^22-27 In keeping with best practices for communicating these results, reports should include clear and concise language, definitions of terms, esthetically pleasing designs, graphical representations of data, timely delivery, and cohort comparisons.^23-26,28 These suggestions mirror the literature on providing quality feedback in higher education.

DSR overcomes the major barriers presented by SCAs. Structuring feedback by diagnostic domains allows performance to be analysed without revealing the test items to students (i.e., test security is maintained). The predetermined structure helps to facilitate timely feedback, which can be further improved using electronic grading to provide near-instant feedback via online platforms. Improving efficiency would allow all students to receive diagnostic information without drastically increasing the time demands placed on instructors.

DSR has been primarily used for large-scale assessments,^{22,24,26,28,29} with scarce, literature on applying the framework to smaller scale testing scenarios such as dental or dental hygiene SCAs. Research on feedback with some elements of DSR is slowly emerging in the health literature.^20,30,31 However, a common methodology for developing valid diagnostic domains and quality reports has not been established. For example, Taylor and Green²⁰ developed domain-based feedback for their medical SCA by mapping test items to a nationally defined competency,³² but they did not provide the feedback in a timely manner, nor did they appear to provide suggestions for improvement. As a result, their feedback had little impact on student learning. Establishing a practical DSR framework for SCAs could guide educators to provide high-quality feedback.

The objectives of this project were to 1) describe a general framework for applying DSR within the context of an SCA, in order to provide valid and quality feedback to students; and 2) to implement that framework within a small-scale dental hygiene SCA.

METHODS

Ethics approval for this study was obtained from the University of Alberta Research Ethics Office (Pro00062297).

A literature review was conducted in the areas of structured clinical assessments, test development, feedback in higher education, and diagnostic score reporting. Key findings were condensed into a practical framework for providing feedback following an SCA. A template report was then created, validated, and implemented using this framework.

Framework for diagnostic score reporting

The 2 major elements of DSR are 1) establishing valid diagnostic domains through assessment blueprinting; 2) translating that information to students through effective reports. Finally, an evaluation component is required to ensure DSR is meeting its objectives.

Assessment blueprinting

An assessment blueprint links the purpose of the assessment, the domains being assessed, and the test items measuring those domains.^22,33,34 Supporting evidence is also provided to validate those links. Although assessments are typically designed based on course objectives (knowledge, skills, and attributes taught within the course), this article demonstrates how assessments can be linked to overarching competencies of the profession where some required skills and knowledge extend beyond courses. A top-down approach to competency blueprinting within the context of an SCA is illustrated in Figure 1, adapted from the evidence-centred design method of test development.³⁴

The first step in this process is to identify all competencies relevant to the assessment based on the test specifications (the purpose of the test).³³ Competencies are then grouped into domains based on similarity or themes using a form of iterative thematic analysis.³⁵ Domains for large assessments often focus on specific areas of knowledge²⁴ (e.g., a specific disease), however an SCA may require different types of knowledge to be grouped together under specific skills (e.g., identifying contraindications), keeping in mind that each domain must be assessed by multiple test items. Developers are encouraged to review the literature and use their expertise, the goal of the assessment, the number of test items, and the student perspective to determine the most appropriate groupings (e.g., by knowledge area, skill or cognitive hierarchy).^23,26 The domains are defined by the overarching theme of the grouped competencies.

Each test item is then mapped to the most appropriate domain. There should be several test items representing a single domain, with more items increasing the domain’s reliability.³⁶ A delicate balance must be struck between reliability (having enough test items within each domain to reliably calculate a score) and specificity (having a domain specific enough that the student finds the information useful).²⁶ As psychometric reliability can be difficult to achieve in small-scale testing, this article focusses on content validity while the practical implications are reviewed in our companion paper.¹

For a more detailed explanation of blueprint development, please refer to The Handbook of Test Development.^33,34 The assessment blueprint forms the main structure of DSR (Figure 2). Domain scores for the student reports are generated by the number of test items they successfully complete within that domain.

The final blueprint should be externally validated to ensure accuracy. One such method is the Delphi technique³⁷, an iterative survey-based approach to reaching consensus among multiple experts. In this technique, several experts independently evaluate the results of the qualitative analysis (the blueprint) by using a numeric rating scale. The blueprint is then adapted until the average/median ratings of the experts are acceptably high based on a predetermined criterion. This method is favoured because of its use of both expert opinions and an objective measure of success.

Report creation

After the assessment blueprint is developed, the next step is to determine what output to provide students as feedback. There are a variety of ways to present information to students, particularly with electronic media, but a minimum number of fundamental components should be included. The purpose of the SCA overall, and the domains individually, should be clearly defined.^23-26,28 Information on how to improve within each domain should also be included, such as references to textbooks and other learning resources, and/or ways to interpret and use the feedback to improve performance.^23,24,36 Performance markers such as cohort comparisons or comparisons to expected levels of achievement may help students better comprehend their results.²⁶ Where appropriate, developers should include graphs, colour, headings, and white space to increase report useability.^23-26,28

Figure 1.

Assessment blueprinting process for diagnostic score reporting

Figure 2.

Framework for diagnostic score reporting

Evaluation

Following implementation of DSR, an evaluation component is necessary to ensure that the content of these reports is not only theoretically valid but also practically useful. Much of the score reporting literature focuses on assessing the usability and interpretability of these reports by stakeholders (in this case the students), which may involve a survey-based approach to gathering student opinions and appraising their understanding of their reports.²⁶ However, the literature on feedback in higher education goes further to suggest that, if the ultimate goal of feedback is to improve student learning, then student outcomes (e.g., impact on performance) must be assessed.³⁸ As this is such a key step in the process of producing quality feedback, it is described in greater detail in our companion paper.¹

Implementing the framework for diagnostic score reporting

The DSR framework described above was applied to a dental hygiene SCA at the University of Alberta in 2016. This history-taking SCA was a comprehensive single client assessment, where students applied their interpersonal communication skills to establish a rapport, conduct a full health and dental history, and identify any risk factors contraindicating or requiring modifications to dental hygiene therapy. The SCA was evaluated using a grading checklist of observable items, such as key messages the student is required to communicate to the client (e.g., offering to provide tobacco cessation strategies), marked yes or no. There were also rating scales to assess global skills used throughout the interaction (e.g., organization and communication). The dental hygiene history-taking SCA assessed a variety of essential practitioner skills. However, feedback had been previously limited due to time demands and test item confidentiality. DSR offered a feasible strategy to provide all students with feedback.

A modified Delphi approach (dictated by expert availability) was used to validate the assessment blueprint. Four dental hygiene clinical instructors were asked to independently rate the appropriate level of fit of the competencies and test items within each diagnostic domain based on a 5-p,oint scale (no fit to excellent fit) (Appendix A). Additional space was available for raters to make comments. Any item that received a median score of 3 or lower was reviewed. Changes were made to the blueprint where appropriate.

RESULTS

Assessment blueprinting

Two researchers (AC & AS) followed the blueprinting process described above, guided by the Canadian dental hygiene entry-to-practice competencies.³⁹ Note that the Canadian competencies for baccalaureate dental hygiene programs⁴⁰ were also available but not used because the University of Alberta had a diploma exit option at the time of the study and most other dental hygiene programs did not offer a bachelor’s degree. As a result, the entry-to-practice competencies seemed more generalizable.

After the researchers reconciled discrepancies through discussion and consensus, 4 competency-based skills were identified in the blueprinting process: 1) effective communication; 2) client-centred care; 3) eliciting essential information; and 4) interpreting findings.

Further modifications to the blueprint were made following the validation process using a modified Delphi technique. Regarding each item’s fit within its assigned domain, the raters gave 2 competencies and 3 test items a median score of 3 or lower. AC & AS reviewed the items and, with the help of the raters’ comments, adjusted the blueprint. Specifically, 1 competency and 2 test items were moved to a more appropriate domain and the definition of 1 domain was made more inclusive.

Table 1 presents the final assessment blueprint, including domain definitions, relevant competencies, and examples of test items (for security reasons actual test items are not provided).

Report creation

The DSR output included all the information from the blueprint (excluding the test items), as well as course expectations, cohort comparisons, and information on how to improve. Best practices of effective score reporting were followed.²⁵ Timeliness was facilitated by making the reports available to students online. Since the SCA was graded using iPads, domain scores were automatically calculated once results were submitted.

Table 1.

Assessment blueprint for a dental hygiene structured clinical assessment

Domain	Definition	Competencies	Test items
Effective communication	This skill emphasizes how you are communicating, as opposed to what is being said. Effective communication strategies make the client feel safe and comfortable. It also means providing information in an appropriate manner that the client can follow and understand.	Use effective verbal, non-verbal, visual, written, and electronic communication. Demonstrate active listening and empathy to support client services. Select communication approaches based on clients’ characteristics, needs, and linguistic and health literacy level. Facilitate confidentiality and informed decision making in accordance with applicable legislation and code of ethics. Convert findings in a manner relevant to clients using the principles of health literacy. Manage time and other resources to enhance the quality of services provided. Create an environment in which effective learning can take place.	2 checklist items and 3 rating scales Example: “Establishes rapport with client”
Client-centred care	This skill emphasizes how well you have incorporated the client in the care discussions and decisions. Client-centred care means respecting the needs, opinions, and autonomy of the client.	Respect the autonomy of clients as full partners in decision making. Respect diversity in others to support culturally sensitive and safe services. Design and implement services tailored to the unique needs of individuals. Consider the views of clients about their values, health, and decision making. Work with clients to assess, diagnose, plan, implement, and evaluate services for clients. Prioritize clients’ needs through a collaborative process with clients. Negotiate mutually acceptable individual or program learning plans with clients. Select educational interventions & develop educational materials to meet clients’ learning needs.	5 test items and 1 rating scale Example: “Addresses the client’s chief concern”
Eliciting essential information	This skill emphasizes what you are asking. Eliciting essential information from the client involves using appropriate prompts and follow-up questions to collect all necessary information prior to starting dental hygiene therapy. Comprehensive questioning is needed to identify status and risks of both oral and overall health.	Collect accurate and complete data on the general, oral, and psychosocial health status of clients. Elicit information about the clients’ oral health knowledge, beliefs, attitudes, and skills as part of the educational process. Assess clients’ need to learn specific information or skills to achieve, restore, and maintain oral health and promote overall well-being.	10 checklist items Example: “Asks the client about current medications”
Interpreting findings	This skill emphasizes how accurately you analyse the information revealed by the client. Proper interpretations include making appropriate modifications to the dental hygiene appointment, providing accurate recommendations, and identifying contraindications to care.	Apply principles of risk reduction for client safety, health, and well-being. Evaluate clients’ health and oral health status using determinants of health and risk assessment to make appropriate referral(s) to other health care professionals. Apply theoretical frameworks to the analysis of information to support practice decisions. Apply evidence-based decision making to the analysis of information and current practices. Apply the behavioural, biological, and oral health sciences to dental hygiene practice decisions. Identify clients for whom the initiation or continuation of treatment is contraindicated based on the interpretation of health history and clinical data. Identify clients at risk for medical emergencies. Formulate a dental hygiene diagnosis using problem solving and decision-making skills to synthesize information. Provide recommendations in regard to clients’ ongoing care including referrals when indicated.	8 checklist items Example: “Recognizes the need for a medical consult prior to dental hygiene treatment”

Figure 3.

Sample diagnostic score reporting output, initial page

Figure 4.

Sample diagnostic score reporting output for a diagnostic domain

The first page of the student report showed their overall score on the SCA and their scores for each diagnostic domain (Figure 3). Their overall score was compared to the passing grade of 70%, and domain scores were compared to the performance of their peers, described as below average, average or above average (based on a fixed mathematical formula which placed the “average” category within plus or minus one-half standard deviation around the mean score). This method was chosen because the majority of students pass this assessment and thus having a second comparison marker relative to their peers might encourage students to improve their performance beyond simply striving to pass. However, other comparison methods may be more appropriate depending on the pass/fail rate of the assessment and the objectives of the feedback. For example, focusing on performance indicators relative to the “just passing” group could alert failing students and students at risk of failing that they need to improve in that area.³¹

Students were prompted to click on each of the diagnostic domains to receive more information. Domain-specific pages displayed the definition of that skill, information on how to improve, and the relevant dental hygiene competencies. The domain-specific output for effective communication is presented in Figure 4.

DISCUSSION

SCAs are a common assessment method in health education, capturing detailed information on clinical skills. However, the opportunity to share this information with students as feedback tends to be neglected.¹¹ DSR offers a possible framework for providing feedback to students, describing test performance by the underlying domains of learning the test intends to measure and including resources for making individual-level improvements.^23-26 As such, this research project sought to establish and apply a general framework for providing DSR within the context of an SCA.

Piloting DSR in the dental hygiene program demonstrated how score reports could be generated efficiently and without revealing the actual test items, maintaining test security. Only a moderate initial time investment was required, after which all students received feedback. However, these reports included only the most basic elements of DSR and could be modified to improve and maximize the feedback.

Reports could provide direct links to sections of textbooks or other online documents that may aid student learning.^24,28 Links to video-based feedback could also be provided, either using trained professionals to demonstrate important interactive skills, or the student’s own video performance for self-review.^41,41-43 Further personalization and individualization may also be beneficial. Although the first page of each student’s report differed based on their performance, indicating their strengths and weaknesses, the information provided on the domain pages was the same for all students. A next step may be to have different information displayed on each student’s domain pages, reflecting the test items that were incorrect and, thus, more acutely, where to focus their learning. Having the basic framework in place will allow improvements to be made year after year without placing a significant time burden on the test administrator.

A limitation of this study is that the Delphi technique typically involves a minimum of 7 to 10 expert opinions, which was not feasible given the small number of clinical instructors in our program. Furthermore, following the modifications to the blueprint, ideally a second round of expert ratings should have been conducted. However, weighing instructor availability against the small number of changes made to the blueprint, this second round was forgone. Consequently, this study refers to a “modified” Delphi technique to acknowledge these limitations. Another limitation is that this article does not describe psychometric techniques to ensure domain reliability, focusing more on user-friendly methods to provide valid feedback.

The main strength of this study is the evidence-based premise in developing resources for assessment and reporting of student performance based on competencies of the profession. The reporting process described above was based on a multidisciplinary literature review across several key factors for providing valid feedback. However, for feedback to be of quality, it must be both valid and practically usefully—i.e., it must have some impact on student learning. Our companion paper describes the effect of providing DSR following a dental hygiene SCA.¹

CONCLUSION

DSR offers a promising framework for providing feedback following an SCA in a timely manner while upholding test confidentiality. This article can guide readers to develop diagnostic domains and provide students with validated score reports.

CONFLICT OF INTEREST

Authors declare they have no financial, economic or professional interests that may have influenced the design, execution or presentation of this scholarly work.