Development of a Quality Improvement Dental Chart Review Training Program

Abstract

Introduction:

Chart review is central to understanding adverse events (AEs) in medicine. In this paper, we describe the process and results of educating chart reviewers assigned to evaluate dental AEs.

Methods:

We developed a web-based training program, “Dental Patient Safety Training” which utilizes both independent and consensus-based curricula, for identifying AEs recorded in electronic health records (EHRs) in the dental setting. Training included 1) didactic education, 2) skills training using videos and guided walkthroughs, 3) quizzes with feedback, and 4) hands-on learning exercises. Additionally, novice reviewers were coached weekly during consensus review discussions. TeamExpert was composed of two experienced reviewers, and TeamNovice included two chart reviewers in training. McNemar’s test, interrater reliability, sensitivity, specificity, positive predictive value and negative predictive value were calculated to compare accuracy rates on the identification of charts containing AEs at the start of training and seven months following consensus building discussions between the two teams.

Results:

TeamNovice completed independent and consensus development training. Initial chart reviews were conducted on a shared set of charts (n=51) followed by additional training including consensus building discussions. There was a marked improvement in overall percent agreement, PABAK correlation, and diagnostic measures (Sensitivity, Specificity, PPV, and NPV) of reviewed charts between both teams from the Phase I training program to Phase II consensus building.

Conclusion:

This study detailed the process of training new chart reviewers and evaluating their performance. Our results suggest that standardized training and continuous coaching improves calibration between experts and trained chart reviewers.

INTRODUCTION

Medical adverse events (AEs) are one of the leading causes of death in the US,^{1, 2} and are widely studied.^3–5 Dental AEs have until recently not received much attention except for the few sentinel events that make the news.^{6, 7} To understand and prevent AEs, various methods of collecting and analyzing data around their occurrence have been utilized including voluntary reporting systems,⁸ retrospective chart reviews,^9–11 mining of administrative or claims data,^{12, 13} natural language processing of discharge summaries,¹⁴ and patient interviews and surveys.^{15, 16} Retrospective chart review, in which the medical record is evaluated for the conditions and documentation of an event, is one of the most common and respected methods for identifying AEs.¹⁷ For example, 25% of all scientific investigation studies published in emergency medicine relied on abstracted data from medical charts.¹⁸ In the dental setting, chart reviews unearth mostly unexpected post-surgical pain, hard tissue injury (e.g. tooth perforation) and soft tissue injuries (e.g. lacerations) as AEs.¹⁹ Chart reviews are often combined with other tools such as structured queries for antecedent or trigger events.^20–23 Dental triggers rely mostly on structured data, such as dental procedure codes (CDT), dental diagnosis, and medications taken by the patient. We have found that pain, soft tissue, hard tissue and nerve injuries are the most common types of AEs.¹⁹

A typical two-stage medical chart review involves nurses initially screening the charts followed by two reviewers performing independent reviews. In case of any discrepancy, the supervising reviewer independently reviews the chart.²⁴ In a chart review (with or without a review tool or triggered discovery²³), physicians use their implicit clinical judgment, including their knowledge, skills, experience and a continuous critical analysis of the information contained within patients’ charts to determine the presence of AEs.^{25, 26} Since this is a subjective process, inter-rater reliability between reviewers can vary, with studies finding low to fair (kappa correlation coefficients between 0.39 to 0.60) agreement across individuals.^27–31 Physicians with greater experience in reviewing charts have higher agreement.^{24, 32}

Methods to improve reliability such as operationalizing variables, use of standardized abstraction form, blinding (masking), periodic monitoring and meeting with abstractors,^{18, 33} have been used to increase validity and the quality of data collected from medical charts. However, there is very little description of the methods used to train abstractors or studies on best practices in training abstractors. Gilbert et al.,¹⁸ in their systematic review of emergency medicine studies found that, although 18% of the studies mention “abstractor training” in their papers, the methodologies used were not described. Given the paucity of scientific literature describing the methodology of training chart reviewers, we conducted a study to determine the impact of structured training on chart review for novice abstractors.

We developed the “Dental Patient Safety” training program using a commercial learning management system (LMS) for independent training. This web-based training program was used in conjunction with consensus building discussions to onboard a novice team of reviewers. Our objectives were to 1) develop and test the training for chart reviewers to consistently identify AEs in the electronic health record (EHR); and 2) assess the quality of the reviews by novice reviewers, using the reviews by a group of experts as the gold standard”.

METHODS

The study was reviewed and approved by the Institutional Review Board (IRB). As part of the study, a Patient Safety Toolkit was developed by eight research team members who are clinicians with extensive clinical and chart review experience. Through an iterative process of independent coding and consensus building discussion, the research team developed the definitions and protocols for defining and categorizing AEs.^{34, 35} For the research, we narrowly defined a dental AE as “physical harm associated with dental treatment within a timeframe relevant to the clinical scenario.”

The methods used to identify and characterize dental adverse events have been described earlier.^{21, 34, 36, 37} In short we developed and validated electronic health record (EHR)- based triggers to find potential adverse events (AEs) in dental patients’ EHRs. AEs were next classified in 12 categories (see Box 2). When identifying AEs, reviewers would also characterize AEs using a modified IHI category indicating if severe or mild permanent or temporary harm had occurred. Concordance was calculated between the individual reviewers within their respective teams (TeamExpert vs TeamNovice).

Box 2: Adverse Event: Classification and AEs identified in the study.

#	Adverse Event	Potential AEs Phase (40/233 charts) n
1	Allergy/Toxicity/Foreign Body response	0 (0.0%)
2	Aspiration/Ingestion of Foreign Body	0 (0.0%)
3	Infection	14 (35.0%)
4	Wrong-site, wrong-procedure, wrong-patient	0 (0.0%)
5	Bleeding	2 (5.0%)
6	Pain	12 (30.0%)
7	Hard tissue injury	1 (2.5%)
8	Soft tissue injury	9 (22.5%)
9	Nerve injury	2 (5.0%)
10	Other systemic harm	0 (0.0%)
11	Other oro-facial harm	0 (0.0%)
12	Other harm	0 (0.0%)
	Severity of Event
	E1	11 (27.5%)
	E2	28 (70.0%)
	G1	1 (2.5%)
	G2	0 (0.0%)

Chart Reviewers

Four chart reviewers who are general dentists and dental public health professionals were split into two teams depending on their expertise; (TeamExpert (>3 years chart review experience) and TeamNovice limited or no chart review experience). TeamExpert members consisted of highly experienced dentists and well-calibrated chart reviewers. TeamNovice members were general dentists and had no prior experience in chart reviews.

Chart Reviewer Training

The training program “Dental Patient Safety” was designed and deployed on a commercial LMS. The training program is comprised of eight modules [see Box 1] that include definitions of patient safety, AEs, contributing factors, degree of harm, detection and error,^{38, 39} WHO’s International Conceptual Patient Safety Framework,⁴⁰ and AHRQ’s Patient Safety Network.⁴¹ Videos and guided walkthroughs honed the skills of chart reviewers in detecting and documenting AEs. At the end of each training component, self-assessments were completed, and concrete feedback was provided to each participant with detailed explanations to help shape their understanding. Lastly, participants were asked to complete six standardized cases and received feedback on their performance for each of those cases. Supporting materials included a standard operating procedure manual as reminders for the process of chart reviews, information regarding the data collection form, coding instruction, and a quick reference guide on AE definition, dental AE classification (Box 2) and dental AE severity rating scale (Figure 1). In addition to the independent training, all four reviewers participated in weekly meetings. At these meetings a select set of AEs that were previously distributed were discussed to further develop a shared mental model regarding what is considered an AE in dentistry.⁴²

Box 1: Dental Patient Safety Training Modules.

1. Patient Safety – Overview of Patient Safety, Definition of Patient Safety, WHO International Conceptual Patient Safety, Key Concepts from the WHO Patient Safety Curriculum (e.g., definitions of “adverse events”, “contributing factor”, “degree of harm”, “detection” and “error”), AHRQ Patient Safety Network and Quizzes

2. Adverse Events (AEs) in Dentistry – Overview of AEs in Dentistry, Definition of Dental AEs, Example of AEs and Non-AEs, Quality of Care Issues vs. AEs, Trigger Definition, Trigger List, Trigger Description and Logic and Quiz

3. Training: Scoring for Severity – AE Classification, AE Classification Examples, AE Severity, AE Severity Examples and Quiz

4. Navigating REDCap for chart reviews – Orientation of REDCap and EHR, Overview of Chart Review Process Overview and lastly, Quizzes on EHR and REDCap

5. Training: Identifying AEs within Charts – Video Demonstration of Identifying AEs within Charts and Calibration Cases

6. Training: Bringing it All Together- Documents of Standardized Calibration case on REDCap, REDCap Reviewer Guide and Quizzes Related to the Calibration Case

7. Self-Assessment for Chart Review - Additional Standardized Calibration Cases and Manual of Standard Operating Procedure for AEs, Study Protocols, Key Areas of Data Collection Forms and Coding Instructions.

8. Calibration: Reliability to Gold Standard – Brief Assessment of Calibration Cases and Results Comparison with Gold Standard Review Results.

Figure 1:

Dental Adverse Event Severity Scale (modified from Institute for Healthcare Improvement)

E1: Temporary (reversible or transient) minimal/mild harm to the patient (healed or resolved without permanent defect or disability)

E2: Temporary moderate to severe harm to the patient

F: Patient transferred to emergency room and/or hospital

G1: Permanent minimal/mild patient harm (healed with permanent defect or disability)

G2: Permanent moderate to severe patient harm

H: Intervention required to sustain life

I: Patient death

Comparisons between the performance of TeamNovice and TeamExpert were carried out in two stages. The first phase of the assessment occurred after the independent training using the LMS. In this phase, Phase I, TeamNovice independently reviewed and documented AEs from 51 randomly chosen charts. They discussed their findings among themselves to consolidate the final list of AEs. At the same time, TeamExpert audited and followed the similar process for these charts. Results from TeamExpert were defined as the gold standard and results of TeamNovice were compared against it. Both teams had multiple face-to-face meetings where the final list of AEs were discussed. No modifications were made to the AE list following these meetings. The second stage of assessment, Phase II, was conducted after seven months of consensus building discussions following training on LMS. Again, both teams independently reviewed 233 specific charts identified by the automated triggers and documented the resulting AEs. As with earlier reviews, each team member first reviewed the charts independently and then again in consultation with their team member. Here we compare the performance of the teams.

Analysis

A descriptive analysis was performed to determine the total number of charts reviewed and the total number adverse events identified. Information on the type and number of adverse events is found in Box 2. Please note that Box 2 identifies all potential AEs identified by all reviewers, before concensus was reached if the identified AE was considered an AE. In addition, the frequency and percent agreement for the type and severity of each adverse event was calculated. In order to determine the percent agreement between the expert reviewers considered to be the gold standard and the novice reviewers, both diagnostic measures and correlation coefficients were computed. The diagnostic measures used were sensitivity, specificity, positive predictive value (PPV), and negative predictive values (NPV) and correlation coefficient computed was the prevalence and bias-adjusted kappa (PABAK) (κ).⁴³ PABAK was selected, as our prevalence index was very high as compared to the bias index. Lastly, all analyses were performed using R (R Version 4.0.2) statistical software.⁴⁴

RESULTS

In Phase 1, TeamExpert and TeamNovice independently reviewed the same 51 patient charts for adverse events. Table 1 shows the resulting confusion matrix detailing the areas of concordance and discordance. Of the 51 charts reviewed, concordance was found in 39 patient charts for an overall percent agreement of 76.5%. There were 12 (23.5%) patient charts where there was discordance between the experts and novice reviewers. The PABAK correlation coefficient was 52.9% (PABAK = 52.9%, 95%CI = [25.0–74.4]) representing “moderate” agreement according to Landis and Koch’s.⁴⁵ The number of adverse events identified by TeamExpert that were also identified by TeamNovice reveals a true positive rate of 37.5% (sensitivity = 37.5%, 95%CI = [8.5–75.5]). The number of patient charts determined to not “be adverse events” identified by TeamExpert that were also classified as “not adverse events” by TeamNovice yielded a true negative rate of 83.7% (specificity = 83.7%, 95%CI = [69.3–93.2]). The positive predictive value was 30.0% (PPV = 30.0%, 95%CI = [6.7–65.2]) and the negative predicted value was 87.8% (NPV = 87.8%, 95%CI = [73.8–95.9]).

Table 1:

Matrix for concordance and discordance pilot chart review (N=51)

TeamNovice (Trained)	TeamExpert (Experts) - Gold standard
		Charts with AE	Charts without AE
	Charts with AE	3	7
	Charts without AE	5	36

In Phase II, TeamNovice and TeamExpert each reviewed 233 total patient charts for adverse events. Table 2 shows the confusion matrix for the concordant and discordant reviews. The overall percent agreement between the TeamNovice and TeamExpert was 80.7% and the PABAK correlation coefficient was 61.4% (PABAK = 63.9%, 95%CI = [50.0–71.1]) representing a “substantial” agreement. The total positivity rate was 71.4% (sensitivity = 73.2%95%CI = [57.7–82.7]), the total negative rate was 83.6% (specificity = 83.6%, 95%CI = [77.3–88.7]), the PPV was 50.6% (PPV = 50.6%, 95%CI = [39.1–62.1]) and NPV was 90.2% (NPV = 90.2%, 95%CI = [84.6–94.3]).

Table 2:

Matrix for concordance and discordance phase II chart review (N=249)

TeamNovice (Trained)	TeamExpert (Experts) - Gold standard
		Charts with AE	Charts without AE
	Charts with AE	40	29
	Charts without AE	16	148 + 16(NA) = 164

Tables 3 and 4 compares the agreement between TeamNovice and TeamExpert classification and severity ratings of adverse events. Among the 40 agreed upon AEs (see Box 2), the reviewers in TeamNovice and TeamExpert were in full agreement with the AE classification 57.5% (23 charts) of the time, in at least partial agreement 77.5% of the time, and full disagreement 22.5% of the time. Additionally, Table 4 shows that among the 40 agreed upon AEs, there was full agreement between the reviewers in TeamNovice and TeamExpert on 29 (72.5%) of the dental patient charts with AEs while 11(27.5%) charts had disagreement.

Table 3:

Comparison of AE Category Classification between TeamExperts and TeamNovice

	Charts with AEs (n=40) [N, %]
No match between TeamExperts and TeamNovice (DIFFERENT)	9 (22.5%)
Both teams used Identical AE category (SAME)	23 (57.5%)
There was partial agreement	8 (20.0%)

Table 4:

Comparison of AE Severity Rating between TeamExperts and TeamNovice

	Charts with AEs (n=40) [N, %]
Both teams rated identically AE severity rating	29 (72.5%)
Both teams had different severity ratings	11 (27.5%)

There was a marked improvement in overall percent agreement, PABAK correlation, and diagnostic measures (Sensitivity, Specificity, PPV, and NPV) of reviewed charts between both teams from the Phase I training program to Phase II consensus building. There was no clear pattern related to the areas of discordance among the categories of AEs. However, we noted that TeamNovice and TeamExperts sometimes had a different understanding of adverse events that could be expected after a treatment, such as denture sores, or if the event occurred during a relevant clinical timeframe.

DISCUSSION

For this project, we used two previously developed trigger tools to facilitate the finding of dental AEs. Specifically, we used the Institute for Healthcare Improvement (IHI) trigger tool, which was developed to effectively help identify AEs in the clinical setting.⁴⁶ Originally developed for the inpatient setting, it has been evolved for the outpatient setting⁴⁷ and, recently, for the dental clinical arena.²¹ Triggers do not in themselves represent AEs; rather “triggered charts” are more likely to document an AE. Hence using triggered charts has proven to be more efficient for detecting AEs than conducting random chart reviews.⁴⁸ Using triggers to find AEs is the first step of quality improvement,⁶ as it identifies harm to patients. Once we identify any harm, only then can we measure it, analyze it, and explore what underlying systems need to be addressed to prevent such harm from occurring again.

As we have noted in previous work, the perceptions of leadership around how well patient safety is managed may be quite different than the perceptions from dental clinic staff.⁷ Making effective changes in underlying systems to diminish patient harm has to start with understanding the clinic’s current culture around patient safety.⁷ Using triggers to conduct targeted systematic chart review on a regular basis to unearth AEs would be a significant sea change for dentistry. However, as our colleagues in medicine have also discovered, it is an effective beginning towards the development of learning organizations,^{49, 50} and we hope of a “learning profession”.

Published case reports provide a window into understanding the nature and extent of dental AEs However, these siloed and incomplete contributions to dentistry’s understanding of AEs in the dental office are not enough to fully understand all threats to dental patients’ safety.^{51, 52} More complete data around patient harm will help inform individual providers, entire clinics, and the profession about underlying systemic issues that need reform. Patient records are valuable data sources that can help identify AEs. Traditionally, a random sample of health records were selected for audit. Classen et al., however, found that a focused chart review identifies more AEs than a random chart review,⁴⁸ and detecting AEs automatically in EHRs greatly facilitates this work.^53–55 However, it is important to realize that our chart review process does not allow us to discern what the underlying cause is of the harm that was caused. There are many reasons for AEs to occur, including diagnostic failure,⁵² inexperience, or case complexity and there is indeed still a lot to learn in the dental arena about why and how AEs happen.^{6, 21, 35, 56} In future work we will conduct an analysis of the AEs and determine contributing factors.

As dentistry enters this realm of quality improvement/patient safety, we envision clinics will run a few specific triggers against their EHR to identify specific patient safety care issues. Dental clinics may encounter turnover in their chart reviewers just as they start feeling comfortable with the process. New chart reviewers will lack the historical knowledge and consensus experience of the original chart reviewers. Collecting accurate and consistent data from retrospective chart reviews is challenging for any chart reviewer, especially when multiple chart reviewers are involved despite having standardized protocols and data collection forms. We believe that our Dental Patient Safety Training will facilitate onboarding of new team members.

Our training yielded successful results, which can be attributed to the combination of reading materials, video demonstration of how to detect AEs, hands-on learning exercises and a unique interactive approach. The reading material supplied TeamNovice with cognitive knowledge, while videos, hand-on learning exercises and interactive quizzes with feedback provided the skills needed to use this knowledge. Another explanation of training effectiveness may be due to the practice participants received.⁵⁷ Attending weekly conference calls also enriched team members by shifting their decision-making process from individual to a collaborative team-based approach through a shared mental model.^{42, 58}

Limitations of this study include the fact that the Dental Patient Safety Training program was developed and tested at one academic institution, using one EHR. Hence, results may not be easily generalizable to non-academic dental practice sites and dental practices that use a different EHR. We only measured training around potential AEs as identified by two triggers. We have developed a number of other validated EHR-based triggers^{37, 59} and also have conducted unstructured reviews using a random sample of charts.³⁶ Additionally, we are starting to understand the importance of the voice of the patient in patient safety measurement.^{60, 61} We acknowledge that the total sample size of reviewers is small and that training included both an online component and consensus building meetings. As such it is difficult to determine the impact of each component separately.

We conclude that it is critical to develop standardized training approaches for calibrating chart reviewers to increase the reliability, validity and quality of collected data as one of the first important steps towards improving patient safety in the dental setting.

CONCLUSION

We developed a web-based dental patient safety training program to train inexperienced chart reviewers. Standardized training with continuous coaching appears to be an effective way to reach calibration between experienced and new chart reviewers.