Measuring Up: Implementing a Dental Quality Measure in the Electronic Health Record Context

Abstract

Background

Quality improvement requires quality measures that are validly implementable. In this work, we assessed the feasibility and performance of an automated electronic Meaningful Use dental clinical quality measure (percentage of children who received fluoride varnish).

Methods

We defined how to implement the automated measure queries in a dental electronic health record (EHR). Within records identified through automated query, we manually reviewed a subsample to assess the performance of the query.

Results

The automated query found 71.0% of patients to have had fluoride varnish compared to 77.6% found using the manual chart review. The automated quality measure performance was 90.5% sensitivity, 90.8% specificity, 96.9% positive predictive value, and 75.2% negative predictive value.

Conclusions

Our findings support the feasibility of automated dental quality measure queries in the context of sufficient structured data. Information noted only in the free text rather than in structured data would require natural language processing approaches to effectively query.

Practical Implications

To participate in self-directed quality improvement, dental clinicians must embrace the accountability era. Commitment to quality will require enhanced documentation in order to support near-term automated calculation of quality measures.

Introduction

“Self-evaluation will ensure that dentistry as a profession can provide evidence to the community at large that its members are responsible stewards of oral health.”¹ Dentistry has made strides in developing dental quality measures in the wake of increasing EHR adoption and enhanced focus on accountability in healthcare. This is a critical, and perhaps overdue², step forward. According to President Clinton's Advisory Commission on Consumer Protection and Quality in the Health Care Industry (1998), “A key element of improving health care quality is the nation's ability to measure the quality and provide easily understood, comparable information on the performance of the industry.”³

In the years since 1998, the clinical quality measure ecosystem has evolved, and the evolution has accelerated with the uptake of electronic health records (EHRs). The Medicare and Medicaid EHR Incentive Programs that are part of the HITECH (Health Information Technology for Economic and Clinical Health) Act provide financial incentives for the “meaningful use” of certified EHR technology. An eligibility criterion for the incentives is the use of the EHR to report clinical quality measures.⁴ (Please see Box 1 for facts about the Meaningful Use program.) Meaningful Use Stage 2 was the first to incorporate oral health measures. They were developed by the Centers for Medicare and Medicaid Services (CMS) through a contract with Booz Allen Hamilton.⁵ In parallel, the Dental Quality Alliance (DQA) has also been defining electronic clinical quality measures. In response to a request from CMS, the American Dental Association established the DQA to frame oral health care quality measures using a consensus-building process among a broad base of stakeholders.⁶ For Meaningful Use Stage 3, the DQA and Meaningful Use efforts may intersect, as the DQA is poised to contribute to Meaningful Use clinical quality measures.⁷

Before the advent of EHRs, quality measures were gathered through administrative claims data, which are proprietary to the payers and are aggregated only at that level.¹ The Meaningful Use program has the potential to transform EHRs into a catalyst for what the Institute of Medicine has called a learning healthcare system, which is defined as a “healthcare system designed to generate and apply the best evidence for the collaborative healthcare choices of each patient and provider; to drive the process of discovery as a natural outgrowth of patient care; and to ensure innovation, quality, safety, and value in health care.”⁸ The learning healthcare system has implications for quality improvement, as well as clinical research and the analysis of the comparative effectiveness of different treatments.⁹ Quality measurement is fundamental to quality improvement, and successful implementation of quality measures depends on the availability of timely, accurate, and reliable data sources. For this reason, we believed it was an opportune moment to implement one of these Meaningful Use measures in a real-world EHR setting in order to evaluate the feasibility and performance of this measure. In particular, the objectives were to (a) evaluate whether the measure could be implemented on the basis of structured data available in the record and (b) to compare performance of an EHR-based query against the findings derived from an in-depth manual chart review.

Methods

We assessed the feasibility and performance of Meaningful Use Stage 2 measure CMS74v3¹⁰,the percentage of children, age 0-20 years, who received a fluoride varnish application. The site chosen was the University of Texas School of Dentistry at Houston. Representatives from different dental institutions (authors) met to frame the exact query definition by reviewing the measure specifications and determining how they may be retrieved from the EHR. The authors represent a range of areas of expertise, as follows: (1) A.B. – dental public health; (2) R.R. – dental public health; (3) E.K. – oral surgery, health policy and management, data standards; (4) A.N. – dental public health; (5) N.B.H. – dental public health; (6) J.M.W. – general dentistry, caries risk management; (7) L.M. – general dentistry; (8) M.F.W. – informatics. Each of the institutions represented uses the axiUm dental EHR. Human subjects approval was received before conducting the retrospective chart reviews. In particular, we followed the procedure delineated below:

Step 1: Total patient population that meets the denominator criteria

We determined the total number of eligible patients who had an encounter (visit) for oral evaluation in the 2013 measurement period by querying the dental EHR. We used evaluation-related CDT¹¹/Treatment codes to determine an encounter as shown in Table 1 for patients who were 20 years old or less before the start of the measurement year.

Table 1.

Procedure codes that were used to query the EHR as part of the fluoride varnish quality measure

CDT Procedure Codes queried in the denominator to determine an encounter

D0120: Periodic oral evaluation

D0140: Limited oral evaluation

D0145: Oral evaluation for a patient under three years of age and counseling with primary caregiver

D0150: Comprehensive oral evaluation

D0160: Detailed and extensive oral evaluation-problem focused

D0170: Re-evaluation-limited

D0180: Comprehensive periodontal evaluation

CDT Procedure Codes queried in the numerator to determine fluoride application

D1203: Topical application of fluoride-child

D1204: Topical application of fluoride-adult

D1206: Topical fluoride varnish

D1208: Topical application of fluoride

Step 2: Sample size necessary for manual review

We determined the sample size requirement using precision of 5% around the estimate at 95% confidence. We obtained data from the State of Texas Medicaid database, estimating fluoride varnish application rate at 44.27% (Yang N, Personal Communication).

Step 3: Population within the sample that meets the numerator criteria through EHR query

We determined the total number of eligible patients who received a fluoride varnish application in the measurement period by querying the set of patients identified in Step 1 (denominator). We used fluoride related CDT/Treatment codes to determine fluoride application (see Table 1).

Step 4: Population within the sample that meets the numerator criteria through manual chart review

Two abstractors (authors AB and NH) conducted independent manual review on sample size determined in step 2 and came to consensus where there were differences. Discrepancies were first identified, after which the reviewers referred back to the definitions of the numerator and denominator and discussed the discrepancies. All differences were able to be resolved in this way. Inter-rater reliability was assessed using the Kappa statistic.

Step 5: Concordance between query and manual review

We compared the performance of the dental EHR query to an in-depth manual chart review, which served as the gold standard with respect to information contained within the record. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Results

We describe the results of each step, outlined in the methods, for creating and evaluating the dental quality measure.

Step 1: Denominator

4376 patient charts met the requisite criteria. The average age of these patients was 8.81 years (±4.80), and 51.2% were male. At the time period covered in this review, the site treated Hispanic as one of the categories of race versus an ethnicity that is separately categorized. Thus, the population was categorized as 50.3% Hispanic, 15.8% African American, 14.3% Caucasian, 4.4% Asian = 4.4, 0.4% American Indian/Alaskan, 1.8% other, and 13.0% Unknown/unspecified/blank. Since the time of this review, race and ethnicity have been separately recorded.

Step 2: Manual review sample size

Sample size calculations determined a minimum of 381 charts to have a precision of 5% around the estimate at 95% confidence. Conservatively, we randomly selected 500 charts for review from the denominator records using a random number generator. The average age of these patients was 9.05 years (±4.90) and 50.2% were male. The race/ethnicity distribution was as follows: 51.8% Hispanic, 13.2% African American, 12.6%Caucasian, 5.3% Asian, 0.2% American Indian/Alaskan, 2.0% other, and 15.0% unknown/unspecified/blank. Ten were excluded from the manual review because they belonged to test (i.e., not real) patients. We did not exclude them from the automated review, as they would have been captured in a real-world automated review.

Step 3: Numerator

Among the 4376 denominator charts, 3209 met the numerator criteria (having had a fluoride varnish) through the dental EHR query.

Step 4: Identify numerator through manual chart review of sampled records

The two chart reviewers had excellent inter-rater agreement with respect to the denominator (Kappa=0.907). Abstractors also had excellent inter-rater reliability (K=0.984) determining if fluoride varnish was applied (numerator). Manual chart review indicated fluoride varnish application in 380 of the 490 charts.

Step 5: Concordance between query and manual review

Using the dental EHR-based query 355/500 (71.0%, 95% CI: 67.2% to 75.2%) of patients were found to have had fluoride varnish compared to 380/490 (77.6%, 95% CI: 73.9% to 81.2%) using the manual chart review. Table 2 shows how the automated EHR query performed compared to the manual chart review.

Table 2.

Results showing performance of the EHR based query against the manual chart review (which served as the gold standard) for determining patients who received fluoride varnish

Sensitivity	90.5% (95% CI: 87.1% - 93.3%) of the time that manual review revealed fluoride varnish application, the query identified fluoride varnish application
Specificity	90.8% (95% CI: 84.2% - 95.3%) of the time that the manual review did not reveal fluoride varnish application, the query did not identify fluoride varnish application
Positive Predictive Value (PPV)	96.9% (95% CI: 94.5% - 98.4%) of the time that the query identified fluoride varnish application, the manual review revealed fluoride varnish application
Negative Predictive Value (NPV)	75.6% (95% CI: 67.3% - 82.0%) of the time that the query did not identify fluoride varnish application, the manual review did not reveal fluoride varnish application.

The procedure code for fluoride varnish was sometimes not entered and therefore not identifiable by the automated query, which was based on structured data. We found that while the fluoride varnish procedure code was documented 90.3% of the time as structured data, in 9.7% of the cases the application of varnish was only noted in a free text clinical note. Reviewers also found the following reasons documented in the note why fluoride varnish was not applied: 1) fluoride was provided earlier during a visit with another dental clinic, 2) patient had a heart murmur and the dentist wanted to contact the cardiologist first, 3) patient was transferred to another clinic after been accepted to the Children's Health Insurance Program (CHIP), and 4) patient's mother denied fluoride application.

Discussion

Our preliminary results following the implementation of a single quality measure in the EHR are promising and show the feasibility of using EHR data to determine dental quality measures. We felt that the time was right to focus on this, as the momentum builds for widespread EHR adoption in dental practices.¹² Indeed, some states now require dental practices to implement EHRs. ¹³

We chose to implement a Meaningful Use measure that relies upon data that is collected in a standardized way across sites and EHR installations. A measure that requires data that not all sites collect or that are collected in different ways at different sites would be more challenging to implement broadly. In the course of our work, we identified quality measure definition best practices that should be followed as dentistry expands its quality measure set. First, the population defined in the denominator should be appropriate to the numerator being assessed. In the measure we implemented, we followed the Meaningful Use specification and therefore did not exclude those patients who were seen only for an emergency visit, though fluoride varnish would be unlikely to be generally appropriate in the emergency setting. Likewise, we did not explicitly exclude any young infants (< 6 months of age), though they would be unlikely to have teeth available for fluoride varnish application. Second, measures should be unambiguously definable. In the case of the measure we assessed, the denominator is specified as “Children, age 0-20 years, with a visit during the measurement period.”¹⁰ In our cross-site discussions, we noted that the term visit is ambiguous because it is not clear, for instance, whether it applies to onetime visits such as a single focused periodontal evaluation or emergency visits. Ambiguity in measure definition leads to cross-site variation in implementation, which inhibits interpretation and comparison.

A great promise of oral healthcare quality measures is the ability to compare and share data across sites. When these data are transformed into knowledge, we establish the foundation of a learning healthcare system. In the case of quality improvement, quality measures are the vehicle by which raw data are first transformed into actionable knowledge. In addition to the importance of consistent specification, there is a need to aggregate measures across sites for shared learning to occur. For the most part, aggregation currently occurs only with administrative claims data at the payor level, as noted by the DQA.¹ This status quo is being challenged by initiatives to aggregate clinical data, such as the BigMouth dental data repository, which authors on this paper have developed. BigMouth thus far is amassing data extracted from six dental schools’ EHRs.¹⁴ One of our goals for BigMouth is that it will either (1) allow the calculation of quality measures based upon raw data deposited into BigMouth or (2) be a repository for quality measures that have been pre-processed at the originating clinical site, likely via something akin to the Health Quality Measures Format. The Health Quality Measures Format is a standard for representing a health quality measure as an electronic document.¹⁵

One of the challenges often mentioned in the framing of dental clinical quality measures is the absence of a standardized dental diagnostic terminology.^{1, 16, 17} The lack of such a terminology has limited a number of bodies, including DQA, in terms of the types of measures that can be proposed; e.g., one DQA proposed measure is “Percentage of adults treated for periodontitis who received comprehensive oral evaluation ....”¹⁸ If a patient has been diagnosed with periodontitis but has not received treatment, s/he would not be captured by such a measure, creating a quality blind spot. In response to this need, our research group has established the DDS Dental Diagnostic System (formerly called EZCodes), which has been designed for use within EHRs in the dental setting.^19-22 The DDS is freely available and is currently installed at 16 academic institutions and several large dental group practices in the United States, Caribbean and Latin America, and Europe. The DDS has been designed to serve as a clinician-friendly interface to rich reference terminologies like, SNOMED, into which SNODENT²³ has been integrated. Broader use of standardized diagnostic terminologies will give the profession and public a more complete picture of dental healthcare quality. With that said, our own experience with dental diagnostic terms confirms that there may be some documentation gaps.²¹ This serves as a reminder that quality measures based upon secondary data analysis represent what was documented, which does not always align with what occurred during the clinical visit.

Clearly, the solution is not to forgo clinical data-based quality measures. We instead must encourage, and sometimes enforce, the entry of important structured data. More sophisticated approaches would included natural language processing²⁴ to identify measure elements from free text notes. We should not, however, make light of the work that will be required to accomplish this. Beyond the simplest rule-based NLP, e.g. looking for the phrase “fluoride varnish”, NLP is analytically complex and requires access to large and varied sources of real-world free-text data.²⁵ In addition to NLP, there also exists the potential to infer missing data from elements that are present.²⁶ Ultimately, clinicians should benefit from their documentation efforts, though additional appointment time might be required to complete the thorough documentation needed to correctly and completely capture the data necessary. Through measurement and the sharing of lessons-learned we can work together to live up to the high standards to which the dental profession and our patients hold us.

Box 1. Meaningful Use Facts.

Meaningful Use Facts

Meaningful Use is a component of the Health Information Technology for Economic and Clinical Health (HITECH) Act, which was itself a component of the 2009 American Recovery and Reinvestment Act.

The HITECH programs have been led by the Centers for Medicare & Medicaid Services and the Office of the National Coordinator for Health Information Technology.

Meaningful Use is defined by the use of certified EHR technology in order to meet defined objectives and quality measures.

Meaningful Use has been rolled out in three stages. The final rules for Stage 1 were introduced in July 2010, and the final rules for Stage 2 were introduced in August 2012. The interim rules for Stage 3 were released in March 2015.

There are incentives for eligible professionals to adopt certified EHRs and to demonstrate Meaningful Use.

If an eligible professional did not begin participation by 2015, there would be negative adjustments to Medicare payments.

Few dental EHRs are certified EHRs.

Oral health quality measures were introduced in Meaningful Use Stage 2.