Quality measures for dental care: A systematic review

Abstract

Objectives

This systematic review aimed to (a) provide an overview of existing quality measures in the field of oral health care, and to (b) evaluate the scientific soundness and applicability of these quality measures.

Methods

A systematic search was conducted in three electronic databases MEDLINE (via PubMed), EMBASE (via OVID) and LILACS (via BIREME). The search was restricted to articles published between 2002 and 2018. Publications reporting on the development process or clinimetric properties of oral health care quality measures for outpatient oral health care in dental practices were included. The identified publications reporting on oral health care quality measures were critically appraised with the Appraisal of Indicators through Research and Evaluation 2.0 (AIRE 2.0) instrument to evaluate the soundness and applicability of the measures.

Results

The search strategy resulted in 2541 unique and potentially relevant articles. In total, 24 publications were included yielding 215 quality measures. The critical appraisal showed a large variation in the quality of the included publications (AIRE scores ranging from 38 to 78 out of 80 possible points). The majority of measures (n = 71) referred to treatment and preventive services. Comparably, few measures referred to the domain patient safety (n = 3). The development process of measures often exhibited a lack of involvement of patients and dental professionals. Few projects reported on the validity (n = 2) and reliability (n = 3) of the measures. Four projects piloted the measures for implementation in practice.

Conclusions

This systematic review provides an overview of the status quo with respect to existing quality measures in oral health care. Potential opportunities include the piloting and testing of quality measures and the establishment of suitable information systems that allow the provision of transparent routine feedback on the quality of oral health care.

1. INTRODUCTION

Oral diseases are highly prevalent and expensive to treat. Against the background of increasing cost pressures in health care, careful choices about the use of available resources are becoming increasingly relevant.1, 2, 3, 4 Robust and comprehensive measures that collect routine data on the processes and outcomes of oral care health care may contribute to a more transparent, evidence‐informed and person‐centred care system.5 These measures need to be transparent and should reflect health processes, outcomes, person/patient perception and costs that are associated with oral health care.6, 7 Measuring the quality of oral care using valid and reliable measures may enable various stakeholders, such as policymakers and dentists, to evaluate and improve the quality of care.8

Several conceptual frameworks exist to define quality of care. The National Academy of Medicine (NAM) defined quality of health care as “the degree to which health services for populations and individuals increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”9 Although oral health care quality measures have been developed over the past couple of years, the NAM highlighted that the lack of quality measures in dentistry was a barrier to quality improvement in oral health care.10 Because of the lack of measures and routinely collected data, dentists and policymakers are currently unaware to what extent delivered oral care is consistent with the best available evidence and whether it satisfies the needs of their patients. Comparative oral health care data may illustrate where further development of care is needed and whether it aligns with the best evidence. Various initiatives for measuring oral health care quality and its determinants have recently been emerging, highlighting room for improvement with respect to the establishment of comprehensive quality measures in dental care.11, 12, 13

The minimum prerequisite for a quality measure is that it is based on scientific evidence, accepted by experts in the field and measured using reliable data sources.14 A reliable measure should be free of measurement errors. To the maximum extent possible, variation in the quality measure should be due to actual differences in the respective population. Another important aspect is the content validity of a measure; that is, the measure is underpinned by scientific evidence and adequately reflects what it intends to measure.15, 16 The better the scientific evidence on which a measure is based, the better the measure reflects a truly important aspect of the quality of care provided.16 Moreover, the acceptance of a quality measure by experts in the field is necessary to minimize disagreement on interpretation of the evidence.14 This can be defined as face validity: a measure has face validity when consensus is reached among experts and the measure accurately reflects the content it intends to measure.14, 15, 16 To use the measures in practice, unambiguous descriptions of numerators and denominators as well as instructions for use are imperative.

The purpose of this systematic review was (a) to provide an overview of the number and type of existing quality measures in the field of oral health care and (b) to appraise the scientific soundness and applicability of quality measures developed to date.

2. METHODS

2.1. Data sources and searches

A systematic search of the electronic databases MEDLINE (via PubMed), EMBASE (via OVID) and LILACS (via BIREME) was performed. To develop a preliminary search strategy, various combinations of search terms were used to identify relevant articles that reported on the development and clinimetric properties (such as validity and reliability) of quality measures for oral health care. Five relevant articles were identified in MEDLINE and used to develop the final search strategy. Based on the keywords and MeSH terms in the previously identified articles, a final search strategy was developed which also captured all identified articles. The detailed search strategy is presented in Appendix S1. Reference lists of the included publications were screened to identify other potential relevant documents such as supplemental quality measure catalogues, instructions and other relevant publications.

2.2. Eligibility criteria

The search was restricted to articles published from 1 January 2002 to 31 December 2017. The decision to restrict the publication year was made in order to exclude quality measures which are based on outdated scientific evidence. There was no language restriction for full‐text articles as long as they had a title, abstract and description of the quality measures in English. Publications that either described the development process or described the clinimetric properties of oral health care quality measures for general dental care were included. Publications were only included if numerators and denominators of the quality measures were defined or if the numerators and denominators could be directly derived from the description of the quality measures. Editorials, randomized controlled trials, conference abstracts and letters to the editor were excluded.

2.3. Article selection and data extraction

Two researchers (AR and GS) independently screened the titles and abstracts. There were a couple cases of nonconsensus between the researchers. In case of discrepancies, the researchers discussed the reason for the discrepancy until consensus was reached. The full text of the potentially relevant articles was reviewed. Two researchers (AR and DD) computed the data using a digital form. Information on methodological aspects of the study such as the purpose of the study, the country of origin, methods used to develop measures and stakeholder involvement in the development process of the measures was included in the form. Furthermore, data assemblance included the number of quality measures developed, the description, numerators and denominators of the measures and the type of quality measure as described by Donabedian17: process‐, structure‐ or outcome measure. In addition, information on the clinimetric properties of the measures was collected.

2.4. Critical appraisal

The Appraisal of Indicators through Research and Evaluation (AIRE) instrument 2.0 was used to appraise the scientific soundness and applicability of the measures.18 The AIRE instrument 2.0 is a validated instrument to assess the methodological quality of the measures. The AIRE instrument 2.0 contains 20 criteria divided into four domains: (a) purpose, relevance and organizational context; (b) stakeholder involvement; (c) scientific methods; and (d) additional evidence, formulation and usage. Each individual AIRE item is scored on a four‐point Likert scale ranging from 1 (strongly disagree) to 4 (strongly agree). The AIRE instrument provides a summary score for assessment of articles, ranging from a minimum score 20 (low rating), to 80 (high rating); see Appendix S2 for further details about the AIRE instrument and its scoring system. Two researchers (AR and DD) independently appraised all publications. Discrepancies were discussed until consensus was reached. The AIRE instrument did not include any items regarding the funding body of the organization developing the quality measures or about competing interest of the authors. Therefore, an additional table was added to provide an oversight of this information (see Appendix S3).

2.5. Calibration of reviewers

The interrater reliability between the two researchers was assessed by comparing the individual scores per AIRE item on two separate articles in which quality measures were developed and by calculating the weighted Cohen's Kappa.19, 20 The interrater reliability amounted to 0.91 (see Appendix S4).

2.6. Data reporting

Based on the results, two authors (AR and DD) categorized all identified quality measures into eight domains: access of care, oral treatment or preventive services, cost of care, disease outcomes, patient experience, public health, health behaviour and organizational aspects of care (see Appendix S5). The domains were established by consensus among the authors and informed by domains described in the included publications. Discrepancies in categorization were discussed among the authors until consensus was reached.

This study is reported in accordance with the PRISMA statement.21

3. RESULTS

3.1. Number and characteristics of publications

Figure 1 shows the flow chart for the selection of articles. The search strategy resulted in 2541 unique and potentially relevant publications (MEDLINE via PubMed: 2005; EMBASE via OVID: 1589; LILACS via BIREME: 28). Based on the title and abstract screening, a total of 110 full‐text publications were reviewed. Sixteen publications met the inclusion criteria. All other publications were excluded because either the numerator/denominator of the respective quality measure was missing or the description of the respective quality measure could not be easily derived. A list of all excluded publications after full‐text assessment can be found in Appendix S6. If concrete quality measures were mentioned, but not clearly described, the authors were contacted by email and requested to provide further information (n = 4). This did not result in additional publications. Eight publications were identified by reference checking. These publications included supplemental quality measure guidebooks and catalogues. Six projects consisted of multiple publications, resulting in a total of 24 included publications (see Figure 1). In total, 12 projects (24 publications) met the inclusion criteria. Ten projects focused on the development process of oral health quality measures, and two publications described additional clinimetric properties of quality measures.

Figure 1.

Flow chart of article selection [Colour figure can be viewed at wileyonlinelibrary.com]

3.2. Aim and methods of included publications

Table 1 provides a description of aims and methods of the included publications. For the development of the quality measures, four publications used a RAND‐modified Delphi procedure, combined with a literature review and/or a World Café method. In addition, three publications used a literature review combined with an expert panel or advisory board and two publications developed quality measures solely based on the opinions of a working group. In one publication, the measures were developed by the authors of the study. Five of the publications were from the United States, two projects (four publications) were funded by the European Commission and the remaining four publications were from individual countries or smaller regions in Europe. Publications meeting the inclusion criteria could not be identified for other regions of the world.

Table 1.

Characteristics of the included publications

Publications describing the development process of measures Author(s), title, year	Country	Study aim	Measure development method (Delphi, expert panel etc.)	Stakeholder involvement
1Dental quality alliance (DQA):Herndon et al (2015a)22 2American Dental Association (2016)23 3American dental Association (2018)24 4Hunt & Ohja (2017)25	United States	To develop and test paediatric oral health care quality measures	Literature review and RAND‐modified Delphi approach	Selected committee of the DQA, with engagement of multiple stakeholders, including clinicians, dental plans, State Medicaid agencies and federal agencies
5Herndon et al (2015b)26	United States	To validate three evidence‐based process of care quality measures related to dental caries prevention in children	The measures were previously developed by the DQA described in Herndon et al.22 Validity testing included the assessment of agreement between administrative data and dental records and assessment of variation in measures performance, using administrative / claims data from private or public dental benefits sources	Stakeholder involvement has been described above
6Herndon et al (2017)27	United States	To validate a DQA measure to determine whether children who have a caries‐related emergency department visit received follow‐up care and to formally validate a set of diagnostic codes in administrative claims data to identify caries‐related emergency department visit	The measures were previously developed by the DQA (Herndon et al).22 The validation process consisted of comparing the diagnosis codes in claims data with manual review of 300 records from a Florida hospital emergency department and by calculating the kappa statistic, sensitivity and specificity	Stakeholder involvement has been described above
Achmea Oral Health Project Hummel et al (2017)28 Projectteam Mondzorg (2015)29	Netherlands	To develop clinical outcome measures for oral health and explore their performance using health insurance claims records and clinical data from general dental practices	Literature review, meetings with an advisory board to select measures based on set criteria, and an implementation pilot to test the measures on feasibility and validity	Advisory board with four experts in quality of oral health care
European Global Oral Health Indicators Development (EGOHID) 1 Bourgeois et al (2008)30 Ottolenghi et al (2007)31 EGOHID catalogue, (2005)32	Europe	To develop a set of measures for monitoring and describing oral health morbidity and different facets of oral health care systems in Europe	Review of existing recommendations on oral health measures and a consensus process, including consultation meetings with a steering group and contributors, and a grading method to shortlist measures with highest importance from a long‐list	A steering group as part of the European health monitoring plan, and contributors, consisting of representatives from European clinical and scientific oral health organizations
Dental Quality and Outcomes Framework (DQOF) Department of Health (2011)33 Department of Health (2016)34	United Kingdom	To develop measures to measure dentists’ performance as part of the dental contract reform pilot of the National Health Services (NHS) in the United Kingdom	Consensus among members of a working group	Working group of 6 members, consisting of dental public health consultants and representatives of the British Dental Association and the Department of Health
National Oral Health Surveillance System (NOHSS) Malvitz et al (2009)35 Centers for Disease Control and Prevention (CDC) (2015)36 Chattopadhyay et al (2008)37	United States	To develop a public health surveillance system and to develop quality measures for adults and children which can be used by state health agencies and health programs to monitor oral health of the population	Limited information was found on the methods used to develop measures. A work group narrowed a list from 72 measures to 7, after which approval was sought by the Council of State and Territorial Epidemiologist (CSTE). All indicators were linked to the Healthy People 2010 objectives	Working group, consisting of content experts from the Center of Chronic Disease and Prevention Division of Oral Health (CDC DOH) and the Association of State and Territorial Dental Directors (ASTDD), forming the NOHSS
Nordic Project National Institute for Health and Welfare (2010)38 Ekornrud & Wilburg (2013)39	Denmark, Sweden, Norway, Finland, Iceland, Faroe Islands	To develop common quality measures for oral health care and improve for the Nordic countries	Oral health measures used in the Nordic countries were mapped. All measures were based on recommendations from the European community health indicators (ECHI), Organization for Economic Cooperation and Development (OECD) and EGOHID and specific Nordic interest. A working group agreed on the final measures	Representatives from all Nordics countries were included in the working group. Function of the representatives was not clear
Baâdoudi et al (2017)5	Europe	To establish measures of oral health for transparent and explicit reporting of routine data to facilitate more patient‐centred and prevention‐oriented oral health care	Four‐stage approach, including scoping of the literature and its appraisal, a meeting of experts, a two‐stage Delphi process and a World Café discussion	Stakeholders from 6 European Union countries, including dental practitioners, patients, insurers and policy makers from the public and private sector
Mattila et al (2002)40	Finland	To measure the quality of children's dental health care from the oral health records of 10‐year‐olds using five outcome measures	Measures were developed based on clinical experience, pedodontic expertise and scientific literature on technical (professional quality)	Not described
Mangione‐Smith et al (2011)41	United States	To describe the processes used to identify a recommended core set of quality measures for children's health care as mandated by the Children's Health Insurance Reauthorization Act of 2009 (CHIPRA) and provide an overview of the selected measures	Measures in use by Medicaid and Child Health Insurance programs (CHIP) were identified. A committee of experts developed criteria to evaluate the validity, feasibility and importance of quality measures. Subsequently a RAND‐UCLA‐modified Delphi process was used to process al measures and measures were assessed on legislative priorities	Stakeholders from the Agency for Healthcare Research (AHRQ) National Advisory Council for Healthcare Research and Quality Subcommittee for Medicaid and CHIP Programs, and members of the CHIPRA Federal Quality workgroup
Hussein et al (2017)42	Germany	To develop an external quality assurance procedure, examining the use of systemic antibiotics in periodontal, conservative and surgical treatment in ambulatory dental health care. The aim of the procedure was to increase patient safety through rational use of systemic antibiotics and increasing the use of first‐line medications	A systematic literature search, an analysis of dental claims data and antibiotic prescriptions. The proposed measures were evaluated by an expert panel using a RAND‐modified Delphi process	An expert panel consisting of dentists, maxillofacial surgeons and patient representatives

Publications describing the additional scientific properties or the implementation of measures Author(s), title, year	Country	Study aim	Methods to assess additional scientific properties or test the implementation of measures	Stakeholder involvement
Bhardwaj et al (2016)43	Unites States	To assess the feasibility and performance of a meaningful use dental clinical quality measure, which measures the percentage of children, aged 0‐20 years, who received a fluoride varnish application	The measure was previously developed as part of the Meaningful Use stage 2 by the centres of Medicare and Medicaid services (CMS). Feasibility was assessed by evaluation of the concordance between an automated query and a manual review (using dental electronic health records)	The measure was developed as part of the Meaningful Use stage 2 and 3 by Medicare and Medicaid and members of the DQA
Neumann et al (2017)44	United States	To adopt a DQA measure designed for administrative claims data to be used in electronic health records. To evaluate the feasibility and validity of implementing this measure to determine whether patients with diabetes received a comprehensive oral or periodontal examination	Adaptation of a dental quality measure, originally developed by the DQA, to be used in electronic health records. Development of an automated query to capture the oral healthcare received by patients with diabetes, and validation of this query by comparing the query with manual chart reviews	Original measure development; see DQA. Adaptation of the measure: researchers from dental Universities and dental professionals

3.3. Description of quality measures

In total, 215 oral health care quality measures were identified (see Table 2). A detailed overview of these measures, including numerators and denominators, can be found in Appendix S5. Of these measures, more than half of the identified measures were process measures (n = 108). A substantial number of outcome measures (n = 84) and a few structure measures (n = 20) were identified. The majority of measures (n = 71) was related to oral treatment or preventive services, 43 of the quality measures were related to oral disease outcomes, 35 measures covered aspects of access of care, ten measures covered aspects of oral health costs, 14 measures covered health behaviour aspects and 14 measures could be categorized as public health measures. Very few measures were related to organizational aspects of care (n = 3) or patient safety (n = 5). The most measures (207 of 215) were developed either specifically for children or without a specific target population. A few measures (eight out of 215) were developed specifically for (frail) elderly.

Table 2.

Study results describing the number of measures, the measure titles and the characteristics of the publications

Authors (year)	Number of measures		Reported tested clinimetric properties by the authors of the publications
Dental Quality Alliance (DQA): Herndon et al (2015a)22; Herndon et al (2015b)26; American Dental Association (2016)23; American Dental Association (2018)24; Hunt & Ohja (2017)25	2016: Total: 11 Outcome: 0 Process: 9 Structure: 2	Updated measures 2018a Total: 24 Outcome: 0 Process: 23 Structure: 1	Importance, feasibility, reliability and validity were tested. For validity, the project reported on face validity, convergent validity and known‐group validation. For the reliability, detailed algorithms outlining how to calculate each measure were developed. Also, a user guide was developed for the consistency in implementation. One measure was not feasible due to data limitations (measure 11)
Achmea Oral Health Project: Hummel et al (2017)28; Projectteam mondzorg (2015)29	Total: 4 Outcome: 3 Process: 1 Structure: 0		Feasibility, face validity Discriminative validity and responsiveness were reported
European Global Oral Health Indicators Development (EGOHID) I: Bourgeois et al (2008)30; Ottolenghi et al (2007)31 and EGOHID catalogue (2005)32	Total: 40 Outcome: 24 Process: 10 Structure: 6		Validity, objectivity, sensitivity and specificity reported as being important in the catalogue and both articles. However, it has not been mentioned further how they assessed these characteristics during the development process. Implementation and validity testing has been planned for EGOHID phase II
Dental Quality and Outcomes Framework (DQOF): Department of Health (2011)33; Department of Health (2016)34	DQOF for 2016‐2017 Total: 13 Outcome: 10 Process: 3 Structure: 0		None
National Oral Health Surveillance System (NOHSS): Malvitz et al (2009)35; Chattopadhyay et al (2008)37; Centers for Disease Control and Prevention (CDC) (2015)36	Updated NOHSS measures from 2015 report Total: 35 Outcome: 24 Process: 9 Structure: 2		None
Nordic Project: National Institute for Health and Welfare (2010)38; Ekornrud & Wilburg (2013)39	2010 Total: 12 Outcome: 5 Process: 3 Structure:4		The 2010 document mentioned that a measure should be valid, reliable and relevant; however, no additional information was provided
Baâdoudi et al (2017)5	Total: 63 Outcome: 15 Process: 46 Structure: 2		Validity testing has been planned for the advocate field studies
Mattila et al (2002)40	Total: 5 Outcome: 3 Process: 0 Structure: 2		None
Mangione‐Smith et al (2010)41	Total: 2 Outcome: 0 Process: 1 Structure: 1		Committee members evaluated the feasibility, validity, reliability and importance of the measures in a Delphi method based on available scientific evidence and the likelihood of available, reliable data sources
Hussein et al (2017)42	Total: 3 Outcome: 0 Process: 3 Structure: 0		The publication only described the provided descriptive frequency information measures
Bhardwaj et al (2016)43	Total: 1 Outcome: 0 Process: 1 Structure: 0		Feasibility and performance of the measure were tested. The automated query was compared with manual chart reviews and sensitivity, specificity, positive predictive value and negative predictive value were calculated
Neumann et al (2017)44	Total: 1 (and 1 DQA measure) Outcome: 0 Process: 1 (and 1 DQA measure) Structure: 0		Performance and validation of the automated query was evaluated by comparing the query with manual chart reviews, and sensitivity, specificity, positive predictive value and negative predictive value were calculated

^aThe DQA measures are updated each year.

In general, most Patient‐Reported Outcomes Measures (PROMs) and Patient‐Reported Experience Measures (PREMs) that were found in this systematic review stem from Europe,5, 28, 34 whereas for the United States, there were more population‐level measures, measures assessing disease outcomes, and measures related to oral treatment and preventive services. Compared to Europe, the publications from the United States developed more measures for the use in electronic health records (EHRs).43, 44 Whereas the quality measures developed in Europe mostly rely on questionnaire or claims data. In total, the measures of two out of 24 publications are suitable for use in EHRs.

3.4. Critical appraisal

In the publications developing quality measures, a large variation in the scientific soundness and applicability was observed (see Tables 3 and 4). In domain I (purpose, relevance and organizational context of quality measures), the scores ranged from 60% to 100%. With respect to stakeholder involvement (domain II), the scores ranged from 33% to 100%. Four out of the 10 projects developing measures scored a 3 or 4 on the involvement of all relevant stakeholders. In projects scoring a 1 or 2, the reason was most frequent that either patients or dental professionals were not included. In domain III (scientific methods), four out of 12 projects scored higher than 50% (see Tables 3 and 4). With respect to the quality of the supporting evidence, three out of 12 studies critically appraised the supporting evidence on which the measures are based (item 11 in Table 3). Five projects reported systematic methods used for the development of the quality measures. For domain IV (additional evidence, formulation and usage), the scores ranged from 22% to 100%. The two publications that described the clinimetric properties and implementation of the quality measures scored 85% and 100%, respectively, on all aspects on which the publications could be judged using the AIRE instrument. From the projects developing quality measures, three projects reported on the validity and discriminative power of the quality measures and two projects reported on the reliability of the quality measures. Although often not specifically mentioned, most projects tested the feasibility of the quality measures by collecting data (ten out of 12 projects). With respect to piloting measures in practice, four out of 12 projects reported that the measures were piloted and two projects reported to have planned a future pilot to test the validity of the measures. Looking at the overall scores, the scores ranged from 38 to 78 out of 80 possible points. The highest scoring publications stem from the last 3 years.22, 23, 24, 25, 26, 27, 28, 43, 44 Appendix S3 shows an oversight of the funding bodies and possible competing interests of the reviewed publications. All except one publication provided the name of the funding body. One of the 10 projects developing measures provided a specific statement that the funding body did not influence the measurement development process.

Table 3.

Critical appraisal of the publications using the AIRE instrument

	AIRE items																								Total score	Remarks
	Purpose, relevance and organizational context						Stakeholder involvement				Scientific methods				Additional evidence, formulation, usage
	1)	2)	3)	4)	5)	Domain score	6)	7)	8)	Domain score	9)	10)	11)	Domain score	12)	13)	14)	15)	16)	17)	18)	19)	20)	Domain score
Publications describing the development process of measures
Dental Quality Alliance (DQA)	4	4	4	4	4	100%	4	4	4	100%	4	4	2	66%	4	4	4	4	4	4	4*	4	4	100%	78	* Only the three measures in Herndon et al (2015b)26
Achmea Oral Health Project	4	4	4	3	4	93%	2	2	2	33%	3	2	2	44%	3	4	4	3	2	4	4	4	4	85%	64
European Global Oral Health Indicators Development (EGOHID) I	4	4	2	4	3	80%	3	3	3	66%	2	3	2	44%	4	4	3	2	2	1	1*	2	3	48%	55	* Implementation and testing in EGOHID phase II
Dental Quality and Outcomes Framework (DQOF)	4	3	4	3	3	80%	2	2	4	55%	2	3	2	44%	4	3	2	1	1	1	1*	1*	3	30%	49	* Pilot is being conducted as part of the dental contract reform pilot
National Oral Health Surveillance System (NOHSS)	3	4	4	3	4	87%	3	2	4	66%	1	3	1	22%	4	4	2	1	1	1	4	4	4	59%	57
A Nordic project	3	3	3	3	3	67%	1*	1*	4	33%	2	3	3	55%	3	3	2	1	1	1	2	3	2	33%	47	*Professional background of the stakeholders has not been described
Baâdoudi et al (2017)5	4	4	3	4	3	73%	4	4	2	77%	4	2	1	44%	4	3	2	1	1	1	1*	4	1	33%	53	* Pilot is being conducted as part of the ADVOCATE Field Publications
Mattila et al (2002)40	3	3	4	1	3	60%	1	1	2	11%	2	2	1	22%	1	3	1	1	1	1	3	3	1	22%	38
Mangione‐Smith et al (2010)41	4	4	4	4	2	86%	3	3	4	77%	4	4	4	100%	2	2	1	3	3	2	2	3	3	44%	61
Hussein et al (2017)42	4	4	4	3	4	93%	4	3	3	77%	4	4	3	88%	4	2	1	1	1	3	4	4	2	48%	62
Publications describing the clinimetric properties or the implementation of measures
Bhardwaj et al (2016)43	4	4	4	3	4	93%	X*	X*	4	100%	X*	X*	X*	‐	4	4	4	4	4	4	4	4	4	100%	59	*Not all fields could be appraised since no measures were developed in the article
Neumann et al (2017)44	4	4	4	4	4	100%	4*	4*	3	88%	4*	3*	2*	66%	4	4	2	4	3	3	4	4	4	85%	72	*Not all fields could be appraised since no measures were developed in the article

X = item is not applicable in this study since no measures were developed.

Table 4.

AIRE items per domain

AIRE domain	AIRE Items
Purpose, relevance and organizational context	1The goal of the study has been clearly described 2The publications rationale for measures development has been clearly described 3The organizational context of the measures has been described in detail (about whom does the measure provide information) 4The quality domains that the measures address have been described in detail 5The health care process covered by the measure has been described and defined in detail
Stakeholder involvement	6The members of the committee developing the measures included individuals from relevant professional groups 7Considering the purpose of the quality measures, all relevant stakeholders have been involved at some stage of the development process 8The measures have been formally endorsed
Scientific methods	9Systematic methods have been used to search for scientific evidence 10The measures have been based on recommendations from an evidence‐based guideline or publications published in peer‐reviewed scientific journals 11The quality of supporting evidence on which the measures have been based has been critically appraised
Additional evidence, formulation, usage	12The numerator and denominator of the measures have been described in detail 13The target patient population of the measures has been clearly defined 14A strategy for risk adjustment has been considered and described (“case‐mix adjustment”) 15The validity of the measures has been evaluated 16The reliability of the measures has been evaluated 17It has been evaluated whether the measures have sufficient discriminative power 18The measures have been piloted in practice 19The feasibility of data collection for the measures has been considered 20Measures have been supported with specific instructions to present and interpret results

4. DISCUSSION

To our knowledge, this is the first systematic review providing an overview and critical appraisal of quality measures in the field of oral health care. To assure measures based on up‐to‐date scientific evidence, publications were included if they were published in the last 15 years. In total, 215 oral health care measures were identified and all publications were critically appraised. The majority of quality measures developed in the reviewed publications are intended for assessment of processes of care, focusing on the provision of oral treatment or preventive services, on outcomes of oral care, including periodontal and dental disease outcomes, or on access to oral care. A relatively low number of structure measures were identified focusing on patient safety, organizational aspects of oral health care or costs of oral health. In addition, the findings from critical appraisal of quality measures using the AIRE instrument indicate a large variation in the scientific soundness and applicability of quality measures. To guide decisions about which of the currently available measures to use to assess quality of oral health care, the authors recommend the measures from those publications that scored highest on the development, testing and validation of the measures. At this moment, the Dental Quality Alliance (DQA),22, 23, 24, 25, 26, 27 Hummel et al,28 Bhardwaj et al43 and Neumann et al,44 showed the most extensive procedure to develop and/or test the quality measures.

Traditionally, quality measurement has focused on access issues, clinical care processes and disease‐specific measures. More than a decade ago, the NAM recognized the importance of patient experience as a domain of quality.9 Since that time, it has been acknowledged that good quality of health care does not solely comprise the technical aspects of care, but patient experiences are also key drivers of quality improvement.45 A number of the included (mostly European) projects pay attention to patient experience measures. However, these measures were not piloted in practice and tested on clinimetric properties. Many of the existing quality measures for oral health care were developed in the United States context. While the majority of these measures are process measures mirroring the quality of oral treatment or preventive services, there are comparably few measures on health outcomes, patient experience and patient safety.

Although many of the quality measures developed so far can be stratified for age groups, a large part of quality measures is specifically focused on children and/or adults, while measures for (frail) elderly are comparably scarce. All included publications describing the measures development process were either from Europe or from the United States. No measures were identified that were published in peer‐reviewed scientific journals from other continents meeting the inclusion criteria. There could still be publications in other languages without the English language abstract, but they were not identified by the search strategy.

The development process of the measures was comprehensive in most publications. Yet frequently, the evidence on which the measures were based was not appraised during the development process. Only three publications critically appraised the supporting evidence on which the measures have been based. In terms of stakeholder involvement, many committees developing the measures included individuals from relevant stakeholder groups. Yet only four publications employed a comprehensive engagement of representatives from various stakeholder groups, including patients, oral health care professionals, health policymakers and health insurers. In many of the publications, the committees consisted mainly of health policymakers and/or scientific expert researchers, whereas oral health care professionals and patients were less often involved in the development of measures. Early involvement of patients and oral health care professionals is an essential step towards successful implementation of quality measures.45, 46

Only few studies piloted quality measures in practice. Testing the quality measures in practice is an essential prerequisite when seeking to implement them for day‐to‐day assessment and quality improvement in health care.8 Moreover, only few studies tested the clinimetric properties, and often, the clinimetric properties were poorly defined. The taxonomy used for describing clinimetric properties was not always consistent across publications, which emphasizes the need for a more harmonized terminology and better standardized criteria to assess quality measures. For example, the Cosmin study could provide useful insights with respect to consistent reporting of patient‐reported health outcomes.15 The DQA exhibits an interesting example of testing the clinimetric properties of quality measures including validity, reliability and feasibility.22, 23, 24, 25, 26, 27

In recent years, EHRs have increasingly become amenable for purposes of quality measurement.47 The adoption of measures based on EHRs has the potential to advance quality measurement by the automation of data collection. Further, measures based on EHRs potentially increase transparency by availing access to information which is not accessible otherwise.47 Currently, only the measures of two out of 24 publications are suitable for use in EHRs. Although a transition from administrative claims measures to measures based on EHRs is ongoing in a number of other health care disciplines,48 similar initiatives in oral health care are scarce. Bhardwaj et al43 and Neumann et al44 are among the first in dentistry to adopt suitable quality measures for use in EHRs.

Measures based on administrative health insurance claims data have appeal as they have been routinely recorded, and no additional investment is needed for data collection (but note that the use of such data can still be costly as per data protection and efforts required for data cleansing and processing); yet they are often designed for purposes other than quality measurement. In general, health insurance claims data provide relatively little details regarding diagnostic and health outcomes data.48 More and more initiatives focus on the development of innovative IT‐infrastructures to pave the way for automated data collection for EHR measures.43, 44 The increasing attention for automated EHR measures seems promising; however, feasibility of data collection relies largely on the underlying IT‐infrastructures and data protection regulations. Large variation may exist with regard to currently available mechanisms of data collection. Depending on the available resources for IT‐infrastructures, it is likely that the feasibility of data collection differs between different country or regions and the level of measurement.16 Hence, one remaining question for future research is whether the current activities in EHR measurement can also be used in other countries.

This systematic review contributes to the literature in two ways. First, the present study identified gaps in the quality measurement field in oral health care. And second, this paper critically appraised the methods used for the development and validation of existing quality measures within the relevant literature. One of the evident strengths of this study is that, as stated above, this is the first systematic review providing a comprehensive overview of quality measures and a critical appraisal of the literature reporting these measures. The chosen methodology has been shown to be a valid and reliable approach to appraise existing quality measures. In addition, this study also gives an overview of the methods currently being used in the field of dentistry to develop oral health care quality measures. It is possible that relevant outcome measures can also be identified from patient experience assessment tools such as the Oral Health Impact Profile (OHIP). However, such instruments were out of this review's scope.

A possible limitation of the present study is that, to the author's knowledge, there is currently no tool available to specifically assess the risk of bias with respect to the methodological strengths and weaknesses of Delphi procedures or similar quality measure development methods. The AIRE instrument provides a suitable tool to appraise publications on quality measures; however, it does not include an item on the funding body and possible competing interests. It should be noted that of all the projects developing quality measures, only one project reported that the funding body did not influence the results. More research is necessary to determine the influence of funding bodies on the development and performance of measures. Further, it should be noted that it is likely that there is a possibility of publication bias due to possible oral health care measures available online which cannot be detected through systematic searches in scientific databases. However, the comprehensiveness of such measures is difficult to ascertain, and often, information about the development of these measures is lacking. Therefore, these measures do not fall within the scope of this review.

In conclusion, this review adds to the previously published literature by providing an up‐to‐date overview and appraisal of quality measures on oral health care which are amenable to assess and improve the quality of oral health care. The study highlights the continuing need for transparent, valid, reliable and feasible quality measures. Future research is warranted to enhance and harmonize the definition, measurement and improvement of quality of oral health care. Thereby, careful consideration should be given to patient‐reported outcome and experience measures as well as to the establishment of suitable information systems that allow provision of routine and transparent feedback on the quality of oral health care.49

Supporting information

 

Righolt AJ, Sidorenkov G, Faggion CM Jr, Listl S, Duijster D. Quality measures for dental care: A systematic review. Community Dent Oral Epidemiol. 2019;47:12–23. 10.1111/cdoe.12429