QUALITY APPRAISAL OF CLINICAL PRACTICE GUIDELINES ON PROVISION OF DENTAL SERVICES DURING THE FIRST MONTHS OF THE COVID-19 PANDEMIC

Abstract

Objectives

To evaluate the quality of clinical practice guidelines (CPGs) on dental services provision during the first months of the COVID-19 pandemic.

Materials and methods

We systematically searched in MEDLINE, EMBASE, LILACS, Epistemonikos, Trip databases, websites of CPG developers, compilers of CPGs, scientific societies and ministries of health to identify documents with recommendations intended to minimize the risk of COVID-19 transmission during dental care. Reviewers independently and in duplicate assessed the included CPGs using the AGREE II instrument. We calculated the standardized scores for the 6 domains and made a final recommendation about each CPG. The inter-appraiser agreement was assessed using the intraclass correlation coefficient (ICC).

Results

Twenty-three CPGs published were included. Most of the CPGs were from America (n = 15) and Europe (n = 6). The overall agreement between reviewers was very good (ICC = 0.93; 95%CI 0.87-0.95). The median score for each domain was the following: Scope and purpose 67% (IQR 20%); Stakeholder involvement 33% (IQR 14%); Rigour of development 13% (IQR 13%); Clarity of presentation 64% (IQR 31%); Applicability 19% (IQR 17%) and Editorial independence 8% (IQR 8%). Twenty two guidelines were not recommended by the reviewers. Only one of the CPGs was recommended with modifications. The median overall rate was 3 (IQR 1). All CPGs were classified as low quality.

Conclusions

The overall quality of CPGs on dental services provision during the first months of the COVID-19 pandemic was low, which makes its implementation difficult for clinicians and policy makers. Therefore, it is critical that developers are transparent and forthcoming about the difficulties that have arisen during the CPG development process.

INTRODUCTION

By the end of 2019, the new Severe Acute Respiratory Syndrome CoronaVirus2 (SARS-CoV-2) was reported in Wuhan China.¹ This pathogen is the cause of the coronavirus 2019 disease (COVID-19).² In just a few months, COVID-19 had spread worldwide, becoming a major public health problem and being categorised by The World Health Organisation as a pandemic disease.³

Epidemiological data show that all ages are susceptible to COVID-19. However, individuals with co-morbidities are more likely to develop severe clinical symptoms.⁴ Health workers are exposed to contaminated fluids/aerosols through close contact with SARS-CoV-2 infected patients’ face, having an unprecedented occupational risk of morbidity and mortality.^{5 , 6} In the field of dentistry, treatments are delivered in close proximity to the patient. Furthermore, dental workers are subject to frequent exposure to saliva, blood and other bodily fluids,⁷ contact with contaminated surfaces,^{7 , 8} and frequent exposure to bioaerosols.^{7 , 9 , 10} Dentists are at a higher risk of infection, and of becoming potential carriers of the disease.^{9 , 11} The use of the dental drill, ultrasonic scalers, and air/water syringes magnify the dissemination of saliva droplets, blood and aerosol emissions, all facilitate environment, equipment, and surface contamination.¹² As a result, dental service providers may also be at a higher risk of cross infection with patients.^{9 , 11}

Currently, vaccines are being tested globally; nevertheless, it will take time until the majority of the population is protected. Meanwhile, biosafety measures are fundamental to stop the spread and transmission of the disease. Therefore, dental services must prioritize the patient's protection and reduce the coronavirus transmission risk, in both public and private dental services, to ensure the health and safety of the dental staff and patients.

To provide appropriate dental services, protocols and clinical practice guidelines (CPGs) focused on important biosafety measures for a low-risk COVID-19 transmission environment have been developed. In order to provide the safest dental practice based on scientific evidence, CPGs should aim for the best quality and be readily available,¹³ thus improving the quality and results of medical services.¹⁴ Consequently, assessing the quality of CPGs developed during COVID-19 pandemic in the dental field is a critical task. In this study we assessed the quality of CPGs on dental services during the first months of the COVID-19 pandemic.

MATERIAL AND METHODS

Design

Systematic Evaluation of CPGs

For the development of this assessment we adhered to the Preferred Reporting Items of Systematic Reviews and Meta-Analysis PRISMA tool.¹⁵ We published the protocol in Open Science Framework.¹⁶

Eligibility Criteria

We included documents in English, Spanish and Portuguese self-declared as a guideline, guidance providing recommendations for walk-in or emergency dental care in the context of the COVID-19 pandemic. A document was also eligible if it was developed by regional, national, or local organizations, a team of experts, by a panel of experts, or if it provided a list of authors/experts involved in the process. We excluded Clinical Practice Guidelines developed exclusively for special-care patients, guidelines developed outside of the pandemic context, previous versions of the same guideline, letters to the editor, and conference summaries.

Sources of Information

We searched in MEDLINE, EMBASE, LILACS, Epistemonikos, and Trip databases up to August 10, 2020. This was complemented by a manual search of guideline developers’ websites, repositories, international dental scientific societies, Ministries of Health worldwide and institutions related to COVID-19 management. This search was also updated in August 2020. (Appendix S1). We also examined the reference lists of the selected CPGs to identify other guidelines that met the inclusion criteria. We did not limit the search by date, language or publication status.

The search strategy and databases used are listed in supplementary material (Appendix S1).

Selection of Guidelines

All references identified were extracted to an EndNote X9 database to facilitate their management.

Titles, abstracts and full texts were independently reviewed by 2 investigators (C.Z, G.E.E., N.F.D., P.M.M, P.P or Y.A.R) in a 3-step process. We resolved disagreements with the help of a third reviewer. In the case of several documents from a single source, we included the most recent.

Data Charting Process

One reviewer (C.Z, G.E.E, N.F.D., P.M.M, P.P or Y.A.R) extracted the data and another reviewer checked all the extracted information for accuracy. We extracted the following information from each article using a previously piloted data collection form: author, year, title, country, organization, language, setting, and target population.

Critical Appraisal of Individual Sources of Evidence

Two researchers (A.S., C.Z., N.F.D., P.M.M or Y.A.R) independently rated the quality of each guideline using the online AGREE II tool (https://www.agreetrust.org/resource-centre/agree-plus/). The AGREE II instrument¹⁷ includes 23 items divided into 6 domains: 1. Scope and purpose, refers to the aim of the guidelines; 2. Stakeholder involvement, represents the views of the intended users; 3.Rigor of development, reflects the quality of the CPG development process and its recommendations; 4. Clarity of presentation, refers to the structure of the guidelines; 5. Applicability, shows the barriers and facilitators for the implementation of the CPG, also considering the financial implications to implement the recommendations; and 6. Editorial independence, illustrating the transparency in the formulation of the guideline recommendations. Each of the 23 items is rated from 1 to 7 points in the Likert scale. The AGREE II instrument also includes 2 overall quality appraisals for each CPG: an overall score of 1 to 7, and whether the reviewer would recommend using the CPG, assessing it as recommended, recommended with modifications, or not recommended.¹⁷

Synthesis of the Results

The total AGREE II score was calculated by summing up the individual items included in a domain and by scaling the total as a percentage of the maximum possible score for that domain.¹⁷ If discrepancies between reviewers were higher than 3 points, or if the standard deviation (SD) in an item was greater than, or equal to 1.5 SD, the item of the guideline was reassessed by the reviewers and agreement was reached.¹⁸ The reviewers considered the overall quality of the CPGs and made a final recommendation about each guideline.¹⁸ We classified the CPG as high quality when at least 3 of the domains showed a score of 60% or higher, including the rigor of development domain.17, 18, 19

Data Analysis

The intraclass coefficient with its 95% confidence interval (95% CI) as an indicator of overall agreement between reviewers for each of the 23 items of the AGREE II instrument was calculated. According to the scale proposed by Landis and Koch,²⁰ the degree of agreement between 0.01 and 0.20 is slight, from 0.21 to 0.40 is fair, from 0.41 to 0.60 is moderate, from 0.61 to 0.80 is substantial and from 0.81 to 1.00 is very good.

Shapiro-Wilk test was performed to analyze the normality of the data. Given the skewed distribution of the AGREE II scores, median and the interquartile range (IQR) were calculated for each domain score for each CPG. We used nonparametric analysis and Mann-Whitney U or Kruskal-Wallis tests, with post-hoc test, to assess the association between total AGREE II score and the characteristics of the CPG such as region, presence of update, and CPGs development agencies, (Ministries or Health departments and government agencies vs scientific societies, dental colleges or universities).

Spearman test was used to analyze the correlation between the score obtained in the AGREE II domains and total score, in order to determine which domains influenced the overall quality of the CPG. Spearman's correlation was classified as: negligible (0.0-0.1), weak (0.1>ρ>0.39), moderate (0.4<ρ<0.69), strong (0.7-0.89), and very strong (0.8-1.0).²¹

Data were analyzed with IBM SPSS v.27 software for Mac,²² considering a significance threshold of 5%.

RESULTS

After excluding duplicates, the search initially yielded 1149 references. Twelve documents were selected from databases and 11 were identified through other sources. Finally, 23 documents were included. Figure 1 shows the selection process flow chart. The agreement between reviewers during all phases of the process was high (Kappa = 0.872).

Figure. 1.

Flowchart of the included CPGs.

Of the CPGs included in the study, fifteen were from America, six from Europe, and one from Asia and Oceania. The developers were mainly Ministries or Health Departments and government agencies (n = 11), scientific societies and dental colleges (n = 13). Following publication of an initial version a total of 10 CPGs were subsequently updated. Main CPG characteristics are listed in Table 1 .

Table 1.

General information of the clinical practice guidelines.

Reference	Guideline title	Country	Organization	Language	Date publication (P) Date of update (U)
41	COVID-19. Recomendaciones para Odontología	Argentina	Ministerio de Salud de Argentina	Spanish	P: 06/10/20 U: no
36	Manual de buenas prácticas en bioseguridad para ambientes odontológicos	Brazil	Conselho Federal de Odontologia, ITI Brasil, ITI México Faculdade ILAPEO, Consejo Mexicano de Rehabilitación Oral y Maxilofacial a.c., AMP, APP, Universidad de Concepción, Centro de Rehabilitación Oral Avanzada e Implantología-Facultad de Odontología	Spanish, Portuguese	P: 03/21/20 U: 05/26/20
39	Orientação de biossegurança. Adequações técnicas em tempos de COVID-19	Brazil	Conselho Regional de Odontologia de São Paulo	Portuguese	P: 04/20 U: No
23	Nota técnica GVIMS/GGTES/ANVISA n. 04/2020 Orientações para serviços de saúde: medidas de prevenção e controle que devem ser adotadas durante a assistência aos casos suspeitos ou confirmados de infeccção pelo novo coronavírus (SARS- CoV-2).	Brazil	Agência Nacional de Vigilância Sanitária - ANVISA	Portuguese	P:1/30/20 U: 02/17/20 U: 03/21/20 U: 03/31/20 U: 05/08/20
42	Nota técnica conjunta n°014/2020-DAPS/DIVS/DIVE/SES/COMSES/SC (02/07/2020). Coronavírus COVID-19: Recomendações ao atendimento odontológico do SUS em Santa Catarina	Brazil	Secretaria de Estado da Saúde de Santa Catarina	Portuguese	P:07/02/20 U: no
33	CDSS Alert – COVID-19 Pandemic: IPC Interim Protocol Update “RE - OPEN SASKATCHEWAN” – A Summary for Dentists	Canada	The College of Dental Surgeons of Saskatchewan	English	P: 04/27/20 U: no
40	Return to practice office manual. Adapting the dental office to the COVID-19 Pandemic	Canada	Canadian Dental Association	English	P: 05/10/20 U: no
32	Orientaciones para atención odontológica en fase IV COVID-19	Chile	Ministry of Health	Spanish	P: 05/18/20 U: no
31	Lineamiento técnico para la prevención y contención de COVID-19 para odontólogos y personal auxiliar de Costa Rica	Costa Rica	Ministerio de Salud Pública de Costa Rica	Spanish	P: 03/27/20 U: no
44	Protocolo para atención odontológica en emergencias y urgencias odontológicas durante la emergencia sanitaria por COVID–19	Ecuador	Ministerio de Salud Pública	Spanish	P: 04/2020 U: no
35	Standard operating procedure. Transition to recovery. A phased transition for dental practices toward the resumption of the full range of dental provision	England	National Health Services. NHS England	English	U: 06/04/20
34	COVID-19 guidance and standard operating procedure. For the provision of urgent dental care in primary care dental settings (from 8 June 2020) and designated urgent dental care provider sites	England	National Health Services. NHS England	English	P:04/15/20 U:06/16/20
29	Protocolo de Bioseguridad Odontológica con énfasis en COVID-19	Guatemala	Colegio Estomatológico de Guatemala	Spanish	P: 05/2020 U: no
43	A statutory body constituted under the dentists act, 1948	India	Dental Council of India	English	P: 05/07/20 U: no
24	Guidelines for oral health services at COVID-19 Alert Level 1	New Zealand	Ministry of Health	English	P: 03/31/20 U: 04/22/20 (Level 3) 11/05/20 (Level 2) 06/08/20 (Level 1) 06/23/20 (Level 1) 07/03/20 (Level 1)
38	Protocolo para atención Odontológica durante la pandemia de SARS-CoV-2 en la República del Paraguay	Paraguay	Ministerio de Salud	Spanish	P: 04/14/20 U: no
25	COVID-19: Procedimentos em clínicas, consultórios ou serviços de saúde oral dos cuidados de saúde primários, setor social e privado	Portugal	Ordem dos Médicos Dentistas, Programa nacional de Promoção da Saude Oral, Programa Nacional de Prevenção e Controlo de Infeções e das Resistências aos Antimicrobianos	Portuguese	P: 05/01/20 U: 07/27/20
27	Resuming General Dental Services Following COVID-19 Shutdown. A guide and implementation tools for general dental practice. For Phases 2 and 3 of dental services remobilisation	Scotland	The Scottish Dental Clinical Effectiveness Programme	English	P: 05/25/20 U: 06/12/20
26	Plan estratégico de acción para el período de desescalada COVID-19	Spain	Consejo de Dentistas de España	Spanish	P: 04/13/20 U: 05/02/20
28	Implications of COVID-19 for the safe management of general dental practice. A practical guide	UK	College of General Dentistry Faculty of General Dental Practice	English	P: 06/16/20 U: no
30	Guidance for Dental Settings. Interim Infection Prevention and Control Guidance for Dental Settings During the coronavirus disease 2019 (COVID-19) Pandemic	USA	Centre for Disease Control and Prevention	English	P: 06/17/20 U: 08/04/20
45	Return to Work Interim Guidance Toolkit	USA	American Dental Association	English	P: 05/07/20 U: 07/23/20
37	Ruta de atención para procedimientos de Odontología Pediatrica durante la etapa de confinamiento o cuarentena de la pandemia COVID-19	–	Asociación Latinoamericana de Odontopediatria.	English, Spanish, Portuguese	P: 04/11/20 U: no

Appraisal of CPGs

The overall agreement between the 2 reviewers with respect to the use of the AGREE II instrument was very good (ICC = 0.93; 95%CI 0.7-0.95). Table 2 shows the standardized domain scores per CPG, as well as the overall recommendation. Of the 6 domains, 3 scored above 60% in at least one CPG: Scope and purpose in 16 CPGs (69.6%),23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38 Clarity of presentation in 15 CPGs (65.2%),23, 24, 25, 26, 27, 28, 29, 30, 31, 32 ^, 35, 36, 37 and Stakeholder involvement in 2 CPGs (8.7%).^{27 , 28} The domain with the highest score was Scope and Purpose (median 67.0%), and the one with the lowest score was Editorial Independence (median 8.0%).

Table 2.

Standardized scores across CPGs per domain (AGREE II).

Reference	Guide Title	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence	Overall rate	Overall Recommendation	Quality
41	COVID-19. Recomendaciones para Odontología	22%	8%	6%	25%	0%	0%	2.0	Not recommended	Low
36	Manual de buenas prácticas en bioseguridad para ambientes odontológicos	78%	44%	5%	78%	10%	0%	3.0	Not recommended	Low
39	Orientação de biossegurança. Adequações técnicas em tempos de COVID-19	58%	33%	7%	64%	13%	8%	3.5	Not recommended	Low
23	Nota técnica GVIMS/GGTES/ANVISA n. 04/2020Orientações para serviços de saúde: medidas de prevenção e controle que devem ser adotadas durante a assistência aos casos suspeitos ou confirmados de infeccção pelo novo coronavírus (SARS- CoV-2).	78%	47%	19%	64%	19%	8%	4.0	Not recommended	Low
42	Nota técnica conjunta n°014/2020-DAPS/DIVS/DIVE/SES/COMSES/SC (02/07/2020). Coronavírus COVID-19: Recomendações ao atendimento odontológico do SUS em Santa Catarina	39%	19%	0%	44%	2%	0%	2.0	Not recommended	Low
33	CDSS Alert – COVID-19 Pandemic: IPC Interim Protocol Update “RE - OPEN SASKATCHEWAN” – A Summary for Dentists	61%	31%	13%	44%	10%	8%	2.5	Not recommended	Low
40	Return to practice office manual. Adapting the dental office to the COVID-19 Pandemic	47%	33%	16%	61%	21%	0%	3.0	Not recommended	Low
32	Orientaciones para atención odontológica en fase IV COVID-19	72%	25%	6%	69%	2%	0%	2.5	Not recommended	Low
31	Lineamiento técnico para la prevención y contención de COVID-19 para odontólogos y personal auxiliar de Costa Rica	64%	28%	8%	64%	8%	0%	2.5	Not recommended	Low
44	Protocolo para atención odontológica en emergencias y urgencias odontológicas durante la emergencia sanitaria por COVID–19	58%	36%	16%	33%	21%	8%	3.5	Not recommended	Low
35	Standard operating procedure. Transition to recovery. A phased transition for dental practices toward the resumption of the full range of dental provision	81%	36%	9%	75%	29%	8%	3.5	Not recommended	Low
34	COVID-19 guidance and standard operating procedure. For the provision of urgent dental care in primary care dental settings (from 8 June 2020) and designated urgent dental care provider sites	78%	50%	11%	56%	19%	8%	3.5	Not recommended	Low
29	Protocolo de Bioseguridad Odontológica con énfasis en COVID-19	81%	42%	22%	69%	23%	0%	3.5	Not recommended	Low
43	A statutory body constituted under the dentists act, 1948	22%	28%	4%	50%	13%	4%	2.0	Not recommended	Low
24	Guidelines for oral health services at COVID-19 Alert Level 1	67%	14%	21%	81%	10%	8%	3.0	Not recommended	Low
38	Protocolo para atención Odontológica durante la pandemia de SARS-CoV-2 em la República del Paraguay	61%	33%	5%	25%	17%	0%	2.0	Not recommended	Low
25	COVID-19: Procedimentos em clínicas, consultórios ou serviços de saúde oral dos cuidados de saúde primários, setor social e privado	61%	36%	7%	75%	19%	8%	3.0	Not recommended	Low
27	Resuming General Dental Services Following COVID-19 Shutdown. A guide and implementation tools for general dental practice. For Phases 2 and 3 of dental services remobilization	89%	86%	51%	72%	31%	58%	4.5	Recommended with modifications	Low
26	Plan Estratégico de acción para el periodo de desescalada COVID-19	83%	42%	21%	86%	29%	8%	4.0	Not recommended	Low
28	Implications of COVID-19 for the safe management of general dental practice. A practical guide	83%	86%	27%	97%	40%	0%	4.0	Not recommended	Low
30	Guidance for Dental Settings. Interim Infection Prevention and Control Guidance for Dental Settings during the coronavirus disease 2019 (COVID-19) Pandemic	72%	33%	17%	78%	29%	0%	4.0	Not recommended	Low
45	Return to Work Interim Guidance Toolkit	28%	19%	13%	36%	27%	33%	3.0	Not recommended	Low
37	Ruta de atención para procedimientos de Odontología Pediátrica durante la etapa de confinamiento o cuarentena de la pandemia COVID-19	78%	42%	16%	64%	19%	13%	3.0	Not recommended	Low
	Mean	63.5%	37.0%	13.9%	61.4%	17.8%	7.8%	3.1
	(SD)	(19.8%)	(18.6%)	(10.6%)	(19.4%)	(10.2%)	(13.1%)	(0.73)
	Median	67.0%	33.0%	13.0%	64.0%	19.0%	8.0%	3.0
	(IQR)	(20.0%)	(14.0%)	(13.0%)	(31.0%)	(17.0%)	(8.0%)	(1.0)

Scope and Purpose

The median score was 67.0% (IQR 20.0%). Of the 23 guidelines, 16 (69.6%)23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38 scored above 60% in this domain. The intended aim was well defined by the CPGs (65.2%) as was the population (patients, public, etc.) for whom the guideline was intended.^{23 , 24 ,} 26, 27, 28, 29, 30, 31, 32, 33, 34 ^{, 36 , 37 , 39} However, some CPGs did not describe the specific health questions covered by the guideline of the specific health issue (eg, recommendation about Personal Protective Equipment, environmental infection control, procedural risk mitigation, etc.).

Stakeholder Involvement

Median score was 33.0% (IQR 14.0%). Two of the CPGs (8.7%)^{27 , 28} scored above 60% in this domain. Scarcity of detailed information about the guideline development group (discipline, institution, description of the role), as well as not taking into account the target users point of view (only 2 CPGs considered this point)^{27 , 28} Most of the CPGs clearly declared a target user (60.8%).^{23 ,} 25, 26, 27, 28, 29, 30 ^, 33, 34, 35, 36, 37, 38 ^{, 40}

Rigour of Development

Median score was 13.0% (IQR 13.0%). In this domain no guidelines were found that scored above 60% and 10 (43.5%)^{25 , 31 , 32 , 35 , 36 , 38 , 39 ,} 41, 42, 43 of the guidelines scored below 10%. The CPGs developed during the first 8 months of the COVID-19 pandemic did not use systematic methods for the development of recommendations. Only 4 declared conducting a systematic search, or that the evidence was provided by guideline panel members (17.4%).27, 28, 29 ^{, 37} Four guidelines (17.4%)^{27 , 28 , 37 , 44} reported some information about selection criteria.

Two CPGs (8.7%)^{27 , 28} declared assessing the body of the evidence, but did not describe the methodology applied. In most of the guidelines the description of the methods used to develop the recommendations, and the means by which panels reached their decisions was not clear. Only 6 CPGs (26.1%)^{24 , 27 , 28 , 37 , 40 , 44} reported scarce information, of these 6, 2 (8.7%)^{28 , 35} reported the use of a Delphi method for the development of recommendations.

Two guidelines reported an obvious relation between recommendations and the supporting evidence (8.7%).^{27 , 30} Ten guidelines noted this aspect for a subset of the recommendations (43.5%).^{23 , 24 , 26 , 28 , 29 , 34 , 35 , 37 , 44 , 45} One the 23 guidelines (4.3%)²⁷ reported some information regarding external peer review prior to publication. Considering the updating process only one (4.3%)²⁷ of the ten updated CPGs (43.5%)23, 24, 25, 26, 27 ^{, 30 ,} 34, 35, 36 ^{, 45} reported some information.

Clarity of Presentation

The median score for this domain was 64.0% (IQR 31.0%). Eight CPGs (34.8%)^{33 , 34 , 38 ,} 41, 42, 43, 44, 45 scored below 60% in this domain. The majority of the guidelines presented specific and unambiguous recommendations (86.9%). In 8 of the guidelines (34.8%)^{23 , 25 , 26 , 28 , 30 , 35 , 36 , 39} specifically listed the different treatment options and in 5 CPGs (21.7%)24, 25, 26 ^{, 28 , 36} key recommendations were easily identified..

Applicability

Median score was 19.0% (IQR 17.0%). All of the guidelines (100%) scored below 60% in the Applicability domain. Two guidelines CPGs (8.7%)^{26 , 45} provided advice and/or tools on how the recommendations could be implemented.

Editorial Independence

For this domain, the median score was 8.0% (IQR 8.0%). Ten of the CPGs (43.5%)28, 29, 30, 31, 32 ^{, 36 , 38 ,} 40, 41, 42 obtained a score of 0.0%, and all of the guidelines (100%) scored below 60%.

Overall Assessment

None of the 22 assessed guidelines was recommended by the reviewers. Only one of the CPGs (4.3%)²⁷ was recommended with modifications. All the guidelines were classified as low quality, since none scored ≥ 60% in at least 3 domains, including the Rigour of Development.

The median overall rate was 3 (IQR 1), the highest score was 4.5,²⁷ and the lowest was 2.^{38 ,} 41, 42, 43

Association Between Agree II Score and Characteristics of the CPG

Updated CPGs scored significantly higher in the AGREE II overall rate. CPGs developed in Europe showed significantly higher scores in the domains Scope and purpose, Stakeholder involvement and Clarity of presentation. However, we did not find significant differences between guidelines developed by Ministries or government agencies and scientific societies. (Table 3 ).

Table 3.

Median and interquartile range (IQR), in percentages (%) for the AGREE II domains according to prespecified predictors.

Variables	n	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence	Overall rate
		Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)	Median (IQR)
Update conducted
Yes	10	78.0 (16.0)	39.0 (18.3)	15.0 (12.5)	75.00 (16.8)	23.0 (12.3)	8.0 (8.3)	3.5 (1.0)
No	13	61.0 (32.0)	33.0 (12.3)	8.0 (10.5)	61.00 (28.0)	13.0 (16.0)	0.0 (8.0)	2.5 (1.5)
P-value		.057	.148	.186	.021*	.067	.049*	.015*
Type of organization
Scientific Societies, Dental Colleges or Universities	12	69.5 (32.8)	39.0 (12.0)	14.5 (14.8)	66.5 (24.5)	20.0 (15.5)	8.0 (11.8)	3.0 (0.88)
Ministries or Health Departments and Government Agencies	11	67.0 (20.0)	33.0 (17.0)	9.0 (11.0)	64.0 (42.0)	17.0 (19.0)	0.0 (8.0)	3.0 (1.5)
P-value		.608	.151	.316	.316	.118	.260	.449
Region
Asia	1	22.0	28.0	4.0	50.0	13.0	4.0	2.0
America	15	61.0 (31.0)†	33.0 (17.0)†	13.0 (10.0)	64.0 (33.0)†	17.0 (13.0)	0.0 (8.0)	3.0 (1.0)
Europe	6	82.0 (10.8)†	46.0 (50.0)†	16.0 (24.5)	75.0 (20.8) †	29.0 (14.3)	8.0 (14.5)	3.8 (0.75)
Oceania	1	67.0	14.0	21.0	81.0	10.0	8.0	3.00
P-value		.027*	.026*	.138	.047*	.057	.371	0.060

^⁎Statistically significant difference.

^†Statistically significant difference between Europe and America.

A high significant correlation between the scores of AGREE II domains and the overall rate was observed, except for Editorial Independence (r = 0.374; P = .078) (Appendix S2).

DISCUSSION

Main Findings

This study showed a low quality for CPGs developed during the first 8 months of the pandemic aimed at minimizing the risk of contagion by COVID-19 during dental care. The domain with the highest score was Scope and Purpose (median 67.0%), and the one with the lowest score was Editorial Independence (median 8.0%). According to the AGREE II instrument the median overall rate was 3.0 (IQR 1.0). Only one of the CPGs was recommended with modifications by the reviewers.

Our Results in the Context of Previous Research

Although the domains “Scope and purpose” and “Clarity of presentation” obtained a mean quality score of over 60%, remaining domains presented an exceptionally poor report reflected in very low scores. The low score for the remaining domains, was also reported for other guidelines carried out in the context of the COVID-19 pandemic.^{46 , 47} Norris et al⁴⁸ who evaluated WHO guidelines generated during 4 previous emergencies related to recent infectious diseases (H1N1, H7N9 virus, MERS-CoV, and Ebola), concluded that few guidelines met international standards for reliable and evidence-based CPGs. Moreover, they pointed out CPGs’ weaknesses, such as limited mention of systematic reviews, failure to perform external peer reviews, and incomplete information with regard to development methods.⁴⁸ Similarly, these aspects were also observed in our study. In our research, only 2 guidelines^{27 , 28} presented a score higher than 60% in at least 3 domains. Notwithstanding the above, whether performed during or outside a pandemic context,^{46 , 47 , 49} the quality of the CPGs in the dental field in general tend to be of substandard quality with AGREE II overall rate.

The “Stakeholder involvement” domain presented a low-quality score, namely as a result of poor information with regard to group participants developing the guide and the information related to the target population. This was also observed among guidelines developed during the COVID-19⁴⁷ pandemic and outbreaks of other diseases.⁵⁰ In a pandemic context, where the rapid development of recommendations is needed, it is difficult to meet some of the aspects in this domain. A contributing factor to the lower quality, may be that participation of all the relevant stakeholders is often not feasible.⁵¹

The domain “Rigour of Development” is usually considered the main domain of evaluation in clinical guidelines, since it assesses the process used to gather and synthesize the evidence and methods to formulate the recommendations.¹⁹ In our research, we pointed out that the guidelines developed in the initial months of the COVID-19 pandemic, presented a poor report, with little information regarding the methods used for the synthesis of the evidence, or for the development of the recommendations. Very few CPGs related the recommendations to the supporting evidence. Furthermore, they did not report conducting external peer reviews prior to publication. This is also consistent with guidelines developed in response to health emergencies for both disease outbreaks,^{48 , 50} and guidelines developed in times of the COVID-19 pandemic.^{46 , 47 , 52}

It is possible that the low score in “Rigour of Development” presented in the guidelines developed during the initial period of the COVID-19 pandemic, was related to the difficulty in obtaining reliable evidence, since very little information of the virus was available. Hence, studies currently available have important methodological limitations.⁵³ For this reason, it is preferable to state that systematic reviews were not used in the process, because none were available in the period in which the guideline was developed, rather than to directly omit the information.⁵⁰

Due to the COVID-19 health emergency, some medical CPGs carried out in the first months of the pandemic based their recommendations in studies developed during the SARS outbreak in 2003.⁵⁴ Even though, the use of indirect evidence based on other viral infections is controversial,⁵⁵ it must be determined a priori which indirect evidence will be considered for the development of the CPG.⁵⁶ For example, the World Health Organization (WHO) developed recommendations for its COVID-19 guidelines based on a previously developed guideline for MERS, given that both are betacoronaviruses.⁴⁹

We consider that some of the recommendations made in the dental guidelines developed in these first months of the pandemic, should have been supported by indirect evidence. For instance, the development of recommendations for reducing AGPs in dental practice should have been based on indirect evidence of other types of coronaviruses. Recommendations based on indirect evidence could have positively influenced the quality assessment of the guidelines, both for the “Rigour of Development” domain, as well as for global quality.

CPGs should incorporate the Grading of Recommendation Assessment, Development and Evaluation (GRADE) approach, since this provides a structured approach to determine the certainty of the evidence, and to make recommendations and decisions.⁵⁷ In our study, none of the guidelines used the GRADE approach to assess certainty of the evidence or to develop the recommendations, as reported in studies that evaluated COVID-19 CPGs.⁴⁶

The “Applicability” domain also rendered a very low-quality score in our analysis, since the guidelines did not adequately report the facilitators and obstacles to its application. This was also reported in the study by Song et al⁵⁸ for other guidelines related with COVID-19. The low score was also reported for dental guidelines carried out outside the pandemic context.⁴⁹ Evidently, the report for this domain should be implemented generally in dental guidelines, not only for those carried out during times of health emergencies.

In this research, “Editorial Independence” was the one domain that presented the lowest quality, with a score of 0% in at least 10 guidelines. In medical guidelines carried out during the current COVID-19 pandemic, this domain also presented a low-quality score, with a score of 0% for most of these.^{54 , 58} In our study, the low score obtained for this domain was the result of incomplete reports due to conflicts of interest. This fact was also observed by Norris et al⁴⁸ in the evaluation of guidelines carried out for other infectious diseases. Dental guidelines carried out outside the COVID-19 pandemic context also present a suboptimal score⁴⁹ for this domain, indicating the need to improve reporting on the influence of the possible financial and institutional conflicts of interest.

In reference to the variables that influence the quality of the guidelines, our analysis showed that the CPGs produced in Europe had higher average scores compared to guidelines developed in other continents. Mubeen et al⁴⁹ found that the guidelines developed in Europe and North America presented better quality compared to guides developed in other countries. Access to the necessary resources both financial and human, may possibly influence score quality of the guideline depending on the geographical area where they are developed. Therefore, adequate access to resources would allow CPG to be carried out in a shorter period. Most of the CPG included were developed by national organizations and only 3 have been developed by local organizations. This may be due to the fact that national organizations have greater resources (both financial and human) and, therefore, can provide a faster response during a health emergency such as the COVID-19 pandemic. Another variable analyzed in the present study was the update of the guidelines. We verified that the updated CPGs presented a higher quality for the “Clarity of Presentation” and “Applicability” domains. This finding underlines the concept that guidelines generated in the first stage of a health emergency are of lower quality. It further accentuates the importance of updating guidelines as new evidence comes to light.

Strengths and Limitations

The main strengths of this study are that the information regarding development process of CPGs was obtained in a systematic search of the literature that included websites of developers and repositories of CPG. In addition, the guidelines were evaluated independently by 2 calibrated reviewers using a standardized tool. The AGREE II instrument is currently the only validated and reliable instrument that enables a quantitative comparison of CPGs, and is designed to help users to evaluate their methodological quality.

However, there are certain limitations in our study. Even though, an extensive search including gray literature was carried out, it is possible that relevant guidelines in a language other than English, Spanish, or Portuguese were not included. Furthermore, we only evaluated the latest version available, and some information reported in previous versions may provide additional information to those reported in the latest version, not being reflected in our evaluation. However, we think that this is unlikely as typically updated versions tend to show higher scores, as our results showed. Finally, the instrument used for quality evaluation (AGREE II) was not developed to evaluate CPGs for periods when timely and prompt answers are needed.⁴⁸ Some domains require longer time for their development impacting the level of quality.

Implications for Practice and Research

Developing trustworthy guidelines in a relatively short timeframe is a challenge, which has been evidenced in this study. Some domains that require more time to develop will always be susceptible to lower quality.⁵⁰ However, guideline developers should establish methods to ensure the guideline is conducted in a transparent and evidence-based manner. Therefore, it is critical that developers are transparent and forthcoming about the difficulties that have arisen during the CPG development process.

The low-quality of CPGs on dental services during the COVID-19 pandemic may not achieve effective health outcomes, or might not appropriately consider the risks of using those recommendations in a specific scenario.¹⁹ This makes their implementation difficult for clinicians and policy makers,⁵⁹ and may interfere with patient adherence to the recommendations.⁶⁰

Future research should be aimed at optimizing the development process of the CPGs and allow updating recommendations as soon as relevant evidence becomes available. While some efforts have been made,⁶¹ there is still uncertainty about the thresholds for changing recommendations or the potential approaches for their prompt publication and dissemination. In addition, efforts should be made to avoid duplication in the development of guidelines. Evidence ecosystem models attempt to address this issue,⁶¹ however, there is a need for a comprehensive organization to guide and support all those involved.

Finally, it is imperative that COVID-19 dental guidelines use a transparent framework for rating the body of evidence, so that users can understand the mechanism used to develop the recommendations.

CONCLUSIONS

The overall quality of CPGs on dental services provision during the first months of the COVID-19 pandemic was low, which makes its implementation difficult for clinicians and policy makers. Therefore, it is critical that developers are transparent and forthcoming about the difficulties that have arisen during the CPG development process.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest with any financial organization regarding the material discussed in the manuscript.