Global Guidelines in Dermatology Mapping Project (GUIDEMAP), a systematic review of atopic dermatitis clinical practice guidelines: are they clear, unbiased, trustworthy and evidence based (CUTE)?

Abstract

Background

Clinical practice guidelines (CPGs) are essential in delivering optimum healthcare, such as for atopic dermatitis (AD), a highly prevalent skin disease. Although many CPGs are available for AD, their quality has not been critically appraised.

Objectives

To identify CPGs on AD worldwide and to assess with validated instruments whether those CPGs are clear, unbiased, trustworthy and evidence based (CUTE).

Methods

We searched MEDLINE, Embase, PubMed, Web of Science, Cochrane Library, Emcare, Epistemonikos, PsycINFO and Academic Search Premier for CPGs on AD published between 1 April 2016 and 1 April 2021. Additionally we hand searched prespecified guideline resources. Screening, data extraction and quality assessment of eligible guidelines were independently carried out by two authors. Instruments used for quality assessment were the AGREE II Reporting Checklist, the US Institute of Medicine (IOM) criteria of trustworthiness and Lenzer’s Red Flags.

Results

Forty CPGs were included, mostly from countries with a high sociodemographic index. The reporting quality varied enormously. Three CPGs scored ‘excellent’ on all AGREE II domains and three scored ‘poor’ on all domains. We found no association between AGREE II scores and a country’s gross domestic product. One CPG fully met all nine IOM criteria and two fully met eight. Three CPGs had no red flags. ‘Applicability’ and ‘rigour of development’ were the lowest scoring AGREE II domains; ‘external review’, ‘updating procedures’ and ‘rating strength of recommendations’ were the IOM criteria least met; and most red flags were for ‘limited or no involvement of methodological expertise’ and ‘no external review’. Management of conflicts of interest (COIs) appeared challenging. When constructs of the instruments overlapped, they showed high concordance, strengthening our conclusions.

Conclusions

Overall, many CPGs are not sufficiently clear, unbiased, trustworthy or evidence based (CUTE) and lack applicability. Therefore improvement is warranted, for which using the AGREE II instrument is recommended. Some improvements can be easily accomplished through robust reporting. Others, such as transparency, applicability, evidence foundation and managing COIs, might require more effort.

What is already known about this topic?

• Atopic dermatitis is a skin disease with a high prevalence and high burden of disease.

• Clinical practice guidelines are essential for good clinical practice and shared decision making, for both patients and caregivers, in order to reduce disease burden.

• Many clinical practice guidelines are available for atopic dermatitis worldwide, but their quality has not yet been critically appraised.

What does this study add?

• Forty guidelines <5 years old were identified, mostly from countries with a high sociodemographic index, many of which are not sufficiently clear, unbiased, trustworthy or evidence based (CUTE).

• Improvement of guidelines is warranted, for example by using the AGREE II instrument and robust reporting.

• Some guideline domains can be improved without much effort, yet improving transparency and applicability and managing conflicts of interest might be more challenging.

Many clinical practice guidelines are available for atopic dermatitis worldwide, but their quality has not been critically appraised as yet. Forty guidelines less than five years old were identified, mostly from countries with a high socio‐demographic index, many of which are not clear, unbiased, trustworthy or evidence based (CUTE) enough; Improvement of guidelines is warranted, e.g. by using the AGREEII instrument and robust reporting

Plain language summary available online

Clinical practice guidelines (CPGs) are essential for delivering optimum healthcare for patients, regardless of which healthcare provider delivers such care: their intention is to describe the available options of care, with their benefits and possible harms. ¹ CPGs provide healthcare providers with diagnostic and treatment options, based on the best available external evidence, and permit integration of clinical expertise and patients’ values and preferences, as per the definition of evidence‐based medicine (EBM) of Sackett et al. ² Shared decision making allows the best personalized diagnostic path or treatment strategy to be chosen. ³ In accordance with that paradigm, CPGs should incorporate the following aspects of evidence‐based medicine: a diverse guideline development group combining all the necessary expertise (clinical and methodological), clinical questions based on patients’ needs, an underpinning systematic review and rating of the evidence, and local patient and/or stakeholder involvement to represent their views and values. Combining all these aspects should result in practical, clinical, graded and nuanced recommendations.

CPGs come in various formats and designs, as was demonstrated recently in a scoping review of available guidelines for the 12 most burdensome dermatological diseases. ⁴ In a follow‐up of that scoping review, teams were formed per skin disease to (re)identify these (possibly updated) CPGs and subsequently appraise their quality. In the present review we assessed the CPGs on atopic dermatitis (AD). Among the nonfatal diseases worldwide, AD ranks number 14, measured in disability‐adjusted life‐years and prevalence. ⁵ For skin diseases specifically, AD ranks number 1, due to its prevalence and overall burden of disease. ⁵ Therefore, CPGs on AD are an important tool in caring for the wellbeing of people with AD.

The core question of this review is: are CPGs on AD clear, unbiased, trustworthy and evidence based (CUTE)? In order to answer that, this study summarizes and reports on the number of CPGs, their origin, their availability and, most importantly: how CUTE they actually are.

Materials and methods

This systematic review follows the PRISMA statement 2020. ⁶ The review is part of the GUIDEMAP project (https://sites.manchester.ac.uk/guidemap). The prespecified protocol for the project, including this review, was published 30 October 2019 at the Open Science Foundation. ⁷

Eligible studies

Any CPG on AD (inclusive of consensus agreement guidelines) developed by local, regional, national or international groups, or affiliated governmental organizations, was eligible. Excluded were consensus statements based on expert opinion solely, single‐author documents, CPGs that lacked recommendations for patients on diagnosis and/or treatment options, standalone treatment algorithms, summaries, reviews and duplicate publications. When updated versions of the same guideline were retrieved, the most recent version was included.

Literature search

The only deviation from the protocol ⁷ was the update of the search dates, which were 1 April 2016 to 1 April 2021. The rationale for the search windows of 5 years is that guidelines are constantly updated, usually every 5 years, or earlier when deemed necessary.

Bibliographical databases that were searched were MEDLINE (OVID version), Embase (OVID version), PubMed, Web of Science, Cochrane Library, Emcare (OVID version), Epistemonikos, PsycINFO (EbscoHOST version) and Academic Search Premier. If possible the CADTH (Canadian Agency for Drugs and Technologies in Health) filter designed for identifying guidelines was used. ⁸ The search was performed on 1 April 2021 by a data specialist (J.W.S.) and was provided to the reviewers deduplicated. The full search strategy is presented in Appendix S1 (see Supporting Information).

The search results were uploaded to Rayyan (https://rayyan.ai) for independent screening by two reviewers (E.J.vZ. and Z.F.), based on title, abstract and keywords. A third independent reviewer (B.W.M.A.) resolved any differences. In addition, a hand search was conducted independently by two reviewers (B.W.M.A. and E.J.vZ.) using guideline resources such as DynaMed, Emergency Care Research Institute, Guidelines International Network, National Institute for Health and Care Excellence, Scottish Intercollegiate Guidelines Network, and Turning Research into Practice. Furthermore, more than 200 websites of dermatological societies who are members of the International League of Dermatological Societies were independently hand searched by two reviewers (B.W.M.A. and E.J.vZ.). No language restrictions were applied.

Records that were deemed eligible were retrieved as full text. Two reviewers (E.J.vZ. and Z.F.) assessed their eligibility and a third reviewer (B.W.M.A.) was consulted to discuss differences and jointly decide. Extra caution and deliberation were taken with consensus‐based publications, as they are mostly based on expert opinion with less apparent, or sometimes without, evidence foundation. Yet when those publications clearly provided clinical practice recommendations, it was unanimously agreed by the GUIDEMAP team to include them. All references of the included CPGs were checked (E.J.vZ.) for additional eligible reports.

Methodologies for appraisal

As per the prespecified protocol, based on the publication of Eady et al. on acne CPGs, ⁹ the instruments used to assess and report on the quality of the retrieved guidelines were the AGREE II Reporting Checklist, ¹⁰ the US Institute of Medicine (IOM) criteria of trustworthiness, ¹ and Lenzer’s Red Flags. ¹¹ See Table 1 for the domains, criteria and scoring per instrument. Assessment was blinded and carried out independently in pairs by four authors (B.W.M.A., E.J.vZ., S.V. and Z.F.).

Table 1.

Assessment instruments with their domains, criteria and scoring

AGREE II10		IOM criteria 1		Lenzer’s Red Flags 11
Twenty‐three items organized within six domains rated on a 7‐point rating scale (1 = strongly disagree to 7 = strongly agree). The scoring per domain is reported as the percentage (higher is better). It is followed by two global rating items (omitted)		Eight criteria; however, we decided to split ‘establishing evidence foundations for and rating strength of recommendations’ into two criteria, as was done previously. 9 Thus nine criteria were assessed as ‘fully met’, ‘partially met’ or ‘not met’		The categorical scores are ‘red flag’, ‘caution’, ‘uncertain’ or ‘no concerns’. A red flag indicates an element known to introduce potential bias. ‘Caution’ indicates an item for which there is not proof that bias is introduced. ‘Uncertain’ indicates that raters could not confidently score the element
Domain 1	Scope and purpose	Criterion 1	Establishing transparency	Sponsor(s) is a professional society that receives substantial industry funding
Domain 2	Stakeholder involvement	Criterion 2	Management of conflicts of interest	Sponsor is a proprietary company, or is undeclared or hidden
Domain 3	Rigour of development	Criterion 3	Guideline development group composition	Committee chair(s) have any financial conflicta
Domain 4	Clarity of presentation	Criterion 4	Systematic review intersection	Multiple panel members have any financial conflicta
Domain 5	Applicability	Criterion 5	Establishing evidence foundations	Any suggestion of committee stacking that would preordain a recommendation regarding a controversial topic
Domain 6	Editorial independence	Criterion 6	Rating strength of recommendations	No or limited involvement of an expert in methodology in the evaluation of evidence
Overall quality of the guideline (1, 2, 3, 4, 5, 6, 7)		Criterion 7	Articulation of recommendations	No external review
Recommended for use (yes, yes with modification, no)		Criterion 8	External review	No inclusion of nonphysician experts, patient representative, community stakeholders
		Criterion 9	Updating procedures

To facilitate equitable appraisal, criterion 5 of the Institute of Medicine (IOM) was split so that evidence foundations and rating the strength of recommendations were evaluated separately. ^aIncludes a panellist with either or both a financial relationship with a proprietary healthcare company and/or whose clinical practice or specialty depends on tests or interventions covered by the guideline.

Data extraction and management

For the characteristics of the included CPGs we used the predefined datasheet that was used in the scoping review. ⁴ For the AGREE II appraisals we used the online AGREE PLUS tool, which facilitates blinded group appraisals (https://www.agreetrust.org/my-agree). After completing the appraisals, the scoring was unblinded. If there was more than a two‐point difference on scoring one of the 23 items, this was discussed and resolved between the reviewers. The consolidated data were exported from AGREE PLUS into a datasheet as a percentage score per domain (0–100%) and graded. These grades were in concordance with our protocol: excellent (≥ 70%), average (≥ 50% and < 70%) and poor (< 50%). We did not assign the CPGs an overall grade, because that would unlikely reflect the diverse strengths and weaknesses of a CPG. Also, the AGREE II user’s manual states ‘The six domain scores are independent and should not be aggregated into a single quality score’, as there is no advice given about the relative weightings of the six domains. For the IOM criteria and Lenzer’s Red Flags we designed forms per reviewer for their assigned and blinded assessments. After unblinding, any difference between reviewers in scoring was resolved and collated.

Statistical analyses

For descriptive statistics we used Microsoft Excel 2010. SPSS version 20·0 (IBM, Armonk, NY, USA) for Windows was used to investigate a possible association between AGREE II scores and gross domestic product (GDP), and for calculating correlations between AGREE II, IOM and Red Flags scores.

Results

Search results

The search provided 5414 records, of which 3603 were duplicates (Appendix S1). Of the remaining 1811 records, 1744 were deleted for not meeting the inclusion criteria based on screening of title, abstract and keywords. The full text was obtained of the 67 potentially eligible reports. The hand search yielded 14 additional full‐text reports. Thorough examination of eligibility resulted in 40 included CPGs on AD, published in 56 reports. ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ The reason for the latter is that some CPGs were published in parts as journal articles (Figure 1). Twenty‐five studies were excluded based on full text (Table S1; see Supporting Information).

Figure 1.

PRISMA 2020 flow diagram for new systematic reviews that included searches of databases, registers and other sources. ⁶ ECRI, Emergency Care Research Institute; G‐I‐N, Guidelines International Network; GP, general practitioner; ILDS, International League of Dermatological Societies; NICE, National Institute for Health and Care Excellence; SIGN, Scottish Intercollegiate Guidelines Network; TRIP, Turning Research into Practice.

Characteristics of the included clinical practice guidelines

We included 40 CPGs, ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ of which the majority (27) were from countries with a high sociodemographic index (SDI), ¹⁴ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ with only two ³⁷ , ³⁸ , ³⁹ , ⁴⁰ from a country with a middle‐low SDI (both from India) and none from countries with a low SDI. Nine CPGs came from Asia ¹³ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁹ , ⁶⁰ , ⁶¹ ; one from Australia ¹⁴ ; 22 from Europe ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ ; four from North America ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ⁶⁷ and four from South America. ¹² , ¹⁵ , ²⁸ , ⁵⁰ Funding was not disclosed in seven CPGs ⁴⁴ , ⁴⁵ , ⁵⁶ , ⁵⁷ , ⁵⁹ , ⁶² , ⁶³ ; eight were funded and/or facilitated by pharmaceutical companies ¹² , ¹³ , ¹⁴ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ⁴⁰ , ⁵⁰ , ⁶⁷ ; 10 were not funded ¹⁵ , ¹⁶ , ³⁰ , ³¹ , ³³ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴³ , ⁶⁴ , ⁶⁵ , ⁶⁶ and the remaining 15 were funded by the government, through a research grant or by the medical societies involved. ²⁸ , ²⁹ , ³² , ³⁴ , ³⁵ , ⁴¹ , ⁴² , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁸ , ⁶⁰ , ⁶¹ Dissemination was mostly done through medical journals (30) and 10 reports ¹² , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³⁴ , ⁴⁹ , ⁵¹ , ⁵⁷ , ⁶³ , ⁶⁶ were only available on the website of either a medical society or a governmental agency. Full public access was available for 36 of them, while four ¹⁴ , ⁴⁴ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁶⁰ needed a login (journal or website). Eight CPGs were in languages other than English: Danish (one), Dutch (one), Finnish (one), Spanish (three), Russian (one) and Ukrainian (one). ¹² , ²⁸ , ²⁹ , ³⁴ , ⁵⁰ , ⁵¹ , ⁵⁷ , ⁶³ In nine CPGs a patient representative was included in the CPG group. ¹⁶ , ³⁰ , ³¹ , ³² , ³⁵ , ⁵¹ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ See Table 2 for the characteristics of the included CPGs.

Table 2.

Characteristics of the included clinical practice guidelines

Country/region	Journal/ website	SDI	GDP US$	Language	Funding	Open access	In scoping review	Patient involvement	GRADE used	AGREE used
Argentina 2019 12	Website	Middle	8442	Spanish	Industry/pharma	Yes	No	No	No	No
Asia 2018 13	Journal	High‐middle	–	English	Industry/pharma	Yes	Yes	No	No	No
Australia 2020 14	Journal	High	51 812	English	Industry/pharma	No	No	No	No	No
Brazil 2019 15	Journal	Middle	6770	English	None	Yes	Yes	No	No	No
Canada 2017 16	Journal	High	43 242	English	None	Yes	Yes	Yes	No	No
Canada 2018 17 , 18 , 19 , 20 , 21 , 22	Journal	High	43 242	English	Pharma	Yes	No	No	No	No
Canada 2019 23 , 24 , 25 , 26 , 27	Journal	High	43 242	English	Pharma	Yes	No	No	No	No
Colombia 2018 28	Website	Middle	5333	Spanish	Society	Yes	No	No	Yes	Yes, to assess guidelines used
Denmark 2018 29	Website	High	60 909	Danish	Society	Yes	Yes	No	Yes, modified	No
Europe 2018 30 , 31	Website	High	33 928	English	None	Yes	Yes	Yes	No	Only mentioned in abstract
Europe 2021 32	Journal	High	33 928	English	Society	Yes	No	Yes	Yes	No
Europe 2020 33	Journal	High	33 928	English	None	Yes	No	No	No	No
Finland 2016 34	Website	High	49 041	Finnish	Government	Yes	Yes	No	No	No
Germany 2021 35	Journal	High	45 724	English	Society	Yes	No	Yes	No	No
Hong Kong 2021 36	Journal	High	46 324	English	None	Yes	No	No	No	No
India 2017 37 , 38 , 39	Journal	Low‐middle	1901	English	None	Yes	Yes	No	No	No
India 2017 40	Journal	Low‐middle	1901	English	Industry/pharma	Yes	Yes	No	No	No
Italy 2018 41	Journal	High	31 676	English	Society	Yes	No	No	No	No
Italy 2021 42	Journal	High	31 676	English	Society	Yes	No	No	No	No
Italy 2019 43	Journal	High	31 676	English	None	Yes	No	No	No	No
Italy 2019 44	Journal	High	31 676	English	Not disclosed	No	No	No	No	No
Italy 2020 45	Journal	High	31 676	English	Not disclosed	Yes	No	No	No	No
Japan 2019 46 , 47 , 48	Journal	High	40 113	English	Government	No	No	No	No	No
Malaysia 2018 49	Website	High‐middle	10 402	English	Government	Yes	Yes	No	Yes, modified	Yes, to assess guidelines used
Mexico 2018 50	Journal	Middle	8347	Spanish	Industry/pharma	Yes	Yes	No	No	No
Netherlands 2019 51	Website	High	52 304	Dutch	Grant/fellowship	Yes	Yes	Yes	Yes, partially	Yes
Poland 2020 52 , 53 , 54	Journal	High	15 656	English	Society	Yes	No	No	No	No
Poland 2019 55	Journal	High	15 656	English	Society	Yes	Yes	No	No	No
Romania 2019 56	Journal	High‐middle	12 896	English	Not disclosed	Yes	Yes	No	No	No
Russia 2020 57	Website	High‐middle	10 127	Russian	Not disclosed	Yes	No	No	No	No
Serbia 2016 58	Journal	High‐middle	7666	English	Government	Yes	Yes	No	No	No
Singapore 2016 59	Journal	High	59 798	English	Not disclosed	Yes	Yes	No	No	No
South Korea 2016 60	Journal	High	31 489	English	Grant/fellowship	No	Yes	No	Yes	No
Taiwan 2020 61	Journal	High	–	English	Society	Yes	No	No	No	No
Turkey 2018 62	Journal	High‐middle	8538	English	Not disclosed	Yes	Yes	No	No	No
Ukraine 2016 63	Website	High‐middle	3727	Ukrainian	Not disclosed	Yes	Yes	No	No	No
UK 2018 64	Journal	High	40 285	English	None	Yes	Yes	Yes	No	Yes
UK 2016 65	Journal	High	40 285	English	None	Yes	Yes	Yes	No	Yes
UK 2021 66	Website	High	40 285	English	None	Yes	Yes	Yes	Yes	No
USA 2017 67	Journal	High	63 544	English	Industry/pharma	Yes	Yes	Yes	No	No

GDP, gross domestic product; SDI, sociodemographic index.

In the preceding GUIDEMAP scoping review, 30 CPGs on AD were included. ⁴ In the present review we could include 40, of which 22 were also reported in that scoping review. The reasons for the differences are that 14 CPGs were published after the search date of the scoping review (1 October 2019) and the remainder were either outdated, or updated and thus replaced with a newer version.

AGREE II scoring

The CPGs from Columbia, ²⁸ the Netherlands ⁵¹ and the UK (antimicrobials) ⁶⁶ scored ‘excellent’ for all six AGREE II domains, whereas CPGs from Poland (phototherapy), ⁵⁵ Romania ⁵⁶ and Serbia ⁵⁸ scored ‘poor’ on all domains. The remainder showed a large variety in scoring and grading per domain (Table 3). A heatmap with in‐depth details per item is presented in Table S2 (see Supporting Information).

Table 3.

Scoring results for AGREE II, ^a Institute of Medicine (IOM) and Red Flags

Guidelineb	AGREE II domains						IOM criteria			Lenzer’s Red Flags
	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence	Fully met	Partially met	Not met
Colombia 2018 28	92	72	90	94	83	92	7	1	1	1
Netherlands 2019 51	81	83	83	94	71	92	7	1	1	1
UK 2021 66	83	86	84	86	75	88	8	1	0	0
Europe 2021 32	97	92	97	97	94	58	8	1	0	1
Malaysia 2018 49	94	61	81	89	88	92	9	0	0	0
UK 2018 64	83	81	71	69	77	92	6	3	0	1
Finland 2016 34	61	75	72	81	52	75	5	3	1	3
Germany 2021 35	75	81	51	82	27	92	6	1	2	3
Europe 2018 30 , 31	64	83	69	86	31	83	6	3	0	2
USA 2017 67	78	72	59	78	27	63	5	2	2	5
South Korea 2016 60	67	44	83	78	33	88	6	3	0	0
Italy 2019 43	72	50	38	72	19	79	1	4	4	3
Mexico 2018 50	97	69	53	75	44	50	0	6	3	3
UK 2016 65	61	47	72	58	56	71	6	3	0	2
Australia 2020 14	72	56	46	78	31	46	2	4	3	6
Canada 2019 23 , 24 , 25 , 26 , 27	72	44	32	78	23	63	0	6	3	7
India 2017 40	72	53	44	72	25	13	1	5	3	4
Japan 2019 46 , 47 , 48	67	58	53	78	29	63	3	4	2	5
Hong Kong 2021 36	50	33	34	64	33	75	1	4	4	3
Italy 2021 42	75	58	30	67	23	29	0	6	3	3
Asia 2018 13	72	33	51	61	38	25	0	6	3	5
India 2017 37 , 38 , 39	28	25	27	78	23	63	1	6	2	3
Brazil 2019 15	39	11	32	58	15	83	1	3	5	3
Europe 2020 33	50	53	26	64	52	50	0	6	3	5
Russia 2020 57	50	58	46	69	63	46	3	4	2	2
Ukraine 2016 63	56	61	39	67	58	42	1	4	4	3
Canada 2017 16	58	64	23	69	40	58	0	4	5	4
Italy 2020 45	58	42	31	69	23	63	0	4	5	4
Italy 2018 41	67	44	34	50	23	54	0	6	3	5
Italy 2019 44	39	44	65	69	27	33	2	6	1	4
Denmark 2018 29	47	39	28	67	25	67	1	4	4	5
Turkey 2018 62	50	33	32	64	35	46	0	3	6	4
Canada 2018 17 , 18 , 19 , 20 , 21 , 22	67	33	27	64	15	42	0	5	4	7
Argentina 2019 12	50	19	21	56	19	13	0	5	4	6
Taiwan 2020 61	47	25	19	50	10	42	2	4	3	3
Singapore 2016 59	42	25	17	50	10	25	0	1	8	4
Poland 2020 52 , 53 , 54	17	25	15	56	10	33	0	3	6	4
Poland 2019 55	39	36	18	42	17	42	0	2	7	4
Serbia 2016 58	36	28	15	39	21	42	0	1	8	3
Romania 2019 56	31	14	8	47	10	8	0	2	7	4

^a AGREE II scores in percentages per domain (higher is better). ^bSorting based on number of AGREE II domains scoring excellent (≥ 70%, green), average (≥ 50% and < 70%, yellow) and poor (< 50%, red). This is no absolute ranking from highest to lowest quality.

From highest to lowest, the results per AGREE II domain reported in median percentages (higher is better) and interquartile range in percentage points were ‘clarity of presentation’ (69·0%, 58·75–78·0%), ‘scope and purpose’ (62·5%, 47·75–74·25%), ‘editorial independence’ (58·0%, 42·0–78·0%), ‘stakeholder involvement’ (48·5%, 33·0–67·75%), ‘rigour of development’ (38·5%, 27·0–68·0%) and ‘applicability’ (28·0%, 21·5–52·0%). Based on the interquartile range, ‘rigour of development’ showed the most dispersion and ‘clarity of presentation’ the least (Table S3; see Supporting Information).

Although guideline development takes considerable resources, we found no association (R ²  = 0·05) between the quality of the CPGs assessed with AGREE II (total sum of six scores) and the GDP per capita of a country or region (Figure 2).

Figure 2.

Gross domestic product (GDP) per capita vs. AGREE II sum‐of‐domain scores. Scatter plot with a simple linear regression line. Source GDP: https://data.worldbank.org/indicator/NY.GDP.PCAP.CD; GDP data for Taiwan and Asia were unavailable; the funding source is based on how it was reported in the guideline.

Institute of Medicine scoring

One AD guideline scored ‘fully met’ on all nine IOM criteria, being that from Malaysia. ⁴⁹ Two scored ‘fully met’ on eight criteria: the European EAACI guideline on dupilumab ³² and the UK guideline on antimicrobial treatment. ⁶⁶ The lowest scoring two, from Serbia ⁵⁸ and Singapore, ⁵⁹ only scored one criterion as ‘partially met’ (Table 3).

Regarding the nine IOM criteria, the criterion ‘external review’ scored the lowest, with 31 of the 40 CPGs not meeting that criterion, followed by ‘updating procedures’ with 26. The highest scoring criteria were ‘transparency’ and ‘management of conflicts of interest’, but only 12 CPGs for each scored ‘fully met’. This indicates that 28 CPGs (70%) did not meet these criteria, or did so only partially. More details are provided in Table S4 (see Supporting Information).

Red Flags scoring

Three CPGs had no red flags: from Malaysia, ⁴⁹ South Korea ⁶⁰ and the UK (antimicrobials). ⁶⁶ The CPGs assigned the most red flags were from Canada, each with seven out of eight. ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ Looking at the domains assessed, most red flags were for ‘no external review’, with 32 of the CPGs flagged, in line with the score of that IOM criterion. Second most was ‘no or limited involvement of an expert in methodology’, with 29 red flags. There were no red flags for the domain ‘the suggestion of committee stacking that would preordain a recommendation’, which was very difficult to assess. Second to that was ‘sponsor(s) is a professional society that receives substantial industry funding’, which was also difficult to assess, with six of the 40 CPGs still being red flagged. ¹² , ¹³ , ¹⁴ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ The data are summarized in Table 3 and Table S5 (see Supporting Information).

Correlations between AGREE II, Institute of Medicine and Red Flags scores

Although AGREE II, IOM and Red Flags are different instruments to appraise a CPG, in terms of domains, criteria and methods of assessment, correlations between the three may be expected because of overlapping constructs. To assess this, we calculated the Pearson correlation coefficient between the sum of the AGREE II domain scores and IOM items (fully, partially or not met) and the number of Red Flags. Higher AGREE II scores were significantly and strongly correlated with more IOM criteria being fully met (r = 0·86) and were moderately related with scoring fewer red flags (r = −0·63) (Table S6; see Supporting Information).

Discussion

CPGs are essential for diagnosing and treating patients. ¹ They have a unique place in medicine, as they bridge the needs of patients – by combining evidence, clinical expertise and patient values – to treatment recommendations that are appropriate and feasible in the local context. This also means they are not globally valid, because they take into account the local healthcare system, availability of treatments and resources. With AD being the leading contributor to the global disease burden in nonfatal skin disease measured with disability‐adjusted life‐years, ⁵ it is commendable that we could identify 40 CPGs. Eighteen were published between October 2019 (end date of scoping review) ⁴ and April 2021, of which 14 were new and four were updates.

The AGREE II domain of applicability, which addresses how recommendations can be put into (local) practice and how results are being monitored, was the lowest scoring domain. Clarity of presentation of recommendations was the best scoring AGREE II domain. However, only 17 of the CPGs mention the strength of the recommendations, as per this IOM criterion. Overall, the scores of AGREE II, IOM and Red Flags could be easily improved by just reporting who the CPG is intended for, in terms of healthcare providers or patients, and the CPG’s update policy (or expiration), and to have the CPG externally reviewed.

Recommendations, the quintessential deliverables of a CPG, are founded on rating of the evidence, before weighing in local context. Many CPGs lack detailed reporting of how rating was conducted. This is reflected in the AGREE II domain ‘rigour of development’, with the majority scoring poor (24 of the 40), and also in the IOM criteria ‘systematic review intersection’ and ‘evidence foundations’, both of which were not, or were only partially, met in 30 CPGs. This is substantiated by the finding that CPGs hardly included methodological expertise (29 red flagged), six (partially) used the AGREE II method, and seven (partially) used the GRADE framework. ⁶⁸ Twelve CPGs scored ‘fully met’ on the IOM criterion ‘transparency’. Transparency could be greatly enhanced with use of AGREE II or GRADE.

GRADE is often viewed as demanding and resource intensive. The GRADE‐ADOLOPMENT Evidence to Decision framework can be used to adopt existing recommendations or adapt them to the local context, or – if needed – to develop new recommendations, reducing the resources and time needed. ⁶⁹ Other available methodologies are the ADAPTE process ⁷⁰ or even RAPADAPTE. ⁷¹ In the commentary on our GUIDEMAP scoping review the authors expressed concerns that ‘resource‐poor nations’ are adopting existing guidelines without taking local considerations into account. ⁷² This need not be the case when countries are adapting existing guidelines using GRADE‐ADOLOPMENT or ADAPTE. That such is possible and feasible is demonstrated by our findings that the Columbian ²⁸ and Malaysian ⁴⁹ CPGs, for example, scored very high in reported quality: both used AGREE II and GRADE.

Special consideration in CPG development is managing possible conflicts of interest (COIs). This is captured in the AGREE II domain ‘editorial independence’. Sixteen CPGs scored poor on this domain, even though only reporting declaration of funding and COIs was required. This finding is substantiated with the IOM criterion ‘managing conflicts of interest’: 28 did not fully meet it. For six CPGs a red flag was raised for ‘sponsor(s) is a professional society that receives substantial industry funding’, 14 for ‘committee chair(s) have any financial conflict’ and 14 for ‘multiple panel members have any financial conflict’. This aspect of COIs, and managing them appropriately, was not that important in the realm of AD until 2018, as all treatments were out of patent except for topical crisaborole, which was not broadly marketed. In CPGs the COIs were thus mostly declared for being involved in emerging, systemic treatments. In 2017 dupilumab was approved as the first new systemic treatment in decades for moderate‐to‐severe AD, making managing of COIs of CPG group members more pertinent.

Four new systemic treatments are now approved or on the brink of being approved: baricitinib, upadacitinib, abrocitinib (all Janus kinase inhibitors) and tralokinumab (an interleukin‐13 inhibitor). These new systemic treatments are currently considered equal and interchangeable because of lack of head‐to‐head studies and real‐world evidence. The only source of comparison is a (living) network meta‐analysis. ⁷³ , ⁷⁴ Usually these new treatments are recommended after conventional systemic treatment (ciclosporin, methotrexate, azathioprine and mycophenolic acid) has been unsuccessful – a threshold that manufacturers of new systemic treatments would like to have removed. For future AD guidelines this means that the interests of CPG group members need to be not only reported, but also rigorously managed. This is difficult, as a CPG group also benefits from the knowledge and clinical expertise of these AD researchers involved in new treatments. Yet it is essential for the trustworthiness of future AD CPGs that the chair has no (conflicts of) interest at all and that members who do have interests are not able to vote on recommendations on the subject of systemic treatment, and that this is also documented. If not reported in the publication itself, then it should be available on request.

A strength of this study is that we conducted a thorough search by an experienced data specialist (J.W.S.), using multiple databases without language restriction. In addition we hand searched all websites of the dermatological societies, and checked the references of included reports. For appraisal of CPGs we used three instruments, each having different aims and domains. That approach showed its strengths: when domains overlapped, the results for each instrument were always in agreement with the other ones, never the opposite. Criteria specific for an instrument provided additional and useful information (e.g. COIs). Last but not least, this study of course included a patient representative with AD (B.W.M.A.).

The limitations are that we cannot be certain that all AD guidelines were found, for instance if a dermatological society was not a member of the International League of Dermatological Societies or if website addresses were not known or not accessible. Four guidelines ²⁴ , ²⁹ , ⁵⁷ , ⁶³ needed to be translated, for which Google Translate was used. This might have resulted in missing nuances in the text that could have been important for the appraisal, although we were very thorough by discussing this in pairs after unblinding.

In conclusion, considering the global burden of disease caused by AD, it is commendable that we could identify 40 CPGs <5 years old. Yet, these CPGs are not as clear, unbiased, trustworthy and evidence based (CUTE) as they could and should have been. There is much room and need for improvement; this could be established by using the AGREE II instrument. Some improvements are easy to accomplish through better reporting. Others, like transparency, applicability, evidence foundation and managing COIs, might require more effort.

Author contributions

Bernd WM Arents: Conceptualization (lead); data curation (lead); formal analysis (lead); investigation (lead); methodology (lead); project administration (lead); software (lead); supervision (lead); validation (lead); visualization (lead); writing – review and editing (lead). Esther J van Zuuren: Conceptualization (equal); data curation (equal); formal analysis (equal); investigation (equal); methodology (equal); project administration (equal); software (equal); supervision (equal); validation (equal); visualization (equal); writing – review and editing (equal). Sofieke Vermeulen: Conceptualization (supporting); data curation (equal); formal analysis (supporting); investigation (supporting); methodology (supporting); project administration (supporting); writing – review and editing (supporting). Jan W Schoones: Data curation (equal); methodology (supporting); software (supporting); writing – review and editing (supporting). Zbys Fedorowicz: Conceptualization (supporting); data curation (equal); formal analysis (supporting); investigation (supporting); methodology (equal); writing – review and editing (supporting).

Supporting information

Table S1 Excluded reports with reasons.

Table S2 Heatmap of the AGREE II items.

Table S3 Descriptive results per AGREE domain.

Table S4 Institute of Medicine scoring in detail.

Table S5 Red Flags scoring in detail.

Table S6 Correlational statistics of the three instruments.

Appendix S1 Search strategy.

Powerpoint S1 Journal Club Slide Set.