Quality assessment and comparative analysis on the recommendations of current guidelines on the management of peripheral arterial disease: a systematic review protocol

Abstract

Introduction

Peripheral arterial disease (PAD) is the third leading atherosclerotic arterial disease. There is evidence that there is a high variation in the quality and recommendations of clinical practice guidelines for PAD, leading to the possibility of confusion among clinicians and patients. This study aims to conduct a quality assessment and comparative analysis of the clinical practice guidelines on PAD written between 2010 and 2020.

Method and analysis

We aim to perform a systematic review of clinical practice guidelines written between 2010 and 2020. A search for guidelines will be conducted through medical databases Scope, Pubmed, TRIP, Guideline Clearinghouses and specialist international organisations’ specific websites. Guidelines that meet the inclusion criteria will be extracted from the search result. The Appraisal of Guidelines for Research and Evaluation II (AGREE-II instrument) will assess the quality of the selected guidelines. The recommendations, level of evidence and other relevant information will be extracted in a datasheet for qualitative analysis. The score for each guideline’s quality will be represented using charts and central tendency measures for comparison. The summary of recommendations will also be represented in tables for easy comparison for similarities and variations across sections. Finally, the level of evidence on which the recommendations are based will also be noted along with other significant characteristics such as the authors’ financial relationship to the biomedical community. We aim to point out deficiencies present in current guidelines and elucidate areas where recommendations are made with low-level evidence. The results will enable the scientific community to design future research to fill in PAD management knowledge gaps.

Ethics and dissemination

No ethical approval was sought. Dissemination will be via journal articles and conference presentations.

PROSPERO registration number

CRD42020219176.

Strengths and limitations of this study.

• This review will be exhaustive, critically appraising all aspects of clinical practice guidelines (CPGs) for peripheral arterial disease (PAD), from the quality, through screening and diagnosis, to all aspects of treatment (pharmacological and non-pharmacological), and to our knowledge, previous reviews have only reviewed aspects of the CPGs, so this will be the first all-encompassing review.

• In the previous reviews, Ferket and colleagues focused on reviewing recommendations on screening across the PAD guidelines, Barriocanal and colleagues focused on reviewing the quality of PAD CPGs and, finally, Chen et al reviewed guideline quality and recommendations across screening and pharmacological aspects of PAD management; however, this review will not be limited in that regard and as such will include quality assessment, and recommendations across screening, diagnosis and investigative evaluations, pharmacological and non-pharmacological interventions.

• Our review will focus on the most recently written CPGs, that is, those written within the last 10 years with an expectation that the more recent advancements revealed through clinical trials and improved standards for writing CPGs should reflect in this work when compared with the previous reviews.

• This study, being a systematic review that will not involve patient recruitment and using a qualitative methodology, will be cost-effective, furthermore, since a qualitative approach is to be used for the CPG recommendation synthesis is that the underlying reasons for variations can be explored in detail with the potential of exposing knowledge gaps in PAD and atherosclerosis management.

• Qualitative analysis is inherently difficult to analyse and summarise especially as there is so much information contained in the CPGs, as such, distilling all that information into useful summaries will be a daunting task, with the potential loss of vital information that may be difficult to prevent and also, given the large amount of information in the CPGs, data extraction will be a time-consuming process.

Introduction and background

Atherosclerotic vascular diseases remain the world’s leading cause of mortality today despite dramatic declines in trend over the last few decades.¹ Epidemiological data show that behind ischaemic heart disease and stroke, peripheral arterial disease (PAD) is the third most common atherosclerotic arterial disease.^2–5

Despite its significant contribution to morbidity and mortality globally, there had been a paucity in the number of randomised clinical trials and high-quality systematic reviews of randomised clinical trials on PAD with consequent low-quality recommendations in practical guidelines. Over time, the results of high-powered RCT have been published with others on the way, which we expect to influence the more recent clinical practice guidelines (CPGs). Also, several recommendations in available guidelines were reached via expert consensus.⁶ It is no surprise that reviews of existing guidelines have revealed variations in PAD treatment recommendations in the past.^{7 8}

CPGs are methodically developed statements aimed at guiding physicians and patients in making safe healthcare decisions based on the best available evidence.^{9 10} The last 30 years have witnessed a skyrocketing in developing CPGs,¹¹ calling into question quality issues; consequently, several reputable organisations have continued to improve the standards for CPG developments.^12–14 Ideal CPG recommendations are based on strong evidence.¹⁵ However, high-level evidence is often unavailable for specific situations for several reasons, giving room for introducing various forms of bias with consequent variations in recommendations across various CPG developers for the same clinical scenario.¹⁶

Literature search reveals high interest among academics in reviewing CPG’s quality for their specialty areas with numerous studies on the topic. Interestingly, very few reviews have been conducted regarding the CPGs available for PAD. Also, to our knowledge, the available reviews focused on aspects of the PAD guidelines such as reviews on screening recommendations,¹⁷ reviewing the quality of the CPGs and reviewing the pharmacologic recommendations.^{18 19} In this study, we aim to conduct a more exhaustive review of the most recent guidelines (written in the last 10 years).

In 2012, a systemic review was conducted on eight guidelines published between 2003 and 2011, comparing their quality and recommendations for PAD screening. The study results revealed that the majority of the guidelines favoured screening for PAD. However, three guidelines did not advocate for PAD screening due to the absence of appropriate clinical trials. The studies were considered inappropriate because the available clinical trials were conducted on individuals with established PAD and were unsuitable to be the basis for clinical advice for the general population. The guidelines’ quality was also assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE-II) tool, with results revealing a range between 33% and 81%.¹⁷

Further work was done on this topic by another team of researchers who reviewed seven guidelines written between 2006 and 2012. The study focused on the quality of the guidelines using the AGREE-II instrument. Their results revealed a significant variation similar to the 2012 study with a range of 45%–72%. The reviewed CPGs were found to have high scores in clarity and editorial independence but with low scores in applicability and rigour of development. In their recommendations, they stated that only one CPG could be recommended for use without modification.¹⁹

The most recent review assessed CPGs written between 2000 and 2017. This study was a more exhaustive review. They assessed the guideline quality using the AGREE-II instrument and the recommendations across screening and pharmacological management. The result revealed a quality range between 39% and 73%, similar to the earlier reviews. However, this work found the CPGs to have low scores in the rigour of development (similar to the previous study) and editorial independence (unlike the previous study where they scored high marks). This difference may be because Chen and colleagues reviewed more CPGs. It was also observed that just two of the CPGs reached the standard for conflict of interest from the Institute of medicine. Regarding the screening recommendations, 8 guidelines out of 14 recommended screening (at different strengths) while the others stated insufficient evidence or were against it. Treatment recommendations also showed conflicts concerning target values for lipid-lowering and antiplatelet therapy.¹⁸

In summarising these findings, the PAD guidelines show considerable variation in quality and variations in their recommendations. The paucity of high-quality research could explain these variations for the specific topics for which recommendations are needed, prompting the need for reliance on lower strengths of evidence such as expert consensus or research conducted on established disease participants. Clearly, there is a knowledge gap that can easily be filled with the right form of interest from the research community.

The rationale for the study

Systematic reviews of CPGs are used to systematically identify, assess and summarise the current state of guidance on a clinical topic. Well-written systemic reviews that adhere to a rigorous methodological approach and use transparent reporting to identify knowledge gaps where improvement in current recommendations can be achieved.²⁰

The previous reviews on the CPGs for PAD have revealed a wide variation in the quality and variations in screening and pharmacologic management recommendations.^17–19 However, these reviews were restricted in their comparators, focusing on aspects of the CPGs rather than performing a more holistic review.

Furthermore, the previous reviews included CPGs written over a wide range of time. Advancements in treatment options of atherosclerotic diseases have advanced considerably in the last decade, with consequent paradigm shift occurring after the results of relatively recent randomised clinical trials. We expect that this will be reflected in more recent CPGs compared with their older counterparts.

The findings of this review will be compared with those of the previous reviews. Significant areas of interest, such as changes in overall quality over time and changes in the strength of pharmacological management recommendations, will be made manifest. Also, a nouvelle comparison of non-pharmacological management will be conducted across the guidelines.

Aim

A quality assessment and comparative analysis of the CPGs on PAD written between 2010 and 2020 to assess the quality of the CPGs and identify the gaps in evidence as reflected by the nature of their recommendations.

Objectives

1. To compare the quality of the CPGs on PAD written between 2010 and 2020 using the AGREE-II instrument.

2. To compare the recommendations for screening for PAD across CPGs on PAD written between 2010 and 2020.

3. To summarise the recommendations for pharmacologic management across CPGs on PAD written between 2010 and 2020.

4. To critically appraise the non-pharmacologic recommendations across CPGs on PAD written between 2010 and 2020.

5. To collate and contrast the follow-up recommendations across CPGs on PADs written between 2010 and 2020.

Methodology

Patient and public involvement statement

Patients who are members of the Peripheral Arterial Diseases Support Group (https://www.facebook.com/groups/pad.pvd.support/members) were involved in this study’s design (in modelling the research objectives) and will be involved in the study when it commences. The Way to My Heart.org (https://www.thewaytomyheart.org/) founded this support group. The patient public involvement will be coordinated through the group’s leaders/founders (also patients themselves are actively involved in providing support to their fellow patients) who are advisory members to the research team. They have identified this research as a priority area for clinicians who provide care to patients living with PAD. The group members will be informed of this study’s results through their group page on Facebook in a newsletter suitable for a non-specialist audience. The patients and public will also be sought in the development of an appropriate method of dissemination.

Guideline identification

A systematic search will be conducted, and eligible guidelines selected based on the attributes listed in table 1. These selected guidelines will be comparatively assessed across quality and recommendations. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement will be used as a reference to report items and results in this review.²¹

Table 1.

Population, Clinical Indication, Comparators, Attributes of Eligible guidelines, Recommendation characteristics (PICAR) statement

	Study-specific criteria
Population/clinical condition	Adults 18 and above, with
	peripheral arterial disease
Intervention	All forms of management.
Comparators	No comparator. All aspects of PAD management will be taken into consideration in the comparisons
Attributes of eligible CPGs	Language; no restriction
	Time range; published from 2010 to 2020
	Publishing region; global
	Versions; latest versions only
	Development process; explicitly evidence-based
	System of rating evidence; must be available and stated clearly
	Scope; to cover all aspects of PAD management
	Recommendations; must be available and clearly stated
Recommendation characteristics	Recommendations covering screening, diagnosis, pharmacological and non-pharmacological management are of interest.
	Levels of confidence; an explicit level of confidence must accompany each recommendation
	Locating recommendations; within the CPG’s texts, tables, algorithms and or decision paths

CPG, clinical practice guideline; PAD, peripheral arterial disease.

One reviewer will perform the search and extraction for recommendations, which will be validated by another reviewer. A third reviewer will be consulted to resolve disagreements if they arise. The AGREE-II instrument will be used to assess the quality of the selected guidelines by four reviewers. One reviewer will extract the recommendations, and another reviewer will validate this.

Search strategy

A systematic search will be performed to identify relevant CPGs on PAD. A concept table will be used to generate appropriate search terms (MeSH, Free text vocabulary, Key Words) depending on the database’s peculiarities.

The searches will be conducted on the following databases:

1	Medical databases	PubMed
		Scopus (which includes Embase and MEDLINE)
		TRIP
		Cochrane
2	Guideline developer website	NICE
		SIGN
		National Library of Medicine—National Institute of Health (USA)
		Canadian Medical Association Infobase
		NewZealand Guidelines Group
		Guidelines International Network
		National Guidelines Clearinghouse
3	Expert contributions/websites of specific societies

Example: A draft of the search strategy for PubMed via MEDLINE.

1. Arterial Disease, Peripheral.

2. Arterial Diseases, Peripheral.

3. Disease, Peripheral Arterial.

4. Diseases, Peripheral Arterial.

5. Peripheral Arterial Diseases.

6. Peripheral Artery Disease.

7. Artery Disease, Peripheral.

8. Artery Diseases, Peripheral.

9. Disease, Peripheral Artery.

10. Diseases, Peripheral Artery.

11. Peripheral Artery Diseases.

12. Peripheral Arterial Disease [MeSH].

13. Intermittent Claudication [MeSH].

14. Limb Ischemia.

15. 1 OR 2OR 3 OR 4 OR 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14.

16. Screening.

17. Treatment.

18. Management.

19. Diagnosis.

20. Pharmacological.

21. Diagnosis[MeSH Terms].

22. Therapy[MeSH Terms].

23. 16 OR 17 OR 18 OR 19 OR 20 OR 21 OR 22.

24. Guidelines.

25. Guideline.

26. Standards.

27. Practice guideline[MeSH Major Topic]

28. 24 OR 25 OR 26 OR 27.

29. Quality.

30. Recommendations.

31. Quality improvements[MeSH Terms]).

32. 29 0R 30 OR 31.

33. 15 AND 23 AND 28 AND 32.

The search was conducted on 11 December 2020. Result: 7014 references.

Guideline selection

The extracted references will be searched through the title and abstract for guidelines that meet the inclusion and exclusion criteria outlined below. This selection will be done by the lead researcher and verified by another researcher. Conflicts of ideas will be resolved by consensus by taking a third researcher’s opinion to minimise selection bias risk.

Inclusion criteria

1. The guideline is developed for people with PAD.

2. The guideline covers recommendations regarding screening, non-pharmacological and pharmacological interventions.

3. The guidelines were written between 2010 and 2020.

4. The guideline is the most recent version.

5. The guideline is available online.

6. Related or international academic organisations wrote the guideline.

Exclusion criteria

1. The topic is only mentioned in the guideline.

2. The guideline is limited to a specific aspect of PAD management, such as screening, pharmacologic management, etc.

Outcomes: the outcomes in this study are

1. Guideline quality.

2. Guideline recommendations.

Quality assessment

Instrument

The updated AGREE-II instrument (online supplemental appendix 1) will be used to assess the quality of the selected guidelines.²² The AGREE-II instrument is a 23-item tool with international certification that evaluates the six domains of methodological quality of a guideline, including scope and purpose, stakeholder involvement, rigour of development, clarity of presentation and applicability and editorial independence.²³ The assessment will be conducted by four reviewers (as recommended by the developers of the tool to minimise bias) using the instrument to assess all selected guidelines. The reviewers will score each guideline across each domain on a Likert scale of 1 through 7 (from strongly disagree to strongly agree).

In addition, the reviewers will give an overall score of the guidelines on a similar Likert scale. As such, each guideline will have two sets of scores: (1) the domain scores and (2) the overall score for the guideline.

Scoring

Domain scores are calculated by summing up all the individual items’ scores in a domain and scaling the total percentage of that domain’s maximum possible score. The example on scoring below was extracted from the user manual.

To give an example, if four appraisers give the following scores for domain 1 (dummy scores are generated for scope and purpose in table 2):

Table 2.

Example of domain scoring for scope and purpose

	Item 1	Item 2	Item 3	Total
Appraiser 1	5	6	6	17
Appraiser 2	6	6	7	19
Appraiser 3	2	4	3	9
Appraiser 4	3	3	2	8
Total	16	19	18	53

Maximum possible score=7 (strongly agree) × 3 (items) × 4 (appraisers)=84.

Minimum possible score=1 (strongly disagree) × 3 (items) × 4 (appraisers)=12.

The scaled domain score will be:

$${\displaystyle \frac{\mathrm{O}\mathrm{b}\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}-\mathrm{M}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}{\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}-\mathrm{M}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}}$$

$${\displaystyle \frac{53-12}{84-12}\times 100=\frac{41}{72}\times 100=57\mathrm{\%}}$$

Interpreting domain scores

There are no fixed cut-offs for high-quality or low-quality guidelines set by the instrument developers. The scores of the domains will be compared against each other between the guidelines. The overall assessment will be arrived at using the domain scores, and for this purpose, we have decided to set out cut-offs in line with the study conducted by Chen and colleagues because of its practicability.¹⁸ If most (four or more) domains scored over 60%, a guideline would be regarded as ‘strongly recommended for use in practice’; if scores of most domains (four or more) ranged 30%–60%, the guideline would be regarded as ‘recommended for use with some modification’; if most of the domains (four or more) scored less than 30%, the guideline would be regarded as ‘not recommended for use in practice.’

Interpreting the overall guideline scores

This will be used as an additional matrix for assessing the guideline as a supporting statistic. It will not provide any direct contribution for the final assessment into high-quality or low-quality guidelines.

Data extraction and management for quality scores

The data from each appraiser for the AGREE instrument will be entered into an initial excel sheet for upload into SPSS V.22 for analysis. The four appraisers’ scores will be aggregated within the SPSS datasheet in line with the formula highlighted above. The final scores, which will be used to generate the recommendation for using the guidelines, will be presented in the Results sections. The preliminary datasheet templates are attached below (see online supplemental appendix 2).

Guideline recommendations

Recommendations extraction

A recommendation matrix will be developed based on the focus areas of the data synthesis in line with the research objectives. The recommendations will be extracted across screening, pharmacological and non-pharmacological treatment modalities for comparative assessment. Systematic methodology will be employed to harmonise specific details of the guidelines, which may vary due to differences in terminology or differences in interventions/comparators. For example, recommendations will be harmonised into themes (thematic analysis), which can then be coded and entered into the software/datasheet.

Particular interest will be paid to the level of evidence on which the recommendations are based. A preliminary review of some guidelines shows variations in the grading system for the level of evidence. The evidence grading schemes for each guideline will be harmonised and standardised to enhance the data synthesis process. Evidence categories will be developed using an iterative process of refinement through discussions within the review team.

Other characteristics of interest, such as the data aimed at evaluating the financial relationship between guideline producers and the biomedical industry, and others outlined in the PICAR statement, will be extracted. A preliminary version of what the data extraction sheet will appear like is attached below, highlighting all the variables that will be extracted (see online supplemental appendix 3). As the study progresses, the datasheet is bound to evolve to fit the study’s objectives better.

Recommendation data management

The recommendation data will be extracted using Nvivo software for qualitative data extraction and management. The extracted information will be summarised through qualitative/thematic analysis. The variables of interest are listed in table 3.

Table 3.

List of variables

S/no	Name of variable	Definition
1	Guideline name	Title of the published guideline.
2	Guideline organisation/society	The name of the organisation responsible for the publication of the guideline
3	Year	Year of publication
4	Funding	Source of funding for guideline production.
5	Country	The country where the guideline was produced
6	Target users	Endusers of the guideline
7	Guideline writers	The authors
8	Evidence grading system	The system used to grade the evidence on which the recommendations are made
9	Recommendations	The recommendations that were made in the guidelines for specific clinical scenarios.
10	Level of evidence	The strength of the evidence used in making a particular recommendation
11	Strength of recommendation	The level of confidence in the accuracy of the recommendation
12	Domain 1	First domain of the AGREE-II instrument; scope and purpose
13.	Domain 2	Second domain of the AGREE-II instrument; stakeholder involvement
14	Domain 3	Third domain of the AGREE-II instrument; rigour of development
15	Domain 4	Fourth domain of the AGREE-II instrument; clarity of presentation
16	Domain 5	Fifth domain of the AGREE-II instrument; applicability
17	Domain 6	Sixth domain of the AGREE-II instrument; editorial independence
18	Overall score	The appraisers overall score for the guideline
19	Cumulative scores for domains 1–6	The aggregate of the scores from the four reviewers
20	Cumulative of the overall score	The aggregate of the overall scores from the four reviewers
21	Final guideline recommendation	The final recommendation for the guideline based on the overall percentage score.

AGREE II, Appraisal of Guidelines for Research and Evaluation II.

Results

1. Flowchart of search strategy.

2. Results of quality assessment using AGREE-II represented by bar charts/histograms, also +overall recommendations.

3. A tabular summary of screening recommendations for PAD.

4. A tabular summary of non-pharmacological recommendations for PAD.

5. A tabular summary of pharmacological recommendations for PAD.

6. Additional relevant information on the guidelines.

The study is proposed to be completed within a period of 26 weeks, with dedicated attention from all participants. The activity breakdown and allotted time for each activity are shown in table 4.

Table 4.

Timeline

1	Title adoption	Done
2	Develop protocol	4weeks
3	Study search	2weeks
4	Study selection	2 weeks
5	Data extraction—AGREE-II+recommendation extraction	8 weeks
6	Data analysis	2 weeks
7	Write up and discussion	4 weeks
8	Review and discussion	4 weeks
Total		26 weeks

AGREE II, Appraisal of Guidelines for Research and Evaluation II.

Significance of the study

This study’s significant finding will be identifying low-grade recommendations in the available guidelines (recommendations based on low-level evidence). The only way to remedy this situation is for researchers to conduct appropriate-sized randomised controlled trials tailored to answering the recommendations’ problems. These shortcomings will be highlighted in the results and discussions, paving the way for improved PAD CPGs in the future.

The results of this study will also serve as a guide for future CPG writers to pay attention to all aspects of CPG development, especially domains where they performed poorly in the quality assessment using the AGREE-II instrument.

Ethics and dissemination

Because this is a systematic review and no human subjects, we do not see the need to seek ethical approval.

We aim to disseminate this work through a journal publication and conference presentation. The work will also be disseminated through our Patient and Public Initiative Network.

Ethics statements

Patient consent for publication

Not required.