STANDING Collaboration: a study protocol for developing clinical standards

Abstract

Introduction

Despite widespread availability of clinical practice guidelines (CPGs), considerable gaps continue between the care that is recommended (‘appropriate care’) and the care provided. Problems with current CPGs are commonly cited as barriers to providing ’appropriate care'.

Our study aims to develop and test an alternative method to keep CPGs accessible and up to date. This method aims to mitigate existing problems by using a single process to develop clinical standards (embodied in clinical indicators) collaboratively with researchers, healthcare professionals, patients and consumers. A transparent and inclusive online curated (purpose-designed, custom-built, wiki-type) system will use an ongoing and iterative documentation process to facilitate synthesis of up-to-date information and make available its provenance. All participants are required to declare conflicts of interest. This protocol describes three phases: engagement of relevant stakeholders; design of a process to develop clinical standards (embodied in indicators) for ‘appropriate care’ for common medical conditions; and evaluation of our processes, products and feasibility.

Methods and analysis

A modified e-Delphi process will be used to gain consensus on ‘appropriate care’ for a range of common medical conditions. Clinical standards and indicators will be developed through searches of national and international guidelines, and formulated with explicit criteria for inclusion, exclusion, time frame and setting. Healthcare professionals and consumers will review the indicators via the wiki-based modified e-Delphi process. Reviewers will declare conflicts of interest which will be recorded and managed according to an established protocol. The provenance of all indicators and suggestions included or excluded will be logged from indicator inception to finalisation. A mixed-methods formative evaluation of our research methodology will be undertaken.

Ethics and dissemination

Human Research Ethics Committee approval has been received from the University of South Australia. We will submit the results of the study to relevant journals and offer national and international presentations.

Strengths and limitations of this study.

• We will develop and evaluate a method for generating and ratifying clinical standards and indicators of ‘appropriate care’ for common health conditions which has been designed to overcome deficiencies in current methods.

• This study will obtain expert consensus on ‘appropriate care,’ underpinned by evidence, for a range of common medical conditions.

• The recruitment of healthcare professionals, patients and consumers to review clinical indicators may introduce selection biases.

• The use of English language clinical practice guidelines may not be representative of all available evidence, and limits the generalisability of study findings.

Introduction

In Australia, ‘appropriate care’ (care in line with evidence or consensus-based guidelines) is provided to adults, on average, only 57% of the time, with large variations across common medical conditions and providers.¹ Problems with clinical practice guidelines (CPGs), standards and indicators (see box 1 for definitions) are commonly cited as one of the barriers to providing appropriate care²; these include large numbers of repositories and guidelines; duplication and overlap among guidelines; differing recommendations for care practices; lack of currency; inconsistent structure and content; voluminous documents which are not easy to assimilate or use^3–5; and recommendations which are often vague and difficult to measure.^{2 6 7} In addition, most CPGs lack detail of how evidence was interpreted and weighted to formulate recommendations, offer little opportunity for end-users to provide formal feedback⁸ and have been developed by people with (often undisclosed) professional or commercial conflicts of interest (COIs).^{2 6 9}

Box 1. Definitions for clinical practice guideline, standard, indicator and tool.

A clinical practice guideline:

‘Statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.^{10 11}

A clinical standard²:

• is an agreed process that should be undertaken or an outcome that should be achieved for a particular circumstance, symptom, sign or diagnosis (or a defined combination of these)

• should be evidence based, specific, feasible to apply, easy and unambiguous to measure, and produce a clinical benefit and/or improve the safety and/or quality of care, at least at the population level.

If a standard cannot or should not be complied with, the reason/s should be briefly stated.

 A clinical indicator²:

• describes a measurable component of the standard, with explicit criteria for inclusion, exclusion, time frame and setting.

A clinical tool2 6 12–14:

• should implicitly or explicitly incorporate a standard or a component of a standard

• should constitute a guide to care that facilitates compliance with the standard

• should be easy to audit, preferably electronically, to provide feedback

• should be able to be incorporated into workflows and medical records.

Implementability of CPGs is a key factor affecting their perceived utility and uptake^{11 15}; in response, international efforts are being directed at developing clinical standards and indicators to identify evidence and service delivery gaps and areas for improvement, and understand and measure the quality of care provided.^{16 17} Emerging schools of thought suggest that ‘appropriate care’ may be enhanced through greater patient (health consumer) engagement.^{12–14 18} This could be facilitated by involving patients and interested laypeople as well as healthcare professionals (HCPs) in CPG development,^{6 19 20} and using online technologies to enhance transparency, accessibility and currency of both content and development processes.^{6 21}

The strategies employed in this protocol aim to mitigate problems with existing CPG development processes (table 1) by adopting a single approach to avoid duplication, using an ongoing and iterative documentation process to facilitate transparent synthesis of up-to-date information and make its provenance accessible, and requiring all participants to declare their COIs.² Indicators will be developed using selection criteria to reflect ‘essential’ clinical practice and be expressed one concept at a time in plain English to help create standards that are clear, concise, measurable and easy to use.

Table 1.

Key aspects of the STANDING Collaboration methodology, designed to mitigate issues with current CPG development

CPG development issue	STANDING Collaboration methodology and rationale
Large numbers of repositories and guidelines2 6	Single approach negates the need for, and replaces aggregated findings from, a large number of repositories and guidelines
Duplication and overlap2
Different recommendations for care practices2 6	Methodology will produce a single set of nationally agreed evidence-based clinical indicators, representative of ‘appropriate care’ for a range of common medical conditions, reflecting consensus recommendations for care practices
Lack of currency2 6	Methodological vehicle (interactive, live and online wiki-based platform) will facilitate ongoing review and ease of updating
Inconsistent structure and content2	Uniform methodological approach and format will ensure consistent structure and content
Hard-to-use voluminous documents2	Methodological vehicle (live and online wiki-based platform) and consistent organisation of indicators according to phases of care will facilitate ease of access and use
Hard-to-measure recommendations2	Uniform methodological approach will ensure consistent structure and content for indicators, which will be formatted to ensure ‘measurable components of the standard, with explicit criteria for inclusion, exclusion, time frame and setting’2
Management of conflicts of interest2 6 9 36 49	Research methodology requires participants to declare any conflicts of interest, and has a defined management strategy for these The source and provenance of each standard, indicator and suggestion will be logged and can be viewed
Inclusive, catering for all target audiences19 23	This method allows for all healthcare professional types and patients or consumers to register and review indicators
Lacking details on how evidence was sourced, interpreted and managed to formulate recommendations8	All reviewers’ comments and recommendations will be logged, classified and presented in subsequent rounds according to whether and why they have been incorporated into the next iteration This allows tracking of the evolution of the standards and indicators from the original recommendations on which they were based to their final iteration, as well as the nature and influence of review feedback in shaping the standard

CPG, clinical practice guideline.

The aims of the STANDING Collaboration study are to (1) provide proof of concept for an alternative method for creating sets of nationally-agreed evidence-based standards and clinical indicators, and (2) obtain consensus on ‘appropriate care’ for a range of common medical conditions. To do so, we will use a three-phase approach to engage relevant stakeholders, develop clinical indicators representative of ‘appropriate care’ (which constitute the standard) for a range of common conditions, and evaluate our processes, products and feasibility. We plan to develop an inclusive, transparent, collaborative process, which allows HCPs and patients or consumers to develop and keep up-to-date clinical standards comprising indicators with defined attributes, using an online curated wiki-based platform to facilitate ongoing review and updating of the standard, or individual indicators, as soon as new evidence emerges. In this study, the term ‘wiki’ refers to an interactive information management system which will allow users (eg, HCPs and patients) to collaborate directly in formulating and refining indicators that are relevant to their clinical practice and lived experience.^{21 22} The source and provenance of each indicator, including all suggestions, will be posted online and updated as necessary.

Methods and analysis

Our three-phase approach (figure 1) comprises:

Figure 1.

Overview of STANDING Collaboration research methodology and definitions for participant groups. CPG, clinical practice guideline.

1. stakeholder analysis;

2. development and test of a process for creating clinical indicators representative of ‘appropriate care’ for a range of common conditions;

3. evaluation of processes, products and feasibility.

Phase 1

In order to gain an understanding of potential barriers, facilitators and the overall feasibility of the STANDING Collaboration methodology, stakeholder perspectives will be captured through a series of semistructured qualitative telephone interviews. HCPs and consumers will be invited to participate. Relevant medical colleges, professional and consumer associations and networks will be contacted, using publicly available information, to request assistance with the recruitment of interview participants. Invitations will comprise email notifications to members and media releases and articles within newsletters, asking potential participants to contact the Research Team members. The telephone interviews will be conducted at a time convenient to the research team member and the participant. Based on the sample sizes reported in similar stakeholder analyses,^{23 24} we anticipate conducting approximately 18–25 interviews in total (9–13 interview participants per stakeholder type), or until saturation is reached.

Using guidelines for stakeholder analyses²⁵ and the schedules from previous qualitative research as a guide,^{19 24} a range of topics will be explored in the interviews (table 2).

Table 2.

Topics (and their rationale) for exploration in the qualitative stakeholder interviews

Interview topic	Rationale
Strength and limitations of current CPGs, clinical standards and indicators (eg, in terms of development, availability and utility)	Background information regarding participants’ understanding of the development, availability and utility of CPGs
Barriers, facilitators and the overall feasibility of the STANDING Collaboration methodology	To refine the STANDING Collaboration methodology according to stakeholders’ needs
Possible integration of standards into patient decision support technologies and what these would comprise	To tailor the content and format of clinical standards and indicators to optimise fitness for purpose
Priorities for standard development topics (ie, medical conditions)	To determine methods and data sources for selecting priority medical conditions for indicator development in the STANDING Collaboration study

CPG, clinical practice guideline.

Interviews will be recorded with consent and transcribed by a professional transcription company. Transcripts and summaries will be returned to participants for the purpose of making comments or corrections and providing feedback on the findings.²⁶ Content analyses will be used to derive common themes. We will use open coding and inductive reasoning with two coders to group similar responses into categories and assign labels capturing specific themes. Any discrepancies will be discussed among Research Team members. After agreement is reached on the composition and label for each category, we will assign (axial coding) categories to the central phenomena of interest (table 2).^{27 28}

Phase 2

Clinical indicators will be developed for individual conditions using a four-stage process:

1. source, select and search relevant CPGs;

2. extract all concepts from each CPG together with the relevant text in which they appear (original recommendation), and tabulate common concepts to select, draft and format the proposed clinical indicators based on identified concepts;

3. review the indicators internally;

4. review the indicators externally.

Stage I: source, select and search relevant CPGs

Interview data from the stakeholder analysis (phase 1) will be used in conjunction with national health priority areas, burden of disease and prevalence data to identify candidate conditions for clinical standard and indicator development.^{1 7} Clinical indicators will be drawn initially from the latest CPGs. A systematic search will be undertaken of national-level Australian CPGs endorsed by the National Health and Medical Research Council, and international-level guidelines from the National Institute for Health and Care Excellence in England, Scottish Intercollegiate Guidelines Network, and the Agency for Healthcare Research and Quality’s National Guideline Clearinghouse in the USA, and the Guidelines International Network. In the absence of Australian national or international CPGs, relevant professional medical college and association CPGs may also be searched, as well as those published at state or professional level and in international journals. Details of the search strategy are provided in appendix A, online supplementary file 1.^{29 30} In order to describe the quality of the evidence sources from which our clinical indicators will be derived, two members of the Research Team will independently appraise CPGs using the AGREE-II (Appraisal of Guidelines for Research and Evaluation II) tool.^31–34

Stage II: select, draft and format proposed indicators

Recommendations (and their key underlying concepts) from each CPG will be collated and used to inform the content of the proposed clinical indicators. Not all recommendations published in CPGs will become indicators. Recommendations will be flagged for potential exclusion based on the following criteria:

• strength of the wording of the recommendation (ie, ’may' and ’could' statements would be excluded; ’should' and ’must' statements would be included)

• vague guiding or aspirational statements and those without recommended actions

• conflicting recommendations from less recent CPGs and those with lower AGREE-II scores.

All clinical indicators will be written in plain English, one concept at a time, using a structured and standardised format (eg, commencing with the inclusion criteria followed by the compliance action)⁷ (table 3). For each condition, indicators will be arranged according to phases of care (ie, screening, diagnosis, assessment, acute care, ongoing care) so that together, they constitute a clinical standard amendable to inclusion as a clinical tool over the patient journey.

Table 3.

Examples of clinical indicators from CareTrack Australia ¹ that were written in plain English, one concept at a time using a structured and standardised format

Condition	Original recommendation (source)	Level of evidence	Indicator
Asthma	Treatment with a preventer medication is recommended for patients who have asthma symptoms more than three times per week or use a SABA more than three times per week50	I	Patients who reported asthma symptoms more than three times per week were prescribed preventer medication (Seretide, Symbicort, Flixotide)
			Patients with asthma who report using SABA more than three times per week were prescribed preventer medication
Hypertension	Initiate antihypertensive drug treatment immediately in patients with hypertension with any of the following: grade 3 hypertension or isolated systolic hypertension with widened pulse pressure (SBP ≥160 mm Hg and DBP ≤70 mm Hg) associated conditions or evidence of end-organ damage (regardless of BP) high absolute risk of cardiovascular disease, based on the presence of markers of high risk or as estimated using a risk calculator51	Consensus-based recommendation	Patients with hypertension and isolated SBP ≥160 and DBP <70 or with widening pulse pressure are prescribed antihypertensive therapy
			Patients with hypertension and end-organ failure are prescribed antihypertensive therapy
			Patients with hypertension and at high risk (diabetes, previous stroke/CVA or chronic kidney disease) of developing coronary heart disease are prescribed antihypertensive therapy

BP, blood pressure; CPG, clinical practice guideline; CVA, cerebrovascular accident; DBP, diastolic blood pressure; SABA, short-acting beta2-agonist; SBP, systolic blood pressure.

Stages III and IV: iterative review using a modified e-Delphi approach

Stages III and IV will involve online wiki-based reviews of proposed clinical indicators; initially by way of an internal review between the Research Team and Curator Group (stage III), followed by an external review by the Curator Group and HCPs and consumers who choose to register to the site (Wiki Registrants) and provide feedback on the proposed clinical indicators.⁷ This approach has been chosen to facilitate the number and spread of individuals’ perspectives obtained and avoid domination of the consensus process by one or a few participants, and to optimise face validity of the final set of clinical standards and indicators.^{7 35} All STANDING Collaboration participants (Research Team, Clinical Champion, Curator Group members, and Wiki Registrants) will be required to complete a COI declaration, which will be taken into consideration when accepting or rejecting suggestions, and logged with the provenance of each indicator for transparency.^{9 36}

Stage III: internal review processes

Internal reviews will first be conducted within our Research Team, and subsequently by our Clinical Champion and Curator Group members who will comprise a mix of at least two members of the following: clinicians (eg, general practitioners, medical specialists, allied health professionals, nurses), researchers, policymakers or public health specialists or healthcare quality improvement experts, and consumers. Depending on their self-reported scope of practice, expertise and interest, Curator Group members (including consumers) may be able to participate in review panels for more than one condition. The total number of invitations to potential Curator Group members will depend on the skill mix of those invited and the overall pattern of recruitment. This selection strategy is supported within the Delphi process literature with studies using similar criteria to choose potential participants (eg, renown, member of an organisation, recommendation, years of experience, willingness to participate, availability, interest).^{7 35}

The internal review will consist of a maximum of three rounds to allow sufficient testing of the subsequent phase of the process (ie, external review). In the first round, drafts of proposed clinical indicators, and the recommendations on which they are based, will be sent via email to the Curator Group members. The review criteria to be used are based on the methods from previous studies for developing and measuring indicators of appropriate care.1 7 16 37–39 Curator Group members will be asked to: recommend indicators for inclusion (with or without amendments) or exclusion, provide comments in relation to three key criteria: evidence, feasibility and importance (appendix B, online supplementary file 2) and make additional suggestions (with supporting material). In addition, Research Team members will pose specific questions to the Curator Group members about individual indicators to highlight inconclusive or conflicting CPG recommendations, or to clarify definitions for inclusion criteria and compliance actions. In particular, consumer members of the Curator Group will be asked to vet the plain English wording of clinical indicators and a linked glossary of terms to ensure that content is appropriately targeted to the consumer audience. In this round, Curator Group members will complete their assignments independently to minimise ‘group-think.’^{40 41} Research Team members (WBR, PDH, LKW, JHS) will collate the feedback and revise the content, structure and format of each indicator. The refined set of indicators (including the original indicators and any feedback and suggestions) will be sent to the same Curator Group members for a second round of scoring. The same approach will be used in the second round, with a request for further refinement and identification of indicators to be included or excluded. If necessary, Curator Group members will discuss the proposed set of indicators via a third round teleconference, with a view to achieving consensus and approving the indicators for the external online wiki-based review process.

Stage IV: external review processes

External reviews will be conducted by HCPs and consumers who have registered to this wiki as reviewers (Wiki Registrants). Relevant medical colleges, professional and consumer associations and networks will be contacted to request assistance with the identification of potential clinical indicator reviewers. Invitations will be by email, media releases and articles within newsletters. HCPs and consumers will self-nominate as reviewers for one or more of the STANDING Collaboration conditions based on their interests, scope of practice and experience.^{35 42} Wiki Registrants for this process will be required to declare their COIs, which will be taken into account by the Clinical Champion and Curator Group when considering reviewers’ feedback on the indicators.^{9 36}

The external review will involve an interactive wiki-based process where indicators for each condition from round 3 of the internal review will be posted on an online wiki site. A software development company will be engaged to purpose design and custom build the wiki for this project. The wiki ‘live’ time for each version will depend on the recruitment rate of reviewers and the progress of their reviews, but is anticipated to be no longer than 3 months per round. Reviewers will provide comments on indicators in relation to the three key criteria: evidence, feasibility and importance (appendix B, online supplementary file 2), make recommendations (ie, inclusion, inclusion with amendments, exclusion, hold) and be able to suggest edits in real time. The Clinical Champion and Curator Group for each condition will follow-up and manage external reviewers’ responses, and make final recommendations for that version regarding the inclusion, content, structure and format of indicators. The Clinical Champion and Curator Group will use supporting references when considering and responding to each suggestion related to whether and why they have been included or rejected. In addition, all external reviewers’ comments and recommendations will be logged, classified and presented in subsequent rounds according to whether and why they have or have not been incorporated into the next iteration. This will allow tracking of the evolution of the standards and indicators from the original recommendations on which they were based to their final iteration, as well as the nature and influence of review feedback in shaping the standard. Once the indicators are ‘stable’ with no further significant changes being suggested, that version of the standard will be published as comprising a set of clinical indicators that represents ’appropriate care' for Australians with the candidate conditions at that time. Endorsement will then be sought by relevant professional bodies and consumer organisations. For each medical condition that has undergone indicator development via the STANDING Collaboration process, it will be possible for evidence to be monitored by the Curator Group (or a subgroup comprised of key members of the Curator Group) in order to update standards and indicators as necessary. For each condition, our initial monitoring plan involves using information from automated database searches and feedback from the wiki to initially update indicators every three months and, once stable, at a minimum of every six months.

Phase 3

A multimethods evaluation of the process and products of phases 1 and 2 of our research methodology will be undertaken (table 4). Three data sources will be used to inform the evaluation: (A) engagement and utilisation statistics sourced from the wiki logs—these will include demographics of users and rates, times and nature of use; (B) the nature and content of Wiki Registrant, Curator Group and Clinical Champion comments (eg, the format and rationale of proposed changes to indicators, level of agreement between reviewers and resulting changes to the indicators); and (C) all users and stakeholders’ perspectives on the process, usability and appropriateness of the vehicle for developing the standards (ie, the wiki) as well as the acceptance and utility of the final sets of standards and indicators themselves.

Table 4.

Description of the phase 3 evaluation of phase 1 and phase 2

Phase 3: Evaluation of processes, products and feasibility	Phase 1: Stakeholder analyses	Phase 2: Development of clinical indicators representative of ‘appropriate care’ for a range of common conditions
Aim and purpose of evaluation	Participants’ perceptions and experiences of the engagement process	Participants’ engagement with, and utilisation of, clinical indicator development process Participants’ perspectives on the process, usability and appropriateness of the clinical indicator development approach used, and the final sets of standards and indicators
Participants	Phase 1: stakeholders	Phase 2: internal reviewers (Research Team, Clinical Champions, Curator Group members) Phase 2: external reviewers (Wiki Registrants)
Methods	Qualitative interviews	Database (wiki) usage and content analyses Qualitative focus groups (internal reviewers) Online survey (external reviewers)
Data sources	Interview data	Wiki logs Interview and survey data
Analysis	Content analysis Inductive reasoning	Descriptive statistics (wiki logs, online survey) and content analyses (wiki, interview and survey data)

Participants will be invited to provide feedback regarding their experiences and perspectives via one of three data collection methods: (1) online user perspectives survey (for external reviewers); (2) semistructured interviews (for Review Panel members and the Clinical Champion); and (3) interviews and focus groups (for phase 1 stakeholders, following publication of the sets of indicators) (table 4). Both quantitative and qualitative analyses of these three data sources will be undertaken including: descriptive statistics for the characteristics of wiki users, their engagement and patterns of use, frequency counts and content analyses of ratings and free-text responses from the online user perspectives survey, semistructured interviews and focus groups (table 4). Using these data, recommendations will be developed regarding the overall feasibility of the wiki process for future indicator development. The next phase of the research is to study the implementation of the standards as a translational tool for clinical practice and ongoing audit.

Ethics and dissemination

Ethical approval

Human Research Ethics Committee approval has been granted from the University of South Australia (protocol number 0000035183). All STANDING Collaboration participants and reviewers will be required to give informed consent (for their chosen study phase(s)) and complete a COI declaration prior to participation, which will be recorded and managed according to an established protocol (appendix c, online supplementary file 3).^43–46

Dissemination

We will submit the results of the study to relevant national and international journals with the intention of publishing the results widely. As well, we will make national and international presentations to stakeholder groups including those involving patients, researchers, clinicians, managers and policymakers.

Discussion

Notwithstanding the large number of CPGs currently available, delivery of ‘appropriate’ healthcare in Australia and internationally is highly variable and leaves considerable room for improvement. A number of major difficulties have been identified with current CPGs and their development. Our alternative approach for keeping evidence accessible and up to date has been designed to mitigate problems with existing processes. Here, we describe a protocol for developing and testing a process for creating clinical standards, embodied in clinical indicators.

This process has been designed to systematically address many of the problems identified with current CPGs and their development. The approach is characterised by being inclusive (HCPs, researchers and consumers), transparent (all reviewers’ suggestions are logged, with their provenance, as accepted or rejected by the Clinical Champion and Curator Group), up to date (revisions will be ongoing after a version is published), easy to use (one concept per indicator), written in plain English and able to be integrated into the sequence of work flow in managing a condition.^{2 29 30}

Findings from this study will be used to inform the design of future studies using Delphi processes to establish consensus on recommended healthcare, and will be relevant for national and international researchers, policymakers, healthcare practitioners and patients. Specifically, there is potential for standards and indicators developed using this methodology to be assembled to comprise the content of electronic tools for the basic care of common conditions (ie, reflects ‘essential’ Australian clinical practice; appendix B, online supplementary file 2). It is envisaged that the clinical tool (box 1) will²:

1. implicitly or explicitly enunciate the clinical standard for the basic care of the condition in question

2. inform HCPs, patients and carers about that condition

3. guide care

4. document what care has been offered and what has not (and why)

5. be amenable to audit (preferably automated) so that feedback can be provided—at clinical indicator, patient, provider, facility and eventually population levels.

We recognise several limitations to our study. The inclusion of only English language CPGs, for pragmatic purposes and contextual consistency, may not be representative of all available evidence and limits the generalisability of our findings. Reviewers will be invited to participate and self-nominate for conditions that are within their scope of practice, interest or experience (ie, healthcare providers and consumers) which introduces a selection bias. Clinical indicators will be developed from recommendations in existing CPGs, which means there is potential for problems with current CPG development processes to contaminate our final sets of indicators and standards. The aim of this study is to provide proof of concept and test a new methodology. Our intention is that the key approaches and characteristics of the STANDING Collaboration clinical standard and indicator development process (ie, governance structure, transparency, HCP and consumer engagement and codesign, access to provenance of both accepted and rejected suggestions, and use of online technologies to facilitate keeping the indicators up to date) will be universally applicable, and be able to be tailored to other healthcare settings and structures.^{30 47 48} We aim to ameliorate these limitations by adopting a collaborative user-centred approach where feedback from both consumer and HCP groups is sought, communicated transparently and incorporated into guidance for others who wish to develop clinical standards and indicators.⁶