Protocol for the development of a core domain set for hidradenitis suppurativa trial outcomes

Abstract

Introduction

Randomised controlled trials (RCTs) should have well-defined primary and secondary outcomes to answer questions generated by the main hypotheses. However, for the chronic, inflammatory skin disease hidradenitis suppurativa (HS), the reported outcome measures are numerous and diverse. A recent systematic review found a total of 30 outcome measure instruments in 12 RCTs. This use of a broad range of outcome measures can increase difficulties in interpretation and comparison of results and may potentially obstruct appropriate evidence synthesis by causing reporting bias. One strategy for dealing with these problems is to develop a core outcome set (COS). A COS is a list of outcomes that are meant as mandatory and should be measured and reported in all clinical trials. The aim of this study is to develop a COS for the management of HS.

Method and analysis

An international steering group of researchers, clinicians and a patient research partner will guide the COS development. 6 stakeholder groups are involved: patients, dermatologists, surgeons, nurses, industry representatives and drug regulatory authorities. A 1:1 ratio of patients:healthcare professionals is aimed for. The initial list of candidate items will be obtained by combining three data sets: (1) a systematic review of the literature, (2) US and Danish qualitative interview studies involving patients with HS and (3) an online healthcare professional (HCP) item generation survey. To reach consensus on the COS, 4 anonymous online Delphi rounds are then planned together with 2 face-to-face consensus meetings (1 in Europe and 1 in the USA) to ensure global representation.

Ethics and dissemination

The study will be performed according to the Helsinki declaration. All results from the study, including inconclusive or negative results, will be published in peer-reviewed indexed journals. The study will involve different stakeholder groups to ensure that the developed COS will be suitable and well accepted.

Strengths and limitations of this study.

• The number of trials is increasing in hidradenitis suppurativa (HS), and a well-defined core outcome set (COS) will reduce the risk of outcome reporting bias and heterogeneity across studies.

• The protocol is designed to systematically identify what to measure in HS trials (an HS-specific COS).

• The protocol ensures input from major HS stakeholders including patients.

• The protocol and the study involves different stakeholder groups from different countries and cultures.

• Limitations include that the qualitative studies will be performed on Western (Danish and US) patients only. Asian and Eastern European patients will be invited to take part in Delphi surveys to compensate for this.

Introduction

Evidence-based and consensus-endorsed outcome measures are necessary to ensure that study results are comparable and that patients, in consequence, receive the most effective and suitable treatments. Measuring disease activity is therefore crucial to development of new therapies.

Hidradenitis suppurativa (HS) is a chronic inflammatory skin disease characterised by repeated outbreaks of painful inflamed nodules or boils in the apocrine gland-bearing regions (axillae, genital area, groin, breasts and perianal region). Nodules may progress to sinus tracts and scarring.¹ The estimated prevalence is 1–4% worldwide, and HS is three times more common in women than in men.^2–4 HS is associated with significant disability and handicap due to pain and subsequent loss of mobility.⁵ The active condition is associated with malodorous discharge that stains clothing, and HS is therefore accompanied by embarrassment, disabling social stigma, low self-worth and a strong negative impact on interpersonal relationships, education and work.^6–9 Interventions for HS are diverse and include topical treatment, systemic antibiotics, anti-inflammatory therapy, biologics and surgical therapy including laser surgery.¹⁰

Clinical measurements for assessing the severity of HS have generally been based on the Hurley staging system.¹ Other used clinical measures include the ‘Modified Sartorius Score’ (MSS),¹¹ the ‘HS Physician's Global Assessment’ (HS-PGA)¹² and, most recently, the ‘Hidradenitis Suppurativa Clinical Response’ (HiSCR).¹³ A recent Cochrane review identified a substantial heterogeneity of outcome measures in previous randomised controlled trials (RCTs) on interventions for HS.¹⁴

Current selection of outcomes for use in clinical trials on interventions for HS

Clinical trials should have well-defined primary and secondary outcomes to answer questions generated by the main hypotheses. For HS, however, reported outcome measures are numerous with a total of 30 outcome measure instruments in the 12 RCTs included in a recent systematic review.¹⁵ No consensus on core outcomes for HS exists. Researchers therefore use various instruments, which may or may not be representative. This can cause the following problems:

• Heterogeneity in instruments limiting meta-analysis.¹⁶

• Outcome reporting bias involving selective reporting of more favourable outcomes. Empirical evidence of this phenomenon has been highlighted in the literature.¹⁷

One strategy to deal with these issues is development of a core outcome set (COS).^18–20 A COS is a list of outcomes that should be measured and reported in all clinical trials. Robust development of a COS ensures that researchers report on outcomes that are relevant to all major stakeholders.²⁰ Cochrane editors have furthermore stated that the availability of COSs would improve reliability of reviews.²¹

Aims and objectives

Aim

The aim of this study is to develop a COS suitable for trials on the management of HS. The COS is intended to suit all types of interventions for HS, regardless of setting or mode of administration.

Objectives

Study objectives are:

1. To identify a list of items and domains previously used in studies on the management of HS by a systematic review of the literature.

2. To develop a list of items relevant to HS disease severity generated by patients.

3. To develop a list of items relevant to HS disease severity generated by HS experts.

4. To combine the results of (1)—(3) into a unified list of candidate HS disease severity items for HS trials and use these to formulate potential core domains.

5. To achieve consensus on core domains for trials by in-person consensus meetings and online Delphi surveys including patient and HCP representatives at all stages.

Scope of the COS

Our intention is that the HS COS should apply to efficacy measures of all interventions in clinical trials on patients with HS globally. In accordance with the COMET definition, this does not necessarily mean that primary outcomes of clinical trials should always be chosen from this COS or that outcome measurers should be restricted to domains belonging to the COS. It is intended that domains of this COS should be considered mandatory for inclusion in all clinical trials on HS so that, in most trials, the primary outcome instrument would usually be one of those contained in the COS. If the primary outcome for a particular trial is not within the COS, then an explanation of the author’s decision should be provided in the protocol and subsequent trial report.²⁰

Methods and analysis

Initiatives like Core Outcome Measures in Effectiveness Trials (COMET)²⁰ and Outcome Measures in Rheumatology (OMERACT)²² provide methodological guidance that will be followed. The methodology involves a stepwise approach for the development of a COS. The first step is to identify which domains one should measure and report in all clinical controlled trials of a specific condition (what to measure: the domain set, COS).¹⁶ The second step is identifying the instruments that should be used to assess these domains (how to measure: the core outcome measurement set, COMS).^{18 20} The second step includes the possible need for development and validation of new instruments for domains that do not have valid outcome measure instruments.

This protocol focuses on the initial step, describing the method we will use to reach consensus on the core domain set. A summary of the study method can be found in figure 1. The OMERACT initiative has recently outlined definitions of key concepts and terms that we will adhere to throughout the protocol.¹⁸ A guideline on the usage of the Delphi technique for determining which outcomes to measure in clinical trials and the enclosed checklist will be followed.²³ The checklist from the COMET initiative will also be adhered to.²⁰

Figure 1.

Study summary. See the text for details. HCP, healthcare professional.

HISTORIC

The HIdradenitis SuppuraTiva cORe outcomes set International Collaboration (HISTORIC) is an international initiative arising from collaboration between the International Dermatology Outcome Measures organisation (IDEOM), the Cochrane Skin Group—Core Outcome Set Initiative (CSG-COUSIN) and Zealand University Hospital, Roskilde. HISTORIC goals are to:

• Develop global COSs for HS trials including the most relevant core domain set and adequate, that is, valid, reliable and feasible, instruments to measure the core domains using an iterative process of evidence synthesis and multistakeholder consensus.

• Implement the COS on a global level.

An international steering group consisting of researchers, clinicians and a patient research partner under the HISTORIC initiative has been formed to guide the development of the COS. The steering group consists of nine members; a patient research partner who is a board member of the HS Patients' Association in Denmark, two dermatologists with expertise in HS research and clinical work; four dermatologists with a special interest in outcome measures in dermatology and in HS; a biostatistician from OMERACT experienced in COS development and an MD Ph.D. student.

Stakeholder involvement

We will involve researchers, HCPs, patients, government agencies and industry representatives.²⁰ The involvement of multiple stakeholders for the development of a COS is strongly recommended by COS methodologists.^{18 20 23} For the development of the HS-COS, the steering group decided on involving six groups of stakeholders:

1. Patients (invites: 30 European, 30 North American, 10–20 Asian/South American/Australian).

2. Dermatologist HS experts (invites: 20 European, 20 North American, 10 Asian/South American/Australian).

3. Surgeons HS experts (invites: 4 European, 4 North American).

4. Nurses HS specialist (invites: 4 European, 4 North American).

5. Industry representatives (invites: 2).

6. Drug regulatory authorities (invites: 2).

For practical reasons—including ease of interpretability—the steering group decided to analyse patients as one stakeholder group and stakeholder group 2–6 together as another combined stakeholder group of healthcare professionals (HCPs). A 1:1 ratio of patients:HCPs will be aimed for with the target of ∼50–70 participants in each group.

Eligibility criteria

Identification and sampling

Patients

All board members from the Danish, English, German, Belgian, Austrian, French, Dutch and US HS patients' associations will be invited. If the number of patients from patient associations are not sufficient, the steering group will identify additional patients. From Austria, Asia, Australia and Africa, three patients from each area will be identified following contact with a local dermatologist with a special interest in HS.

Inclusion criteria:

• Confirmed diagnosis of HS.

• Agrees to participate in all Delphi rounds needed for this COS.

Exclusion criteria:

• Inadequate English skills.

• No access to internet.

HCPs

The steering group will identify the HCPs from the limited but growing community of HCPs working with patients with HS. Purposive sampling will be conducted according to the following criteria:

1. Commitment: Agrees to participate in all Delphi rounds needed for this COS

2. Knowledge: A clinical background with at least 5 years of experience with HS and at least six patients with HS treated per year

3. Involvement: Publications on HS or participation in scientific meetings on HS (Dermatologist HS experts only)

4. Representability: Geography

5. Representability: Occupation (dermatologist, nurse, etc)

6. Representability: Institution (clinic, university, regional hospital, non-governmental organization, etc)

7. Representability: Sex

Information sources

Identification of initial list of candidate items and potential core domains

The initial list of candidate items will be obtained in a three-step manner.

Systematic review of the literature

A recent systematic Cochrane review on interventions for HS and another systematic review on outcome measure instruments have reviewed existing instruments and mapped them according to domains.^{14 15}

Qualitative studies

A representative group of patients with HS will be identified from the outpatient clinic at the Department of Dermatology, Zealand University Hospital, Roskilde, Denmark.

The patients will be interviewed individually and in focus groups. The number of patients sampled will be determined by an assessment of when saturation is achieved, and no new knowledge is obtained from subsequent interviews. On the basis of previous experience, a sample size of 20–30 patients should be sufficient to reach saturation. Purposive sampling of a wide diversity of age groups, sex, treatment types and disease severities will be employed.

Patients will be identified primarily among those undergoing treatment at the Department of Dermatology at Roskilde Hospital, as well as through the patient association in Denmark. Eligibility is based on the diagnosis of HS (conforming to the modified Dessau criteria and provided by a specialist in dermatology)¹⁰ and willingness to participate. All three Hurley stages will be represented among the interviewees. Analysis of the interviews aims at identifying central topics of general relevance using thematic analysis.

The interviews will be tape recorded and transcribed verbatim and initially examined for units of meaning, coded as items and grouped into categories whenever a topic is identified as having HS-specific importance. The individual interviews will be supplemented by focus group interviews, improving the item-generating process. The focus group participants may be a mix of former participants and new participants, which also gives room for new items.

The focus group interviews will be semistructured based on the analysis of the single interviews. The focus group interviews will also be tape recorded, but only partly transcribed, omitting ‘obligatory small talk’. The analysis will be an iterative process moving back and forth between transcripts and analysis of single interviews and focus group interviews. On the basis of this analysis, a list of items will be developed, covering a whole range of individual perspectives on HS. This qualitative method provides a depth of insight into the complexity of meaning and reasoning in relation to HS. Focus group interviews will continue until the point of saturation when no further new items are identified. Qualitative interviews do not require ethical approval in Denmark.

A similar qualitative study will be performed in North America. Semistructured interviews will be performed in person by one interviewer. Transcripts will be reviewed line-by-line after each interview and words, phrases and passages related to symptoms and effects on life will be coded using NVivo V.10 (QSR International, Burlington, MA). These codes will be used to inform subsequent interviews. Preliminary codes will be reviewed, and then the final analytical codes will be applied and grouped into themes. The interviews will proceed until thematic saturation. This project has been approved by the institutional review board of the Penn State College of Medicine.

The two lists of potential items generated from the Danish and US qualitative studies will be combined into one patient-generated item list.

Identification of items of importance to HCPs

To identify outcomes of importance to HCPs, an item generation Delphi round 0 will be conducted among the HCPs stakeholder group (HS experts; dermatologists, surgeons, nurses, industry representatives and regulators). Participants will be invited by email (see online supplementary material A). Two reference articles on COS development will be attached to the invitation.^{18 20} Delphi round 0 will be presented as an online survey using the SurveyMonkey software. Initially, the HCPs will be provided with one page of background information on the rationale for the development of the COS (see online supplementary material B). They will then be asked to list all items that they consider important or relevant to HS, with items being related to any aspect of the disease, or treatment of the disease (see online supplementary material C). The open questions will ensure that no one will impose their views on the participants and thus introduce bias into the study. This implies that participants will suggest potential items without being prompted or guided by facilitators, the steering group or by reviews of the literature. This is in line with the recommendations outlined in Delphi guidelines.²³

The steering group will review the list of items suggested by the HCPs. A preliminary list of potential items will be produced by combining the results from the systematic reviews, from the qualitative studies and from the HCPs’ item generation survey.

Consensus process

Methods to reach consensus on the core domain set

To make sure that the views of participants are obtained by a method that gives equal influence to all participants and to ensure that no individual participant will be overtly influenced by the opinions of any others, an anonymous Delphi approach will be applied. The Delphi method facilitates consensus with geographically scattered participants through consecutive Delphi rounds. Participants are asked to rate the importance of the listed items. Participants will also be provided with an option to add additional items that they think are missing together with a scoring for each item added. A minimum of three Delphi rounds is planned. Two face-to-face consensus meetings are planned in Europe and the USA between e-Delphi rounds two and three to ensure the global reach of the COS. Representatives from the USA will be present at the European meeting and vice versa. The steering group decided not to invite pharmaceutical industry representatives to the face-to-face meetings to ensure that the process is driven by patients and HCPs.

The Delphi survey will be presented online using DelphiManager software. Panel members will be asked to participate in all rounds of the survey (see online supplementary materials A and D). A study information sheet will be attached to the patient invitation (see online supplementary material E).

Once participants have registered for the survey, names and email addresses will be stored in a separate database in the system and will be used for providing the participant with a unique identifier code. This code will allow identification of participants completing all rounds of the Delphi survey.

People with minority opinions are more likely to drop out of a consensus process, so attrition as rounds progress can lead to overestimation of consensus in the final results. To prevent attrition bias, participants will receive an explanation of the importance of completing the whole Delphi process with a paragraph adapted from the guideline on the usage of the Delphi Technique²³ (Box 1). The number of participants invited to participate will be documented together with the number of recruited participants. The number of participants completing subsequent rounds will also be documented and attrition assessed.

Box 1. Commitment to completing the entire Delphi process.

Thank you for agreeing to participate in the process aimed at identifying the most important features in clinical trials about HS. It is very important that you complete the survey in each round. You may feel that you answer the same questions several times, or the results of previous rounds may show that other participants do not agree with you. Anyway, it is very important that everyone participates in all surveys. If some participants drop out of the study before it is complete, the total results may become useless.

Participation will be fully anonymised, and participants will not know the specific answers that any other individual gave.

The steering group will design the questionnaires and will be responsible for sending out invitations and reminders to panel members, analysing the responses and drawing up feedback reports. Invitations will be sent by email. Each round will be online for 7–10 days, and reminder emails will be sent.

Questionnaires used in the Delphi survey will be pilot-tested by at least two members of the steering group, including the patient representative, and at least two selected other panel members.

Items/domain scoring

Participants will be asked to score each item/domain listed using the Grading of Recommendations, Assessment, Development and Evaluations scale from 1 to 9. In the Delphi exercise, scores 1–3 will be labelled ‘not important’, 4–6 labelled ‘ important but not critical’ and 7–9 labelled ‘ critical’.²⁴

Definition of consensus

The classifications outlined in table 1 will be used to decide whether consensus is reached or not.

Table 1.

Definition of consensus

Consensus classification	Description	Definition
Consensus in	Consensus that item/domain should be included in the core domain set	70% or more participants scoring 7–9 AND <15% participants scoring 1–3
Consensus out	Consensus that item/domain should not be included in the core domain set	70% or more participants scoring 1–3 AND <15% of participants scoring 7–9
No consensus	Uncertainty about importance of item/domain	Anything else

Criteria for ‘rule in’ and for ‘rule out’ are symmetrical. To reach consensus that an item/domain should be in the COS requires agreement by the majority regarding the critical importance of the item/domain, with only a small minority considering it to be not important at all. By comparing the proportions voting critical among HCPs and patients, when both proportions are above the 70% threshold from the Delphi exercise, this will indicate that these domains should be considered part of the COS. To reach consensus that an item/domain should not be in the COS, on the other hand, requires agreement by the majority regarding the lack of importance of the item/domain, with only a small minority considering it to be critically important. While the choice of thresholds is somewhat arbitrary, prospective specification of the definition of consensus should reduce the chance of consensus being defined post hoc in such a way as to bias the results towards the beliefs of the research team.

Delphi round 1

The invitation for the first round will be sent to all participants by email (see online supplementary material F). The ‘plain language summary’ from the COMET initiative,²⁵ defining outcome measures and explaining the purpose of COS development, will be attached to the invitation. The survey will be presented in an online format. First, the participants will be asked to provide background demographic information including age, geographical region, confirmed diagnosis of HS (for patients), years living with HS (for patients), years of experience with HS and number of HS contacts per year (HCPs).

On the next page, participants will be provided with some background information explaining how the candidate items have been identified and will then be asked to rate each of the items listed, based on their importance in being measured as an outcome in all clinical trials for HS (see online supplementary material G). It is emphasized that the items are all relevant to the overall care of people with HS, but for the purposes of the survey it is essential that participants decide whether each item is important/essential to be measured in all clinical trials on HS. The order of the candidate items will be randomised. The panel members will be encouraged to provide arguments for their choices and to suggest modifications of definitions or wording of the items. Panel members will also be asked to suggest items not represented in the list together with a rating for each item added.

Items suggested by the panel members will be reviewed by the steering group to ensure that they represent new items. If they do, the new items will be added to the list for the next round. For each item, the number of participants who have scored the item and the distribution of scores will be summarised by the stakeholder group (patients and HCPs). All items will be carried forward to round 2.

Delphi round 2

Only participants who have taken part in round 1 and have provided scores will be invited to participate in round 2. Survey participants will be shown the number of participants who have scored each item and the distribution of scores (as a percentage of the total) as graphs by the stakeholder group (patients and HCPs). Participants will be asked to consider responses from the other panel members and to re-score the items. Response options will be the same as in the first round.

Consensus meetings

First meeting

After the two first online surveys, a face-to-face consensus meeting will be held in Europe with representatives from all stakeholders including US stakeholders. An equal ratio of HCP:patients will be aimed for. Stakeholders who are not able to participate in person will be invited to join the meeting via Skype (Microsoft, USA).

The meeting format will be a mixture of small group work followed by plenary sessions, based on nominal group theory. The meeting will begin with a broad introduction to the HISTORIC collaboration and a summary of the reasoning for COS development. The study method and results from the first two rounds of the Delphi survey will be presented. Topics for the small group sessions will be item ranking, items to exclude, grouping of items into domains and ranking of domains. For each task, two small groups will work independently and in parallel, supervised in each case by a non-voting facilitator. Results from each group will be presented to all participants in the subsequent plenary session. The aim is to get all items either designated to an appropriate domain or excluded by in-person consensus.

Consensus meeting 2

Since the first consensus meeting will take place in Europe, making it difficult for North Americans to participate, a second consensus meeting located in the USA is planned.

Group work sessions will be held to confirm and adjust results from the first consensus meeting. Questions that will be raised are:

1. Should any deleted item be retained?

2. Do all the items fit within their designated domain?

3. Should any combined items form their own domain(s)?

4. Is the naming for each domain appropriate?

Confirmation of results from the consensus meeting by the larger Delphi group

A feedback report of the results of the two consensus meetings will be emailed to all Delphi participants. Participants will be asked the same four questions that were addressed at the second consensus meeting. The steering group will consider all comments and suggestions for changes and adjust if there is consensus in the steering group.

Delphi round 3

In round 3 of the e-Delphi, the items that have not been excluded at the consensus meetings will be shown under their designated domain, following the work from the in-person meetings. Items that have been excluded at the consensus meeting will be shown separately under no specific domain. Participants will be shown the distribution of scores in the second round for each item as graphs by the stakeholder group. Participants will also be shown the proportion voting ‘critical’ in round 2 by the stakeholder group and combined for the two groups. They will be asked to consider the responses from the other members and to re-score each item.

Results of the two stakeholder group responses will be compared with each other and reported as the percentage of agreement, giving an equal weight to the two stakeholder groups. Each item will be classified as ‘consensus in’, ‘consensus out’ or ‘no consensus’ according to the classification in table 1. Items classified as ‘consensus in’ will be used to provide a detailed definition of each domain. The name of the domains will be adjusted if needed. If an item that was excluded at the consensus meeting reaches ‘consensus in’, the item will be included in one of the existing domains or will be carried forward to round 4 as a separate domain.

Delphi round 4

In round 4, participants will vote on the domains only. Domains will be defined by their included items.

Each domain will be classified as ‘consensus in’, ‘consensus out’ or ‘no consensus’ according to the classifications in table 1. Domains classified as ‘consensus in’ will be the final core domains.

Statistical considerations

Sample size

Until now, there is no existing standard method for sample size calculation in Delphi surveys. Therefore, a pragmatic approach will be taken, ensuring broad demographic and geographic representation from patients with HS and HCPs. Sample sizes of 40–60 patients and 40–60 HCPs are expected to provide sufficient representation. Efforts will be taken to maximise the response rate across stakeholder groups.

Discussion

The absence of evidence-based, consensus-driven and widely accepted outcome measures is likely to impede access of patients with HS to the most effective and appropriate treatments available. Consensus regarding outcome measures further benefits researchers and HCPs by providing critical benchmarks for making treatment recommendations and evaluating patient progress. Payers will also receive information needed to determine the efficacy of various treatments, information that is key to shaping payment policies. The absence of clear outcome measures could result in payers providing minimal to no coverage for various therapies, curtailing patient access to care.

The number of RCTs of HS therapy is limited, but the interest in the disease and planned number of trials is significant. Therefore, a COS process is particularly timely for HS and identifying a COS for HS should significantly impact future developments within the field.¹⁰

At the time of writing, there is no published COS for HS. The development of COS for the management of HS aims to improve the interpretation and comparison of future studies on HS and to reduce the risk of outcome reporting bias and heterogeneity across studies. The study will involve different stakeholder groups to ensure that the developed COS will be suitable and well accepted. The initial steps described in this protocol have been conducted. Consensus meeting 2 and Delphi round 3 and 4 await.

Dissemination

The study will be performed according to the Helsinki declaration. All results from the study, including inconclusive or negative results, will be published in peer-reviewed journals.