Identifying what to measure in acne clinical trials: first steps towards development of a core outcome set

TO THE EDITOR

Outcome measures to assess efficacy of acne treatments are heterogeneous and inconsistently applied (Lehmann et al., 2002). The lack of universal adoption of a minimum set of well-validated outcome measures is a major obstacle to evidence synthesis. To date, the main focus has been on physician-assessed methods such as lesion counts and global severity scales. Instruments assessing impacts of acne on patients have only been used occasionally. The goal of the Acne Core Outcomes Research Network (ACORN) is to develop a set of core outcome measures to assess acne in a clinical trial setting following the Harmonizing Outcome Measures for Eczema (HOME) roadmap (Schmitt et al., 2015) with consultation from the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN). The goal of this study was to identify what to measure in an acne trial, in other words, the outcomes or domains of greatest importance.

In advance of this exercise, it was decided to include adverse events (AEs) as a core outcome domain as they are part of the overall risk/benefit assessment of any intervention as described by the OMERACT group (Boers et al, 2014). However, ACORN will not be developing novel outcome measures to assess AEs. A structured iterative process based on a pre-specified protocol was used in the “what to measure” exercise (Supplemental Materials and Methods). The study was registered in the COMET database. A preliminary list of candidate items was generated using acne-specific quality of life instruments, an acne specific patient reported outcome measure, a rigorous methodologic review of acne trials and a current ACORN systematic review of acne impacts (Motley and Finlay 1989; Layton et al., 1997; Girman et al., 1996; Gupta et al., 1998; Rapp et al., 2006; Alexis et al., 2014; Lehmann et al., 2002). Two authors (EAE, AML) and one patient appraised the list of items for omissions or ambiguity. Wording was edited to optimize understanding by all participants and definitions were provided (Supplemental Materials and Methods).

Potential participants were contacted using email lists for the Acne Priority Setting Partnership, ACORN membership, the Acne Global Alliance, the American Acne and Rosacea Society (AARS) and also via the AARS website. At least one editor of each of the top 30 dermatology journals was invited. Although consensus-based methods typically involve less than 50 participants, this number was considered too few to obtain unbiased international representation of multiple stakeholder groups. Although our target was a minimum of 200 participants in Round 1,500 participants from 54 countries responded to email requests to select the 12 most important items to measure from among a list of 24 items (Figure 1 and see Supplemental Materials and Methods for survey). Responses were monitored and efforts made to increase numbers for any stakeholder group which appeared to be under-represented. Two hundred and sixty participants left email addresses indicating that they wished to take part in Round 2.

Figure 1. International participation in defining core outcome domains in acne clinical trials.

(a) Demographics of the 500 participants in Round 1 (Respondent type adds up to >500 as some participants selected more than one option). (b) Demographics of the 116 participants in Round 3. There were 164 participants in Round 2: 112 professionals (including 9 editors and 10 industry personnel) and 52 patients (data not shown) (c) Responses to ranking importance of the initial 24 items in Round 1. The top 12 items are shown in black. Two items in red were taken forward into Round 2 following consultation with the Outcomes Identification group. (d) Seven core outcome domains were identified after all three rounds had been completed. Domains in gray and green were decided by consensus in Rounds 2 and 3, respectively. Adverse Events (in red) was chosen prior to the exercise based on recommendations in the literature (see text).

HCP, health care practitioner.

The top 12 items of importance (Figure 1c) were taken forward. Sixty-nine people added suggestions for other items but when appraised, most were covered by an existing item. (Supplemental Materials and Methods) Two suggestions that required further consideration were acne scars/dark marks and skin texture/quality. Concern was raised that two widely used outcomes in acne trials, global acne severity assessed by a physician and health-related quality of life (HRQoL) ranked 13 and 16 respectively. In order to resolve these ambiguities, the ACORN Steering Committee consulted the Outcomes Identification Group (OIG) and CSG-COUSIN. Their recommendation was to also take forward global acne severity assessed by physician, HRQoL and acne scars/dark marks into Round 2 where participants were invited to score 15 items on a Likert-like scale where 1 = not at all important and 9 = extremely important (See Supplemental Materials and Methods for survey). For data analysis, scores of 1–3 were considered unimportant, 4–6 equivocal and 7–9 important. To be included in the core domain set, consensus was defined a priori as being achieved when at least 70% of patients and 70% of professionals scored the item ≥7 and ≤15% scored the same item three or less. Any items for which over 70% of patients and professionals gave a score ≤3 would be dropped.

Response rate, summary scores for each item and items selected for inclusion as a core domain from Round 2 are shown in Table 1. “Satisfaction with treatment received” reached consensus among professionals but was 0.2% short of consensus among patients. The OIG and CSG-COUSIN agreed that this item should be included as a core domain. They were also consulted regarding the disagreement between patients and professionals regarding who should assess the clinical features of acne. The recommendation was to move signs and symptoms forward as a core domain recognizing that who and how they should be assessed would need to be resolved in the next phase to develop outcome measures. No items met the criterion for exclusion in this round (Table 1).

Table 1.

Summary of scoring by participants in Rounds 2 and 3.

	Signs by phys	Signs/Sx by pt	Speed of improve	Satisf with appear	Phys glob sever	Self-esteem	HR QoL	Long-term control	Satisf with Rx	Value of Rx to pt	Phys global improv	Pt global improv	Long-term conseq	Scars/dark marks	No flare
ROUND 2: 15 items. Scores 7–9 Important; 1–3 Not important. Overall Response rate = 63%
Patients, n = 52
Median score	7 [3]	8 [2]	8 [2.5]	8 [2]	6 [5]	8 [2]	7 [3]	8 [1]	8 [3]	7 [3]	7 [4]	7 [3.5]	8 [2]	8 [2]	8 [2.5]
% scores 7–9	64.2	77.4	75.5	83	47.2	79.2	54.7	88.7	69.8	60.4	50.9	67.9	77.4	77.4	75.5
% scores 1–3	15.1	3.8	7.5	9.4	26.4	9.4	9.4	5.7	5.7	9.4	17	11.3	1.9	5.7	5.7
Professionals, n = 112
Median score	8 [3]	7 [4]	7 [3]	8 [2]	7 [4]	8 [3]	7 [4]	7 [1]	7 [3]	7 [3]	7 [3]	8 [3]	7 [4]	8 [3]	7 [2]
% scores 7–9	71.2	60.4	51.4	76.6	59.5	67.6	57.7	75.7	71.2	50	63.1	70.3	59.5	73	65.8
% scores 1–3	9.9	13.5	9	6.3	7.2	5.4	7.2	4.5	6.3	10.8	7.2	3.6	11.7	6.3	6.3
Consensus to include	NO	NO	NO	YES	NO	NO	NO	YES	YES*	NO	NO	NO	NO	YES	NO
ROUND 3: 9 items. Vote “Yes” or “No” for inclusion. Overall response rate = 70.7%
Patients, n = 34
% Yes			76.5		44.1	81.6	73.5			31.2	20.6	61.8	73.5		67.6
Professionals, n = 82
% Yes			53.7		82.9	43.9	79.3			30.5	59.8	69.5	37.8		39
Consensus to include			NO		NO	NO	YES			NO	NO	NO	NO		NO

Key to Items: 1. Physical signs of acne assessed by a physician; 2. Signs and symptoms of acne assessed by the patient; 3. Speed of improvement in acne; 4. Satisfaction with appearance; 5. Global acne severity assessed by a physician; 6. Self-esteem; 7. Health-related quality of life; 8. Long-term control of acne; 9. Satisfaction with treatment received; 10. Value of treatment to the patient; 11. Global improvement in acne assessed by a physician; 12. Global improvement in acne assessed by the patient; 13. Long-term consequences of treatment received; 14. Extent of scars and dark marks; 15. Absence of acne flares.

HRQoL, health related quality of life. Rx, treatment. Sx, symptoms.

Concern was raised that feedback of high median scores (Table 1) to participants in Round 3 would likely encourage the minority who gave low scores in Round 2 to increase them. This could result in lack of discrimination with far more items being retained than is practicable for a core outcome set. The study protocol provided an option to ask a simple Yes/No question as to whether an item should be included in the core domain set if consensus was not achieved. The decision to proceed with the Yes/No option was supported by the OIG. The criterion for consensus in Round 3 was 70% of patients and 70% of professionals voting yes. Items on which consensus had already been reached in Round 2 were not taken forward into this round. Participant information, response rates and the responses obtained for Round 3 are shown in Table 1. (See Supplemental Materials and Methods for survey). Consensus for inclusion was achieved for one item, HRQoL. There was considerable disagreement between patients and professionals on several remaining items, one of which, global acne severity assessed by a physician, is very widely used in acne trials and is recommended in the US Food and Drug Administration (FDA) (Center for Drug Evaluation and Research 2005). A summary of the core outcome domains resulting from this study are presented in Figure 1d. Next steps involve identification or development of measures to assess each of the core outcomes.