Evaluation of test characteristics for outcome measures used in Raynaud's phenomenon clinical trials

Abstract

Objective

Randomized controlled trials (RCTs) in Raynaud's phenomenon (RP) have shown conflicting efficacy data. Also, there is no consensus on the outcome measures that should be used. Our objectives were: 1) assess the reliability of individual core set measures used in 3 RCTs; 2) evaluate the placebo response for individual core set measures; and 3) determine if a composite of individual core set measures will decrease the placebo response which may improve our ability to see treatment effects in future trials.

Patients and Methods

We analyzed core set measures from 249 patients in the placebo-treated groups from 3 RCTs. Core set measures analyzed included Raynaud's condition score (RCS), patient and physician assessment of RP, pain, numbness, and tingling during an RP attack, average number of attacks/day, and duration of attacks. ICC correlation coefficients were calculated during the run-in period to the RCTs.

Results

ICC coefficients of ≥0.70 were observed for RCS, attack symptoms, and average attacks/day. A high placebo response rate was observed for all individual core measures except the duration of attacks. For the RCS, the placebo response ranged from 56% with >10% improvement to 20% with ≥60% improvement. In contrast, placebo response rates of 10–20% were observed when several core set measures were combined to develop a composite score.

Conclusions

Outcome measures used in RP RCTs are associated with marked variability. Combination of outcome measures is associated with low placebo responses. Future studies are needed to assess if a composite score will be able to differentiate placebo from an effective agent.

INTRODUCTION

Raynaud's phenomenon (RP) is a condition where there is vasospasm of blood vessel, caused by an imbalance of vasoconstricting and vasodilating factors, resulting in decreased blood flow in the extremities causing anoxia, paroxysmal pallor and/or cyanosis. This recurrent ischemia can eventually result in loss of function of fingers or toes and in extreme cases gangrene or even amputation [1]. Primary RP is generally thought to be induced (e.g. cold, emotions) whereas secondary RP tends to be associated with other diseases that have endothelial abnormalities (e.g. systemic sclerosis) but can be exacerbated by factors known to precipitate primary RP. Although the prevalence of secondary RP is difficult to characterize as it depends on the underlying disorder, prevalence rates of primary RP have been reported to be 11% in women and 8% in men in the United States with yearly incidence rates of 2.2% and 1.5%, respectively [2]. RP is usually associated with a detrimental effect on day-to-day activities and secondary RP is an important contributing factor for developing digital ulcers and ischemia [2].

Previous clinical trials have used different core set measures to assess efficacy and have included patient-reported frequency and duration of RP attacks, RP attack symptoms (such as pain, numbness, and tingling), the RP Condition Score (RCS), and patient and physician global assessments of RP. These studies have differed in the outcome measures used and the primary outcome measure specified. This is reminiscent of the 1980s when rheumatoid arthritis trials were being conducted with lack of consensus on a group of core set outcome measures to assess efficacy. The lack of uniform outcome measures impedes drug development and hampers the meta-analysis to assess efficacy.

In this study, our objectives were: 1) to assess the reliability of individual core set measures used in RP randomized controlled trials (RCTs); 2) evaluate the placebo response for individual core set measures in RCT's; and 3) determine if a composite of individual core set measures will decrease the placebo response. We used the placebo data from 3 large RCTs that assessed efficacy using a novel preparation of nitroglycerin in patients with primary and secondary RP.

PATIENTS AND METHODS

Patients

We analyzed the placebo-treated groups from 3 randomized controlled clinical trials (MediQuest Therapeutics). No patient participated in more than one trial. Data was collected from a total of 249 placebo-treated patients with primary or secondary RP conducted in the US during the period of 2006–2008. All types of secondary RP patients were included in the three studies. Each patient with secondary RP was then identified by the type of primary disease; majority of the patients had SSc.

Clinical Trial Descriptions

Study 05-002 (NCT00266669) was a multi-center, placebo-controlled RCT enrolling 219 subjects with a clinical diagnosis of primary RP or secondary RP due to a connective tissue disease[3]. Subjects discontinued vasodilator therapies for 2 weeks, followed by a 2-week single-blind placebo run-in phase to assess disease severity. Those with at least 5 RP attacks during a 7-day period entered the 4-week double-blinded portion of the study. Subjects were randomly assigned to receive placebo or 0.9% MQX-503 a microemulsion formulation of topical nitroglycerin for rapid local delivery with less systemic side effects, and were instructed to apply the gel immediately before or within 5 minutes of a RP event (maximum of 4 applications daily). Subjects used an electronic diary to record each gel application and/or Raynaud event. Each day, subjects recorded a RCS and a composite self-assessment of the severity of RP rated on a scale of 0 (no attacks) to 10 (severe attacks). The primary outcome was the change in mean RCS at the target week (the treatment week that matched the run-in period in terms of ambient temperature) compared to baseline (run-in period).

Study 06-004/5 (NCT00419419) was a randomized double blind crossover study in patients with moderate to severe primary RP or secondary RP due to a connective tissue disease that enrolled 110 patients (57 randomized to the placebo group and 53 to MQX-503), Subjects discontinued vasodilator therapy for two weeks, followed by a two week placebo run in period to assess disease activity. This was followed by six weeks of treatment in a crossover design; three weeks on active drug followed by three weeks on placebo versus three weeks on placebo followed by three weeks active drug. Subjects were then followed for one week after last treatment visit. The primary outcome was to assess changes in the RCS scores. Subjects used an electronic diary to record each treatment and/or RP event. Each day, subjects recorded the RCS and self-assessment of the RP severity rated on a scale of 0 (no attacks) to 10 (severe attacks).

Study 07-005 (NCT00577304) was a randomized double-blind crossover study that recruited a total of 164 patients with moderate to severe primary RP secondary RP due to a connective tissue disease with 84 assigned to the placebo group. Subjects were allowed to continue vasodilator therapy at a stable dose. There was a two week run in period to evaluate baseline disease activity. Subjects were given pouch applicators of either the nitroglycerin preparation or placebo for use on both hands up to 5 minutes prior to an anticipated Raynaud's attack or up to 5 minutes after the beginning of an actual attack. Patients were limited to 4 applications daily with at least two hours between applications. The primary outcome was change in the RCS scores. Subjects used an electronic diary to record each gel application and/or RP event. Each day, subjects recorded the RCS and self-assessment of the severity of RP rated on a scale of 0 (no attacks) to 10 (severe attacks).

Core set measures analyzed

Eight core set measures were assessed from patients in these 3 studies: physician assessment of RP and the following patient-reported outcome measures—the RCS, patient assessment of RP; pain, numbness, and tingling with each attack; average number of attacks per day; and duration of attacks. Daily patient logs of RP attacks were used to compute the average number of attacks per day, the average duration of attacks, the daily averages of patient reported pain, numbness, and tingling associated with each reported attack and a daily RCS. Patient and physician assessments of RP were recorded weekly. Daily averages were used to compute weekly averages for the period between physician visits, and the weekly averages were averaged for the run-in period and for the treatment period. Percent improvement between run-in and treatment periods was calculated for 8 core set measures.

Statistical analysis

The reliability of the outcome measures between the run-in period and treatment periods was assessed using the standard definition of intra-class correlation (ICC):

$$\mathit{ICC}=\frac{{\sigma }_{\alpha }^2}{{\sigma }_{\alpha }^2+{\sigma }_{\epsilon }^2}$$

where ${\sigma }_{\epsilon }^2$ denotes the between-subject variation and ${\sigma }_{\epsilon }^2$ denotes the within subject variation. ICC was assessed during run-in period before the patients were randomized to their group. Our hypothesis was that if the outcome measures are reliable they should not differ appreciably between run-in and treatment periods, and the ICC should be high. An ICC of ≥ 0.70 was considered satisfactory for group comparisons [4].

Also, for current analysis, the pain, numbness and tingling symptoms of RP attacks had high correlation coefficients (0.77–0.78) and were grouped together into attack symptoms, by selecting the percent improvement of the outcome with the highest degree of improvement. This resulted in 6 individual core set measures. We also assessed preliminary definitions of improvement and required ≥ X% improvement in Y of the 6 variables where X was set at 10%, 20%, 30%, 40%, 50%, and 60% and Y was set as 2, 3, 4, 5, or 6 variables, similar to performed by Paulus et al [5].

RESULTS

Patient Characteristics

A total of 249 placebo patients were included in the analysis. The mean (SD) age for 3 RCTs was 47.5 (12.4) years, 92% were female, 80% were non-Hispanic Whites, and 53% had secondary RP (Table 1). Baseline scores for the outcome measures are presented in Table 1. There were no baseline differences in the demographics between primary RP versus secondary RP groups. In comparing baseline scores between primary RP versus secondary RP groups, patients with primary RP had fewer RP attacks (p < 0.05). In contrast, pain and numbness were significantly greater in patients with primary RP (Table 1).

Table 1.

Baseline Characteristics of Study Participants

Variables	All Patients (n=249)	Secondary RP (N=132)	Primary RP (N=117)
Age, years	47.5 (12.4)	49.2 (11.1)	45.5 (13.4)
Gender: Female, N (%)	230(92.4)	93.9%	90.6%
Race
Caucasian, N (%)	200(80.3)	98 (74.2%)	102 (87.2%)
African-American, N (%)	20 (8.0)	15 (11.4%)	5 (4.3%)
Hispanic, N (%)	10 (4.0)	5 (3.8%)	5 (4.3%)
Asian/Pacific Islander, N (%)	14(5.6)	10 (7.6%)	4 (3.4%)
Others, N (%)	5 (2.0)	4 (3.0%)	1 (0.9%)
Baseline score, N, Mean % (SD)
Patient Assessment of RP on a VAS, (0–100)	N=249, 53.6 (17.6)	N=132, 54.1 (17.3)	N=117, 53 (17.9)
Physician Assessment of RP on VAS, (0–100)	N=249, 48.2 (18.7)	N=132, 48.5 (19.1)	N=117, 47.8 (17.9)
Attack Symptoms, (0–100)	N=215, 34.6 (23.0)	N=117, 31.4 (22)	N=98, 38.6 (23.4)
Pain*	N=223, 33.1 (23.3)	N=120, 29.4 (22.7)	N=103, 37.3, (23.4)
Tingling*	N=223, 30.9 (22.8)	N=120, 28.3 (21.3)	N=103, 33.9 (24.2)
Numbness *	N=223, 41.7 (24.8)	N=120, 38.3 (24.9)	N=103, 45.8 (24.1)
Average attacks per day‡	N=247, 2.1 (1.4)	N=132, 2.3 (1.5)	N=115, 1.9 (1.3)
Duration of attacks (minutes)	N=221, 28.4 (16.0)	N=119, 28.1 (14.1)	N=102, 28.8 (18.0)
Raynaud's condition score (0–100)‡	N=247, 3.6 (2.0)	N=132, 3.5 (2.1)	N=115, 3.6 (1.9)

^*Data not available for 26 patients

^‡Data not available for 2 patients. p< 0.05 for Attack Symptoms, Pain, Tingling and RCS between primary vs. secondary RP. All other comparisons are not significant at p≥0.05

Intraclass Correlation Coefficients (ICC)

Patients had a high degree of variability in their core set measures. The ICC was acceptable for RCS, attack symptoms, and average attacks/ day (ICC ≥ 0.70). Patient and physician global assessments and the duration of attacks had ICC coefficients < 0.70 (Table 2). The ICCs for individual studies are presented as Appendix 1 and shows variability within the 3 RCTs. For example, ICC ranged from 0.47 to 0.71 for RCS in the 3 trials. ICCs were similar between patients with primary and secondary RP (Table 2).

Table 2.

Intraclass correlation analysis among the different core set measures assessed in patients in 3 clinical trials

Core set measures*	All Patients (n=249)	Secondary RP (N=132)	Primary RP (N=117)
Patient Assessment of RP on a VAS	0.47	0.49	0.46
Physician assessment of RP on a VAS	0.54	0.52	0.57
Attack Symptoms:	0.76	0.78	0.72
Pain during RP attack on a VAS	0.78	0.79	0.76
Numbness during RP attack on a VAS	0.77	0.81	0.70
Tingling during RP attack on a VAS	0.77	0.76	0.79
Average Attacks / day	0.79	0.75	0.86
Duration of the attacks in minutes	0.61	0.63	0.61
Raynaud's Condition Score (RCS)	0.70	0.74	0.65

^*Units for each core set measures are provided in Table 1

Change in individual Outcome Measures for a Given Level of Improvement

We also assessed the variability in different core set measures by calculating the change in each core set measure for a given level of improvement (range from <10 to ≥60% improvement (Table 3). There was generally a high placebo response for all individual core set measures (except duration of attacks which ranged from 1.4% to 36.3%). As an example, for the RCS, the placebo response ranged from 56% with ≥10% improvement to 19.5% with ≥60% improvement. The mean placebo response for the all three trials is shown in Figure 1 and the range of the 3 trials is shown in Appendix 2.

Table -3.

Proportion of patients who achieved a pre-defined percentage of improvement for each core set measure in the 3 clinical trials

% Improvement	Patient Assessment	Physician Assessment	Duration of Attacks	Average Attacks /day	RCS	Attack Symptoms
≥ 60%	13.9	14.7	1.4	10.3	19.5	22.3
≥ 50%	21.2	20.8	1.9	21.4	22.4	32.1
≥ 40%	29.4	31.0	4.7	31.3	29.0	41.4
≥ 30%	45.3	39.6	6.5	43.6	35.3	50.7
≥ 20%	55.5	53.5	14.4	51.4	45.6	62.3
≥ 10%	66.1	64.1	26.0	63.4	56.0	71.2
< 10%	74.7	75.1	36.3	72.8	65.1	77.2

Values refer to the percent patients showing the specified level of improvement for each core set measure; RCS= Raynaud's Condition Score

Figure 1. Percent of patients showing the improvement in 6 core set measures.

Data shows the mean improvement for all 3 clinical trials.

Change in Percent Improvement in Core Set Measures in Relation to the Number of Outcome Measures Examined

Similar to the development of the Paulus criteria [5], we explored if simple arithmetic combination of percent improvement using 6 core set measures will result in a decreased placebo response. Therefore, we assessed the percent improvement relative to the number of core set measures included in the analysis (Figure 1 and 2). 92.8% of placebo-treated patients showed ≥10% improvement in at least one core set measure (Figure 2d) with 38.5% showing ≥60% improvement in at least one core set measure. An increased percent improvement along with combining core set measures decreased the placebo response rate. For example, 78% of patients had an improvement in individual core set measures by ≥30% for 1 of 6 core set measure, 53% for 2 of 6, 39% for 3 of 6, 26% for 4 of 6, 13% for 5 of 6, and 2% for 6 of 6 measures, respectively. There were no significant differences between primary and secondary RP (Figure 3a and 3b; Appendix 3a and 3b)

Figure 2. Percent of patients showing improvement when assessing 1–6 core set measures.

Plots show Improvement over the range of 10% improvement to 40% improvement (a–d) as assessed against the number of core set measures included in the analyses.

Figure-3. Percent of patients with a given improvement in core set measures.

The data shows the percent of patients with a given level of improvement for all 6 core set measures. (a) secondary RP, and (b) primary RP patients.

DISCUSSION

Current pharmacologic therapies used to treat Raynaud's phenomenon (RP) have modest efficacy and include agents such as calcium channel blockers, angiotensin II antagonists, alpha-1 adrenergic blockers, and selective serotonin reuptake inhibitors [6]. Apart from calcium channel blockers, which are considered the first line therapy, and prostacyclins for severe cases in patients with secondary RP, evidence for the effects of other agents is limited and side effects are common [6–8]. RCTs in RP have used different outcome measures and results have been conflicting. This is likely due to the episodic nature of RP and lack of standardization of outcome measures. Using placebo data from 3 large RCTs, we show that outcome measures in RP have marked intra-rater variability and are associated with a high placebo response. In addition, composite index of the individual outcome measures reduces the measurement variability and placebo response.

Meta-analysis of RP trials have shown modest efficacy of current pharmacologic agents [9, 10]. In addition, trials have provided conflicting results. For example, in a RCT with oral iloprost in 103 patients with RP secondary to SSc [11] that evaluated frequency of RP, daily duration of RP, severity of RP, and physician global assessment of RP as outcome measures, the duration and severity of RP showed a statistical improvement after 6 weeks of active treatment vs. the placebo group. However, another trial of 308 patients with RP secondary to SSc [12] that evaluated the average duration of RP attacks, average number of RP attacks, and RCS, there was no statistical improvement in the iloprost group vs. placebo group. More recently, tadalafil was assessed in RP in double-blind, placebo-controlled RCTs. One RCT showed improvement in RCS, duration and frequency of attacks compared to the placebo group [13], whereas the other RCT that evaluated RCS, frequency and duration of RP attacks failed to show a statistically difference compared to the placebo group in any outcome measure [14]. The above data highlights the difficulty in determining whether pharmacologic interventions are efficacious since there is no standardization of outcome measures in RCTs in RP [13, 15].

The lack of reliable biomarkers or surrogate endpoints in RP that reliably predict efficacy has required that investigators utilize endpoints that can result in high placebo response rates due to the variability in these core set measures between patients. Using varied core set measures between trials, however, makes it impossible to compare trial results and in some cases the absence of core set measures may make investigators decide which outcome measures to report and may only report those that show significance. In addition, using a parameter insensitive to change will result in a therapy that may have otherwise showed activity being scored as ineffective. This has hampered the approval of new drugs for RP and left physicians with minimal options for treatment.

In the 1980s, patients with RA have also shown similar variability in individual core set measures when attempting to measure drug efficacy [16]. Approximately 10 different individual core set measures had been used in an attempt to gauge efficacy of new agents in RA. However, due to statistical chance, the possibility that any one parameter will change in response to therapy made RCTs difficult to interpret. This was first addressed by Paulus et. al. who proposed a composite score based on statistical analysis to gauge the activity of DMARDs [5]. The criteria, known as “Paulus criteria”, required the improvement in 4 out of 6 core set measures: ≥ 20% in each of the following core set measures: morning stiffness, ESR, joint pain/tenderness index, joint swelling score, patient overall assessment of current disease severity, and physician overall assessment of current disease severity. In the initial analysis, very few placebo treated patents qualified for improvement whereas significantly more patients treated with DMARD's improved. A consistently low placebo response is essential for any composite score to ensure that responses observed with an agent represent real improvements. The Paulus criteria were later modified to develop the American College of Rheumatology 20% (ACR20) improvement criteria [17], the gold standard for approval of drugs by regulatory agencies for RA.

Using similar methodology as the Paulus criteria, our analysis suggests a composite index for RP that can decrease the placebo response rate to an acceptable level allowing for a better evaluation of a therapeutic efficacy with new treatments. The choice of how many core set measures should be combined, whether a certain percentage improvement in RCS and patient global assessment is required (similar to ACR20 where 20% improvement in the swollen joint count and tender joint count is required) needs to be determined in future studies. Further, some of these individual measures, although widely used, are subjective and it is debatable whether they should be included. It also needs to be determined if combination of individual core set variables will lead to higher discrimination between an effective drug vs. placebo in RP. If predictive of efficacy, this composite index will enhance our ability to evaluate new therapies for RP patients and should expedite approvals in this area of high medical need.

Other studies have evaluated core set measures and response measures in RP. Using data from a large RCT of using oral iloprost in SSc and RP, Merkel and colleagues conducted a factor analysis and found 4 factors—of these, 2 factors assessed RP associated disease activity and severity measures and the other 2 captured digital ulcers and mood/tension measures. All RP measures are included in the current study and formed the basis of this core set. In addition, we captured tingling and numbness during an acute attack of RP as these are common symptoms in patients with RP [18]. In another study, Khanna et al estimated the minimum clinically important improvement from one of the trials [19] and found an improvement of 1.4–1.5 points in RCS (0–10 scale) meets MCID criterion. However, this analysis did not address the variability of RCS and the placebo response.

Our study has many strengths. First, we included patients from 3 large RCTs in RP where individual patient data was available. Second, uniform core set measures were incorporated in these studies providing strength to our analysis. We have carefully evaluated the psychometric characteristics of the core set measures.

Our study is not without limitations. Lack of an effective therapeutic agent makes it difficult to assess if a composite index would be successful in discriminating an effective drug vs. placebo. However, this study did not assess this as 2 of 3 primary trials have not yet been published and the data was not available to evaluate. Future studies will evaluate this in the context of this and other trials.

In conclusion, analysis of placebo groups from 3 large RCTs show that there is marked variability in the individual core set measures used in RP clinical trials and combination of these variables reduces variability.

SIGNIFICANCE AND INNOVATIONS

• Outcome measures currently used in Raynaud's phenomenon clinical trials show marked intra-rater variability and are associated with a high placebo response in analysis of 3 large RCTs.

• Composite index of the individual outcome measures is associated with a reduction in the measurement variability and a lower placebo response.

Appendix

Appendix I.

Intraclass coefficient for the 3 individual trials

Variables	Study # 05-002	Study # 06-004/5	Study # 07-005
Raynaud's Condition Score	0.71	0.47	0.68
Patient Assessment of RP on a VAS	0.44	0.49	0.45
Physician Assessment of RP on a VAS	0.52	0.45	0.60
Attack Symptoms:	0.69	0.70	0.80
Pain during attack of RP on a VAS	0.69	0.67	0.82
Numbness during the attack of RP on a VAS	0.72	0.73	0.80
Tingling during the attack of RP on a VAS	0.70	0.70	0.82
Duration of the attacks in minutes	0.38	0.68	0.77
Average Attacks/day	0.71	0.47	0.68

VAS=visual analog scale

Appendix 2.

Range for each core set measure who achieved pre-defined percentage of improvement in the 3 clinical trials

% Improvement	Patient Assessment	Physician Assessment	Duration of Attacks	Average Attacks / day	RCS
>60%	7.4–19.4	7.4–19.6	0–2.5	5.6–12.3	13.6–25.9
>50%	17-3–24.1	13.6–28.6	0–3.7	14.8–25.9	17.3–27.8
>40%	26.8–32.4	22.2–38.0	1.2–7.4	22.2–35.2	27.2–33.3
>30%	44.6–45.7	30.9–46.3	2.5–9.9	31.5–49.1	32.1–42.6
>20%	51.8–60.0	44.4–63.9	10.0–17.3	38.9–58.3	37.7–57.4
>10%	58.9–72.2	53.6–74.1	18.8–37.0	50.0–69.1	47.2–68.5
>0%	67.9–80.6	69.6–80.6	28.7–50.0	59.3–79.0	57.5–72.2

RCS= Raynaud's Condition Score

Appendix 3a.

Proportion of patients who achieved pre-defined percentage of improvement for each core set measure in the 3 clinical trials with secondary RP

Secondary RP
% Improvement	Patient Assessment	Physician Assessment	Duration of Attacks	Average Attacks / day	RCS	Attack symptoms
≥ 60%	12.9	11.4	1.7	13.6	18.5	25.6
≥ 50%	18.9	20.5	1.7	25.8	20	36.8
≥40%	27.3	30.3	3.4	34.8	28.5	43.6
≥30%	44.7	36.4	6.8	46.2	33.1	53.8
≥20%	58.3	52.3	14.5	55.3	43.8	65.8
≥10%	66.7	64.4	26.5	67.4	53.8	72.6
<10%	75.8	75	36.8	75	63.8	78.6

RCS= Raynaud's Condition Score

Appendix 3b.

Proportion of patients who achieved pre-defined percentage of improvement for each core set measure in the 3 clinical trials with primary RP

Primary RP
% Improvement	Patient Assessment	Physician Assessment	Duration of Attacks	Average Attacks / day	RCS	Attack symptoms
≥ 60%	15	18.6	1	6.3	20.7	18.4
≥ 50%	23.9	21.2	2	16.2	25.2	26.5
≥40%	31.9	31.9	6.1	27	29.7	38.8
≥30%	46	43.4	6.1	40.5	37.8	46.9
≥20%	52.2	54.9	14.3	46.8	47.7	58.2
≥10%	65.5	63.7	25.5	58.6	58.6	69.4
< 10%	73.5	75.2	35.7	70.3	66.7	75.5

RCS= Raynaud's Condition Score