Defining Skin Ulcers in Systemic Sclerosis: Systematic Literature Review and Proposed World Scleroderma Foundation (WSF) definition.

Abstract

Purpose:

There is a lack of a valid, definition for skin ulcers in SSc to be used in clinical trials. Our aim was to develop a consensus definition for SSc-skin ulcers based on the results of a systematic literature review (SLR) for skin ulcer definitions and expert opinion; and to evaluate its face validity, reliability and feasibility.

Methods:

SLR for skin ulcer definitions was conducted using PubMed, Web of Science, and Cochrane library for articles published from inception to January 1^st, 2016. SSc experts were to discuss the definitions’ categories and vote for the relevant terms. Reliability of the definition were tested in a second expert meeting, seven SSc experts evaluated 7 SSc pts with skin lesions twice. Face validity and feasibility evaluated by sending out case report forms(CRFs) to 4 SSc experts, they were asked to use the definition in 5 pts each.

Results:

A total of 3464 abstracts and titles were screened, and 446 articles were fully evaluated. Of these, 66 met eligibility criteria and skin ulcer definitions were extracted. SSc experts discussed, refined and voted on the consensus definition using nominal process. Kappa for inter-, intra-rater rater agreement was 0.51, 0.90 respectively. The mean time to decide if the lesion is an ulcer was 7.4 sec. All investigators endorsed the face validity of the new definition in the CRFs.

Conclusion:

Using a SLR and a nominal technique, we developed a preliminary consensus-based definition of SSc-skin ulcers. Face validity, feasibility and reliability were demonstrated for the developed definition.

Introduction

Systemic sclerosis (SSc) is an immune mediated disease with multiple phenotypic presentations, driven by interplay of autoimmunity and vasculopathy, leading to dermal and internal organ fibrosis. In SSc, skin ulcers are a major challenge usually secondary to vasculopathy and less frequently to trauma, calcinosis or gangrene (1) (2) (3). Skin ulcers are frequently found on finger tips, on toes or over the extensor surfaces and bony prominences (such as the elbow). Most ulcers are painful and often result in considerable impairment of hand function (4). Digital ulcers (DU) tend to recur, with up to 66% of patients having more than one episode, despite use of vasodilators ((5). Additionally, there is a risk of subsequent irreversible tissue loss, as well as other significant complications including osteomyelitis, gangrene, and amputation (3). It is estimated that up to 5-10% of SSc patients experience gangrene or amputation (6) (7) (8) (9). The risk of gangrene and amputation rises to 20% in patients with DUs while the incidence of amputation ranges from 1 to 2% of patients/year (10) (11). Patients with DU show significantly disability characterized by impaired hand function, increased pain and altered quality of life (QOL) (12). In SSc, DU are also a considerable financial burden, as patients require more hospitalizations (including cost of antibiotics) than those without DU (13).

There are a variety of indicators for assessing DU, although they are not validated, including measuring their size, number, location, loss of function, pain, infection and evolution to gangrene and time to healing (3).

Given the effect of DU on QoL and hand function, a valid and reliable definition of a SSc ulcer is an unmet need. Despite DU prevention and healing have been primary endpoints in clinical trials there has been a difficulty to define what is precisely a SSc ulcer( (14) (15).

With support from the World Scleroderma Foundation (WSF), we set out to develop a new consensus-based, validated definition of digital ulcers for the purposes of clinical trials.

As the first step in developing such a definition, we evaluated the definitions of “skin ulcer” in the literature. The primary aim of the present work was to develop a consensus based definition of ulcers in SSc, using the descriptive terms stated in the literature. We also tested the face validity, feasibility and reliability of this definition.

Methods

Literature search

Since we anticipated a paucity of reported definitions and/or classification of ulcers in SSc, we included other related autoimmune diseases (systemic lupus erythematosus (SLE), rheumatoid arthritis (RA) and vasculitis) in our search. Diabetic ulcers are also of interest because there has been much effort regarding defining skin ulcers in diabetes. Pressure ulcers were excluded because their pathogenesis is varied and often unrelated to the pathogenesis of SSc- related ulcers. We conducted a systematic literature review (SLR) examining the clinical studies reporting a definition of skin ulcers in SSc, in autoimmune diseases (SLE, RA, vasculitis), and in diabetes mellitus.

Data Sources:

Database searches were carried out by two investigators (DEF, YS) and a library information specialist(B.M.). PubMed, Web of Science, and Cochrane database were searched for articles published from inception to January 1st, 2016. The search was limited to the English language.

Search terms:

Keywords and MeSH terms for the following concepts were used in the search: skin ulcer, nonhealing wound, or chronic wound; scleroderma, systemic sclerosis, systemic lupus erythematosus, rheumatoid arthritis, vasculitis, diabetes; classification or definition; and clinical trials.

Study selection

Studies satisfying the following Inclusion criteria were selected for full data abstraction:

1) peer reviewed studies in SSc, SLE, RA, vasculitis and/or diabetes mellitus, 2) reporting skin ulcers as an outcome, 3) 10 or more patients, 4) adults (age ≥18 years).

Exclusion criteria were: 1) not in humans (e.g. genetic basic research or research in non-human animals); 2) not in disease(s) of interest; 3) not in body areas of interest such as oral ulcers, nasal ulcers, gastro-intestinal ulcers and not associated with the pathogenesis of skin ulcers, 4) pressure ulcers; 5) review articles; 6) skin ulcer is not outcome; 7) duplicate publication; 8) case reports/case series with <10 patients; 9) patients < 18 years of age.

Data abstraction

Three groups of investigators (YS, MA, NB, EP, CB and LC) with two investigators per group, each independently reviewed the title and abstract of each citation and applied the inclusion and exclusion criteria to select studies for full review. A standardized data abstraction form was used. Cross-review of 10% of studies among reviewers established consistency of review. If there was disagreement, consensus was reached among the 2 reviewers by discussion. If necessary, a third reviewer (DEF) arbitrated.

Outcomes

The primary outcome was skin ulcer definitions reported in the publication. The secondary outcome was the classification systems for subsetting the ulcers to develop a “grading system” of skin ulcers in SSc. During the above extraction process, those terms which were not part of the definition were also extracted.

Definition Formulation

In a face-to-face meeting, SSc experts from North America (n=6) and Europe (n=6) participated in developing the new SSc definition. Extracted definitions and descriptive terms were categorized into domains according to the defective skin layers, depth description and mitigating factors (size, site, calcinosis, pain, etc.) Unclear or non-specific definitions were excluded from inclusion in our voting process. SSc experts discussed the definitions using a nominal process and voted for the pertinent definition terms during the Scleroderma World Congress 2016 held in Lisbon, Portugal. The Nominal group technique(NGT) was utilized. NGT is a structured variation of a group discussion to reach consensus. NGT gathers information by asking individuals to respond to questions posed by a moderator, and then asking participants to prioritize the suggestions or ideas of all members of the group (16).

When there was disagreement, discussion ensued with possible further refinement until consensus was achieved. A consensus was defined as >70% agreement among participants.

Reliability

Reliability is defined by OMERACT as repeatability, consistency, and reproducibility which represents the extent to which a measurement procedure yields the same result on repeated determinations (17).

A second face to face meeting was conducted at the Royal Free Hospital (London, UK) to evaluate reliability, and feasibility of the newly developed definition: 7 rheumatologists discussed and refined the developed definition. Each investigator assessed 7 SSc patients with skin lesions twice. Each patient was identified through her/his initials and was sitting on a chair in front of a table marked with a number, with 7 separate tables placed in the same room with a circular disposition. Each clinician was given up to 30 seconds to sit in front of the patient and decide if the definition suited the patient lesion, then moving to the next table, counterclockwise. After the first round was concluded, all clinicians moved to a second room while a nurse prepared patients for the second round using an online available randomizing software (www.random.org) to change patients’ order. The second round was repeated as above.

Feasibility and Face validity:

Feasibility was defined per the Outcome Measures in Rheumatology (OMERACT) FILTER. Feasibility in the OMERACT filter encompasses the practical considerations of using an instrument, including its ease of use, time to complete, monetary costs and interpretability of the question(s) included in the instrument. Face validity was defined as expert opinion regarding credibility of the measure (18).

Photographs of 11 SSc skin lesions were evaluated before and after definition development to examine the face validity and feasibility of the definition and to allow further refinement of the definition

Case report forms containing the newly developed definition were sent to four investigators to use the new definition in assessment of skin lesions and evaluate if they found it credible as well as to ascertained the time (in seconds) taken to decide if a skin lesion is an ulcer or not.

Analysis

Descriptive statistics were used to summarize the data. The results adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for systematic literature reviews and meta-analyses (19). Inter-rater agreement was evaluated for each separate round using Fleiss’ Kappa, while intra-rater agreement was evaluated using Cohen’s Kappa statistic. Stength of agreement was interpreted as follows: 0 = poor, 0.01-0.20 = slight, 0.21-0.40 = fair, 0.41-0.60 = moderate, 0.61-0.80 = substantial, and 0.81-1 = almost perfect.

Results

Systematic literature review driven definitions

Database searches yielded 3464 publications. After removing duplicates (n= 251), 3213 titles and abstracts were screened, resulting in 446 articles with data referring to skin ulcers as an outcome for full text review (Fig. 1). Sixty-three abstracts were case reports, 1373 abstracts reported diseases not of interest, 38 studies were in those less than 18 years of age, and 675 abstracts did not report skin ulcer as an outcome. Ultimately, skin ulcer definitions were extracted from 66 studies, after consensus by the 2 investigators in each group. Definitions were extracted from 34 SSc studies, 28 diabetes and 4 rheumatoid arthritis articles. Supplementary Table 1 shows the extracted definitions (supplementary).

Figure 1:

showing the systematic review diagram.

Table 1:

showing terms used to define skin ulcers

Definition categories	Number of times used
Loss of Epidermis	11
Loss of Epidermis and Dermis (full thickness)	34
With Depth	4
Denuded	4
Non-specific/unclear/healing definitions:	27
Ischemic necrotic ulcer
Open sore
Loss of tissue
Open wound
Skin break
Necrotic lesion

Formulation of a new Definition

The results of the literature search were reviewed and a list of domains were developed to group descriptive terms. The domains (e.g. loss of epidermis) are listed in Table 1. The potentially important mitigating factors (e.g. site, size, etc.) were also considered and are listed in Table 2.

Table 2:

showing mitigating factors that were used in skin ulcer description

Mitigating/Clarifying factors
Site (10 times): for DU (finger tip, distal digits, distal to DIP, distal to proximal interphalangeal digital crease, distal to PIP), for diabetes (malleoli, below knee)
Size (8 times): for SSc (at least 2 mm), diabetic (0.5 to 30 cm2)
Calcinosis (6 times): either as an inclusion or exclusion in ulcer definition.
Painful (4 times): used as part of the definition only in the scleroderma ulcers
Scab (1 time): part of the definition of an indeterminate ulcer (hardened covering of dried secretions (as blood, plasma, or pus) that forms over an ulcer).
Gangrene (1 time): as an exclusion
Skin fissure (8 times): in SSc as exclusion
Healing ulcer (22 times): in SSc as inclusion (amenable to healing or definition of healing) In Diabetes, inclusion not healing in 30 days or definition of healing.
Pitting scar (5 times): defined as [pinhole sized depression with hyperkeratosis}and excluded.
Pathogenesis (4 times): ischemic, traumatic (mechanical), infectious

At the first face-to-face meeting, the domains and mitigating factors were presented to the SSc experts (11 rheumatologists, 1 dermatologist), and voting on inclusion and exclusion of each domain or factor ensued. The final definition was developed by consensus using NGT for utilization in clinical trials.

The proposed WSF definition is:

“Loss of epidermal covering with a break in the basement membrane (which separates dermis from epidermis). It appears clinically as visible blood vessels, fibrin, granulation tissue and/or underlying deeper structures (e.g. muscle, ligament, fat) or as it would appear on debridement” Definition and exclusions are in Figure 2.

Figure 2:

Showing the World Scleroderma Foundation (WSF) Definition for SSc-skin ulcers to be used in clinical trials

Size, site depth and pain were not mentioned in the definition because they were not considered critical by the voting experts during the first face to face SSc experts meeting.

Validation of the new definition

Face validity was shown by applying the newly developed definition to photographs of 11 SSc skin lesions. Further intensive discussions took place during the 2^nd F2F meeting resulted in modification of the definition for inclusion in a clinical trial. The main item for discussion was the need for debridement to be able to fully assess an ulcer (e.g. if covered by an eschar), with resultant addition of the sentence “as it would appear on debridement” (4^th line in the definition in italic see Figure 2).

Reliability of the newly developed definition between and within raters was then tested in the 2^nd face-to-face meeting. The Kappa for inter-rater agreement during was 0.51 during the first round and 0.49 during round two(moderate). The mean intra-rater Kappa is 0.90(excellent).

Feasibility was evaluated by examining the time taken to apply the definition to patients during a clinic visit, where 20 patients with SSc related skin lesions were examined by 4 investigators. It took a mean of 7.4 (3-20) seconds for each investigator to apply the definition.

Discussion

In SSc, studies of skin ulcer treatment in systemic sclerosis frequently did not show treatment effects or showed variable or unclear results (20) (21).It is quite possible that unclear or inconsistent definitions of the ulcer contributed to these poor or unclear results. Our study has systematically started to define an ulcer in SSc and has also started the process of validating such a definition. We conducted a systematic literature review to identify published definitions of ‘skin ulcers’ in the literature. The resulting definitions were then categorized into domains and were voted upon by the co-authors using a modified nominal group technique to develop a consensus-based definition of SSc-skin ulcers. Face validity, reliability and feasibility of the new SSc skin ulcer definition were also evaluated. The intra-rater reliability was excellent and inter-rater reproducibility was moderate as is nearly always true when validating measures of response (22). This points to the need to have the same evaluator measuring ulcers for each patient, as is also required for trials of rheumatoid arthritis( (23). We wanted to ensure a certain degree of uniformity among included SSc-skin ulcers in clinical trials by excluding factors that might confound the inclusion decision and the resultant treatment effect (e.g. calcinosis, infection), hence promoting improved precision among included SSc-skin ulcers.

The development of the exclusion list started during the extraction process, as well as during extensive discussions throughout the electronic search process. The presence of an expert dermatologist was a very positive influence and provided for a further refinement of the definition. Histologic explanation of skin layers and the difference between erosion (abrasion) and skin ulcer were discussed. The skin ulcers overlying calcinosis were considered traumatic and excluded. The issue of whether infected skin ulcers were to be allowed when defining SSc-skin ulcers in clinical trials was considered and their presence was agreed to be an exclusion for trial inclusion.

Another significant aspect in the definition is the phrase “or as it would appear on debridement”. This is clearly of importance since some SSc-skin ulcers are covered with a scab or eschar, as was discovered during the reliability exercises. This points to an area requiring further research and will need to be addressed when ulcer studies are designed Wound bed preparation and debridement are thus helpful to allow the assessment of the underlying ulcer and evaluate if it meets the definition (24). An important unresolved issue whether or not to require debridement of each included ulcer in a clinical trial. It was agreed that this aspect of DU definition can be decided at the time the trial is designed.

Our systematic literature review highlighted the lack of uniformity for SSc-skin ulcer definition and facilitated the development of the domains from the published definitions.

The study by Baron et al., classified SSc-skin ulcers into three categories: active, healed and indeterminate for the purpose of clinical trials (25). Their definition was included as one of the significant definitions extracted. Their definition of ‘active’, “inactive” and “indeterminate”, were defined in the manuscript; nevertheless there are difficulties with these definitions. For example, “active” implies an ulcer can exist but be “inactive” and many believe, quite reasonably, that an “inactive” ulcer is no longer an ulcer at all. Likewise, “indeterminate” seems to indicate that the observer is not quite sure whether there is an ulcer or not. This status, as “indeterminate” can lead to greater variability and, hence, less ability to come to a clear result. This issue may be clarified, for example, by debridement or use of ultra-sound but will require further clarification.

A recent study by Hughes et al, evaluated whether the reliability among rheumatologists grading DUs improves by providing the assessor with clinical information. They used 80 images and 51 rheumatologists (web based). The addition of clinical information did not statistically improve the inter- or intra- rater reliability. they concluded that the inter and intra rater reliability of DU grading did not improve by providing patients’ clinical context (26). This study emphasizes the need for a more uniform definition that is widely acceptable and reliable.

An earlier study by Herrick et al, evaluated the inter- and intra-rater reliability among SSc experts to assess their ability to define an active SSc-skin ulcer via SSc-skin ulcer images (15). Their overall intra-rater weighted kappa coefficient was 0.81, while the inter-rater kappa co-efficient was 0.46, generally similar to ours. They did not use a consensus definition; instead they used 13 exemplar lesions agreed upon by the clinicians who designed the study.

Our study had the strengths of utilizing a literature-based systematic review to derive potential ulcer definitions, the advantage of having experts from both rheumatology and dermatology, the use of both experienced and less experienced experts (thus ensuring a more generalized representativeness), and the use of a nominal technique to develop the consensus, thus assuring face validity. Further reliability and feasibility were directly measured.

The limitations of our study included lack of direct input from patients, although, for clinical trial purposes this was not essential. In addition, content, construct, criterion validity and response/discrimination remain to be evaluated, although these psychometric properties were deliberately left for the future. Another limitation is the small number of pts used in our reliability study, although we believe this is sufficient in this early stage of validation. Larger numbers of ulcers are warranted in future studies. The inter-rater reliability in our study was only moderate, other studies in RA( (22)and SSc (27) have demonstrated that teaching and standardization improve inter-rater agreement and we believe that such teaching is warranted and appropriate when evaluating ulcers as well. We would recommend such standardization when embarking on a clinical trial.

In conclusion, we used systematic literature review and nominal techniques to derive a consensus-definition of skin ulcers in SSc. We demonstrated face validity, feasibility and reliability of the SSc-ulcer definition for clinical trials of interventions in SSc-skin ulcers. Other aspects of validation and responsiveness remain to be completed.