Quantitative analysis validation for sclerotherapy treatment of lower limb telangiectasias

Abstract

Background

The evaluation of sclerotherapy efficacy for lower limb telangiectasias, which is the standard treatment for such condition, is commonly assisted by scores based on before and after pictures. This method is marked by its subjectivity, which impairs the precision of studies on the subject, making it unfeasible to evaluate and compare different interventions. We hypothesize that a quantitative method for evaluating the effectiveness of sclerotherapy for lower limb telangiectasias may present more reproducible results. Reliable measurement methods and new technologies may become part of the clinical practice in the near future.

Methods

Before and after treatment photographs were analyzed using a quantitative method and compared with a validated qualitative method based on improvement scores. Reliability analysis of the methods was performed, applying the intraclass correlation coefficient (ICC) and kappa coefficient with quadratic weights (Fleiss Cohen), for analysis of inter-examiner and intra-examiner agreement in both evaluation methods. Convergent validity was evaluated by applying the Spearman test. To assess the applicability of the quantitative scale, the Mann-Whitney test was used.

Results

A better agreement between examiners is shown for the quantitative scale, with a mean kappa of .3986 (.251-.511) for qualitative analysis and a mean kappa of .788 (.655-.918) for quantitative analysis (P < .001 for all examiners). Convergent validity was achieved by correlation coefficients of .572 to .905 (P < .001). The quantitative scale results obtained between the specialists with different degrees of experience did not show statistical difference (seniors: 0.71 [−0.48/1.00] × juniors: 0.73 [−0.34/1.00]; P = .221).

Conclusions

Convergent validity between both analyses has been achieved, but quantitative analysis has been shown to be more reliable and can be applied by professionals of any degree of experience. The validation of quantitative analysis is a major milestone for the development of new technology and automated, reliable, applications.

Article Highlights.

• •Type of Research: Retrospective analysis of prospectively collected data from the Polidocanol vs glucose in the treatment of telangiectasia of the lower limbs (PG3T) trial
• •Key Findings: Quantitative scales show a better strength of agreement, with a mean kappa of .3986 (.251-.511, fair to moderate) for qualitative analysis and a mean kappa of .788 (.655-.918, substantial to almost perfect) for quantitative (P < .001 for all examiners).
• •Take Home Message: Quantitative analysis of lower limb telangiectasias should be considered over qualitative analysis. The validation of quantitative methods is a major milestone for the development of automated applications.

Telangiectasias are dilated, superficial, visible venules or capillaries in the dermal layer of the skin, with a diameter varying between 0.1 and 1 mm, being bluish or reddish in color.^1,2 Although they may cause symptoms such as pain or pruritus, the aesthetic complaint is the main reason for patients to seek specialized therapy.³

Therapeutic techniques include transdermal laser ablation and chemical sclerotherapy, which is the treatment of choice.4, 5, 6, 7, 8, 9 The technique consists of the obliteration of these vessels and subsequent fibrosis of the targeted segment caused by the injection of a sclerosing agent into its lumen.² These agents may be osmotic, detergents, or chemical irritants, according to their different mechanisms of action.10, 11, 12

The effectiveness of the treatment depends on the extent of damage to the endothelium of blood vessels and their visual extinction, which is currently measured by qualitative analysis of pre-treatment and post-treatment photos and patient satisfaction scales.4, 5, 6, 7 Therefore, most studies about results of sclerotherapy treatments involve a significant subjectivity bias. This subjectivity makes it unfeasible to systematically collect accurate data on the extent and characteristics of these treated telangiectasias. There is a lack of precise studies regarding these vessels and the effectiveness of different treatment modalities.

A detailed review of the methods for measuring comparative results revealed a significantly reduced number of articles that present a reproducible methodology; the vast majority is based on the subjectivity of visual assessment, and even fewer studies present the coefficient correlation between the examiners, which is essential for establishing the reliability of the data.13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23

In recent studies presented by this team,²³ manual quantitative measurement of the telangiectasias has been presented, using the ImageJ open-source software for a supposedly better qualification and quantification of sclerotherapy results.

This study is justified as a way of examining the hypothesis that quantitative analysis may represent more reproductible results. Validating the methodology and proving its equivalence or superiority in relation to the methods described so far, is indispensable to clearly answer whether this technology should replace or complement the current way of evaluating these results and to open doors to new technology that may assist in the process.

Materials and Methods

Data assessment

The present work involves the assessment of photographic data of 50 treated limbs contained in our team’s database of 98 patients submitted to lower limb telangiectasias sclerotherapy (Fig 1).23, 24, 25 All data is available upon request. Requests should be addressed to the Department of Surgery and Orthopedics of Botucatu Medical School – Sao Paulo State University via the institution’s address or directly to the corresponding author.

Fig 1.

Patient flowchart for assessment and reassessment.

All patients signed an informed consent form that had been approved by the internal review board of Botucatu Medical School, Sao Paulo State University, Protocol 4127-2012. The photographic database is part of the PG3T trial,²³ registered at ClinicalTrials.gov (NCT02657252).

The images were acquired before and after treatment using a high-definition D7000 digital camera (Nikon, Tokyo, Japan) and AF-S 18-105 mm lens (1:3.5-5.6 G). The images were obtained in a controlled environment, isolated from sunlight, under white fluorescent light from a fixed distance of 60 cm. The treatment area was defined as a 150 cm² (15 × 10 cm) on the lateral aspect of the tight of one limb. No anatomical references were uncovered. One Velcro template integrated with a centimeter scale ruler was positioned 5 cm above the knee joint line. No lotions or creams were applied on the skin before or after the procedures in order to preserve the natural refractivity and reflexibility of the human skin. Photos were taken in a supine position.

All participants had mild venous disease (C1, only telangiectasias or reticular veins) accordingly to the clinical-etiology-anatomy-pathophysiology classification (CEAP).²⁵ In the forementioned study, these patients were randomized into two treatment groups: Group 1 (51 patients) treated with diluted 0.2% polidocanol in 70% glucose; Group 2 (47 patients) treated with 75% glucose. Only photographic data was analyzed. The images from each group were randomly selected, and the objective was not to confront previous findings but to validate the method. Precise details on the technique are available on previous publications.^23,24,26,27 The physician, the patients, and the data analyst were blinded. Qualitative and quantitative analyses were performed in a 7-day interval and in the forementioned order. Pre-treatment and post-treatment analyses were performed with a 60-day interval.

Qualitative analysis

For qualitative analysis, we reproduced the scale published in 2007 by Philippe Kern,¹⁴ which was validated in a previous study in 2004.¹³ The scale aims to evaluate the effectiveness of sclerotherapy treatment for telangiectasias of the lower limb through the evaluation of pre-treatment and post-treatment photographs. Kern’s study involved a sample of 100 patients; photos were analyzed by independent specialists that were not involved in the treatment. Such images were graded in an efficacy score that ranged from 0 (no improvement) to 10 (complete disappearance of the vessel).¹⁴

In our study, five examiners, two senior specialists with 10-15 years of activity (M.B., P.D.C.) and three junior specialists with 2-3 years of activity (P.M.F., V.G., N.S.) graded 50 cases using the Kern score.

Quantitative analysis

The measurements proposed by our study were performed through the evaluation of pre-treatment and post-treatment digital images and calculation of an improvement percentage. The same two senior specialists and three junior specialists assessed the total length of the telangiectasias after software training and image standardization.

The digital image post-processing for standardization and the measurements were achieved by using the ImageJ software. ImageJ is an open-source software for image analysis, edition, calibration, post-processing, and measurements.

For standardization, one of the team’s researchers selected a random sample of 50 images from our database and applied post-processing calibrations of brightness and contrast to standardize the pictures (ImageJ > Adjust > Brightness/Contrast). That researcher did not participate in the evaluation process.

The lengths of the telangiectasias were assessed in pixels and converted to centimeters through a ruler fixed in the photographic field (ImageJ > Straight line > Analyze > Set scale > Known distance). The same monitor was used for all calculations: 16:9, 4k resolution, 144 Hz refresh rate, 350 cd/m², 800:1 contrast rate, and 45% NTSC color gamut.

After the measurement in centimeters, the calculation of improvement was performed through the formula: Improvement Percentage = 100∗[(Initial Length–Final Length)/Initial Length]. The post-treatment images were reassessed in a random order after 2 months, and the examiners did not have access to the previous evaluation for comparison. An example of the analysis can be seen in Fig 2.

Fig 2.

Example of the pre-treatment (A) and post-treatment (B) of telangiectasias and qualitative total length analysis.

Reliability

Both qualitative and quantitative analysis were performed for 50 patients twice in a 2-month interval after treatment. The reliability of both methods was given by the criteria of equivalence and stability of the scales, through a test to evaluate the inter-examiner agreement (evaluation of images between different examiners), and a test to evaluate the intra-examiner (second evaluation of images by the same examiner) agreement, respectively.

The evaluation of inter-examiner agreement of the qualitative scale was performed by applying the kappa statistical method with quadratic weights (Fleiss Cohen). The quantitative scale was evaluated through the intraclass correlation coefficient (ICC). After applying the statistical models, the agreement indicators obtained were compared, according to the magnitude of agreement as proposed by Landis and Koch: <0 indicating no agreement, 0 to 0.20 as slight, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and 0.81 to 1 as an almost perfect agreement.²⁵ The intra-examiner agreement of each scale was evaluated by three specialists using the same statistical model after 2 months.

Convergent validity

Convergent validity was evaluated through analysis of the correlation between the results of both scales. The correlation test chosen was the Spearman rank correlation coefficient, considering that in the distribution of delta values for each of the scales after application of the Kolmogorov-Smirnov normality test, a non-parametric distribution was observed. Through the results obtained by the application of the correlation test, convergence between the measures obtained by the two scales can be confirmed, which may lead to the conclusion that both are valid methods.

Applicability

To evaluate methodological applicability and reproducibility by senior and junior specialists, the results obtained from each specialists’ group was compared with the other specialists through the Mann-Whitney test, considering the non-parametric distribution of the data for both scales.

Results

Reliability

The inter-examiner agreement analysis (Tables I and II) shows a better agreement between examiners on the quantitative scale. In the intra-examiner agreement analysis (Table III), both scales presented adequate agreement indicators of at least moderate according to Landis and Koch, showing satisfactory stability.²⁸ Mean kappa values of 0.3986 (0.251-0.511) for qualitative analysis and 0.788 (0.655-0.918) for quantitative analysis (P < .001 for all examiners) were observed, showing a statistically significant higher kappa for the quantitative scale. Quantitative analysis has been shown to be more reliable.

Table I.

Inter-examiner reliability for the qualitative scale

	Qualitative scale
	K	95% CI	Pb	Strength of agreementa
E1 × E2	0.251	0.129-0.374	<.001	Fair
E1 × E3	0.352	0.153-0.550	<.001	Fair
E1 × E4	0.476	0.346-0.606	<.001	Moderate
E1 × E5	0.511	0.383-0.638	<.001	Moderate
E2 × E3	0.388	0.258-0.518	<.001	Fair
E2 × E4	0.432	0.295-0.569	<.001	Moderate
E2 × E5	0.278	0.167-0.388	<.001	Fair
E3 × E4	0.444	0.276-0.613	<.001	Moderate
E3 × E5	0.363	0.194-0.531	<.001	Fair
E4 × E5	0.491	0.378-0.605	<.001	Moderate

CI, Confidence interval; E, specialist; ICC, intraclass correlation coefficient; K, linearly weighted Kappa.

^aStrength of agreement accordingly to Landis and Loch, 1977.²⁸

^bP value associated with Kappa.

Table II.

Inter-examiner reliability for the quantitative scale

	Quantitative scale
	ICC	95% CI	Pb	Strength of agreementa
E1 × E2	0.772	0.474-0.888	<.001	Substantial
E1 × E3	0.774	0.601-0.872	<.001	Substantial
E1 × E4	0.918	0.855-0.953	<.001	Almost perfect
E1 × E5	0.757	0.430-0.882	<.001	Substantial
E2 × E3	0.658	0.162-0.839	<.001	Substantial
E2 × E4	0.856	0.632-0.933	<.001	Almost perfect
E2 × E5	0.655	−0.170 to 0.871	<.001	Substantial
E3 × E4	0.784	0.614-0.878	<.001	Substantial
E3 × E5	0.866	0.744-0.927	<.001	Almost perfect
E4 × E5	0.840	0.389-0.937	<.001	Almost perfect

CI, Confidence interval; E, specialist; ICC, intraclass correlation coefficient; K, linearly weighted Kappa.

^aStrength of agreement accordingly to Landis and Loch, 1977.²⁸

^bP value associated with F test applied to the values of ICC.

Table III.

Intra-examiner reliability for qualitative and quantitative scales for senior and junior specialists

	Qualitative scale				Quantitative scale
	K	95% CI	Pb	Strength of agreementa	CCI	95% CI	Pc	Strenght of agreementa
E1 (Senior)	0.505	0.355-0.655	<.001	Moderate	0.987	0.976-0.993	<.001	Almost perfect
E2 (Junior)	0.713	0.600-0.826	<.001	Substantial	0.597	0.285-0.772	<.001	Moderate

CI, Confidence interval; E, specialist; ICC, intraclass correlation coefficient; K, linearly weighted Kappa.

^aStrength of agreement accordingly to Landis and Loch, 1977.²⁸

^bP value associated with Kappa.

^cP value associated with F test applied to the values of ICC.

Convergent validity

Convergence was based on data from the analysis of each specialist and a second analysis performed by a senior specialist and two junior specialists (Table IV; Fig 3). In most tests, there was an adequate correlation (moderate at least). Kappa correlation coefficients ranged from .572 to .905 (P < .001). Convergent validity was achieved successfully.

Table IV.

Correlation between values obtained via the qualitative and quantitative scales

Specialist	Rs	P	Interpretation of the correlation coefficienta
E1	0.743	<.001	Moderate
E2	0.848	<.001	Very strong
E3	0.572	<.001	Fair
E4	0.809	<.001	Very strong
E5	0.608	<.001	Moderate
E1 (second analysis)	0.783	<.001	Moderate
E2 (second analysis)	0.818	<.001	Very strong
E4 (second analysis)	0.905	<.0001	Very strong

P, P value associated to Spearman’s correlation test; Rs, Spearman’s correlation coefficient.

^aInterpretation of the correlation coefficient according to Chan, 2003.²⁹

Fig 3.

Dispersion graphs of the correlation analysis between values obtained via the qualitative and quantitative scales.

Applicability

The applicability of the quantitative scale was assessed by comparing the results obtained by the senior specialists and the junior specialists. The Mann-Whitney test was performed for the values obtained by the qualitative scale with significant difference between the senior and junior specialists (seniors: 8 [2/10] × juniors: 9 [0/10]; P = .023). On the other hand, the quantitative results obtained between specialists did not show significant differences (seniors: 0.71 [−0.48/1.00] × juniors: 0.73 [−0.34/1.00]; P = .221). Quantitative analysis results have shown no statistically relevant difference between juniors and seniors, demonstrating that it can be applied by professionals of any degree of experience.

Discussion

Blood vessel segmentation metrics were proposed in the mid-1900s, and there is a vast scientific literature addressing this topic. However, several issues still need to be addressed to standardize the evaluation methods, such as the need to develop specific applications for an extensive array of tissues and imaging modalities. Up to this point, there is no universal segmentation method.30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50 Patient-based scales of satisfaction does not show statistically significant correlation with the sclerotherapy results.²³ Patients are only exposed to subjective evaluations of the treatment, often by themselves, which may change over time and are exposed to many types of bias. An objective evaluation of the treatment may positively serve as a counterpoint to the patient’s self judgement.

One of the main challenges on the topic is the segmentation of pathological vessels, which may show a broad spectrum of morphological manifestations. Unfortunately, there is still not much evidence on this subject and very few articles addressing segmentation metrics for lower limb telangiectasias.

The higher strength of agreement shown by the quantitative scale (substantial to almost perfect) demonstrates that it presents better equivalence than the qualitative scale (fair to moderate), therefore, being more reliable.

Qualitative scales, especially for aesthetic treatments, may be strongly affected by a diverse array of bias such as interviewer’s bias, which is a systematic misinterpretation of the data commonly noted when the patient status is known to the interviewer; chronology bias, which occurs when the examiner is aware of when the intervention was administered; and performance bias, which is the misinterpretation of the cause-effect relation of a known intervention.⁵¹ Although quantitative scales may be affected by the same biases, our evidence suggests that it was less affected by those. Further precautions may–and should–be taken to further address those biases such as masking, randomization, allocation concealment, and prevention of deciphering.⁵² The prevention of deciphering is not possible to be achieved when improvement scales are applied, but the improvement delta may still be calculated for quantitative scales.

Independently of the specific characteristics of the telangiectasias, the values obtained through the quantitative scale converge to the values obtained by the qualitative scale when subjected to the Spearman rank correlation coefficient, demonstrating convergent validity in addition to the internal validity shown by the strength of agreement of the proposed instrument.

In addition to the scale’s validation, the proposed quantitative scale can be applied by specialists with different degrees of experience, as indicated by the absence of statistical differences between senior and junior specialists.

Even though the analysis is relatively simple, imaging, post-processing, camera standardization, image adjustments, and some practice on image management may be necessary. Time spent on each image analysis may be an important limitation to the professional application of this technology.

This validation may support subsequent efforts to automate the method through artificial intelligence. However, this next step will require additional mechanisms to standardize image acquisition, because quality variations strongly affect the performance of automatic segmentation, which may represent an additional limitation in comparing different methods.

Conclusions

Both scales presented adequate agreement indicators, showing satisfactory stability, but a stronger strength of agreement is seen for the quantitative scale. The validity of the quantitative scale is demonstrated by the convergence of values obtained by both scales. The quantitative scale can be reproduced by specialists of different degrees of experience, whereas the qualitative scale should not; therefore, future research on telangiectasias should consider using quantitative methods.

The validation of quantitative analysis for telangiectasias is a major milestone for the development of automated methods and opens the door to a whole new field of research for artificial intelligence in vascular surgery, which may, ultimately, create the necessary conditions for reliable applications that provide results with minimum bias.

Author Contributions

Conception and design: MB, LR, MS, PL

Analysis and interpretation: CG, LR, PL

Data collection: MB, PMF, PDC, VG, NS

Writing the article: PMF, CG

Critical revision of the article: MB, PDC, VG, NS, LR, MS, PL

Final approval of the article: MB, PMF, CG, PDC, VG, NS, LR, MS, PL

Statistical analysis: PL

Obtained funding: MB

Overall responsibility: MB