Inter- and intra-observer variability for the assessment of coronary artery tree description and lesion EvaluaTion (CatLet^©) angiographic scoring system in patients with acute myocardial infarction

Abstract

Background:

Previously, we developed a novel Coronary Artery Tree description and Lesion EvaluaTion (CatLet^©) angiographic scoring system, which was capable of accounting for the variability in the coronary anatomy and assisting in the risk-stratification of patients with acute myocardial infarction (AMI). Our preliminary study revealed that the CatLet score better predicted clinical outcomes for AMI patients than the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery score. However, the reproducibility of the CatLet score in both inter- and intra-observer remains to be evaluated.

Methods:

A total of 30 consecutive AMI patients, admitted in September of 2015, were independently assessed by two experienced interventional cardiologists to evaluate the inter-observer reproducibility of the CatLet score. Another set of 49 consecutive AMI patients, admitted between September and October in 2014, were assessed by one of the two interventional cardiologists on two occasions 3 months apart to evaluate the intra-observer reproducibility of the CatLet score. The weighted kappa was used to express the degree of agreement.

Results:

The weighted kappa values (95% confidence interval) for the intra- and inter-observer reproducibility of the CatLet Score were 0.82 (0.59–1.00, Z = 7.23, P < 0.001) and 0.86 (0.54–1.00, Z = 5.20, P < 0.001), respectively, according to the tertile analysis (≤14, 15–22, >22). Regarding the adverse characteristics pertinent to lesions and dominance parameters, the kappa values for the inter-observer variability were 0.80 (0.56–1.00, Z = 6.47, P < 0.001) for total number of lesions, 0.57 (0.28–0.85, Z = 3.03, P < 0.001) for bifurcation, 0.69 (0.43–0.96, Z = 5.06, P < 0.001) for heavy calcification, 1.00 (0.72–1.00, Z = 6.93, P < 0.001) for tortuosity, 0.54 (0.26–0.82, Z = 3.78, P < 0.001) for thrombus, 0.69 (0.48–0.91, Z = 6.29, P < 0.001) for right coronary artery dominance, 0.69 (0.41–0.96, Z = 4.91, P < 0.001) for left anterior descending artery length, and 0.22 (0.06–0.51, Z = 1.56, P = 0.06) for diagonal size. Equivalent values for the intra-observer variability were moderate to almost perfect (range 0.54–1.00).

Conclusions:

The reproducibility of the CatLet angiographic scoring system for evaluation of the coronary angiograms ranged from substantial to excellent. The high reproducibility of the CatLet angiographic scoring system will boost its clinical application to patients with AMI.

Introduction

Recently, we have developed a novel Coronary Artery Tree description and Lesion EvaluaTion (CatLet^©) angiographic scoring system, which is capable to account for the variability in the coronary anatomy and to assist in risk-stratification of patients with acute myocardial infarction (AMI) and collection of angiographic data.^[1,2] According to this new score's algorithm, only a lesion ≥50% diameter stenosis in vessels >1.5 mm in diameter is scored and further modified if appropriate, and the adverse characteristics pertinent to the lesion are not scored anymore, but only qualitatively recorded instead.^[2] The Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) score, another angiographic scoring tool, has been widely used to grade the severity and complexity of coronary artery disease (CAD) and to predict clinical outcomes for CAD patients.^[3–10] The limitations of the SYNTAX score include its failure to reflect the variability in the coronary anatomy, and its relatively lower reproducibility in both inter- and intra-observer, with reported kappa values ranging from 0.45 to 0.78.^[2,11–15] Our preliminary study demonstrated that the CatLet score had a better predictive value for the long-term prognosis of AMI patients than the SYNTAX score.^[1] However, the reproducibility of the CatLet angiographic scoring system in both inter- and intra-observer remains to be evaluated. The current study sought to evaluate the reproducibility of the CatLet angiographic scoring system for the stenosis lesions, their pertinent adverse angiographic characteristics, and the dominance parameters in AMI patients.

Methods

Ethical approval

Written informed consent was obtained from all patients before the procedure was conducted. The study protocol was approved by the Ethical Committee of Soochow University (No. 2020089).

Patients

A total of 66 suspected AMI patients referred for coronary angiography (CAG) between September 1, 2014, and October 9, 2014, were consecutively enrolled to assess the intra-observer reproducibility. Likewise, a total of 54 suspected AMI patients between September 7, 2015, and September 30, 2015, were consecutively enrolled to assess the inter-observer reproducibility. All of the patients were from the Chest Pain Center, the First Affiliated Hospital of Soochow University. All of the procedures, including antiplatelet loading, approaches to the culprit or non-culprit lesions, and maintenance of dual antiplatelet therapy, were performed in accordance with the standard recommendation. The exclusion criteria included prior stent implantation, normal CAG results, chronic total occlusion, incomplete CAG data, and failures in CAG exams.

The CatLet score and the lesion evaluation

The CatLet angiographic scoring system and its tutorial are available at www.catletscore.com (Internet Explorer or Microsoft Edge browser required). The CatLet score has been described elsewhere in detail.^[2] In short, this is a newly developed angiographic scoring tool based on the 17-myocardial segment model, law of flow conservation, and law of competitive blood supply. In the CatLet score, right coronary artery (RCA) was classified into six types: posterior descending artery (PDA) zero, PDA only, small RCA, average RCA, large RCA, and super RCA; left anterior descending artery (LAD) was classified into three types: short, average, and long; diagonals (Dx) was classified into three types: small, intermediate, and large, together resulting in a total of 54 types of coronary circulation pattern to be used to account for the variability in the coronary anatomy. The weighting assignment was according to the number of segments coronary arteries supply. Only the significant lesion with diameter stenosis ≥50% in vessels ≥1.5 mm in diameter was scored whereas the adverse characteristics pertinent to this lesion were not scored any more. The two observers independently evaluated the CAG images of AMI patients to assess the inter-observer reproducibility of the CatLet score. One of the two observers, blinded to the initial scoring results, reevaluated the initially-evaluated AMI patients to assess the intra-observer reproducibility of the CatLet score on two occasions at least 3 months apart to minimize the memory retention. The investigators were blinded to baseline characteristics, procedural data, and clinical outcomes.

Statistical analysis

Statistical analyses were completed with STATA/SE 15 (State Corp LP, College Station, TX, USA). The justification for a sample size of greater than 30 coronary angiograms has been provided elsewhere.^[15,16] In brief, a power calculation at 0.9 revealed that a group size of 30 coronary angiograms would be sufficient to achieve a reasonable kappa value.^[17] We used the kappa values to assess the agreement of inter-observer and intra-observer for the calculated CatLet score, the adverse characteristics, and the dominance parameters.^[18] Quantitative classification of kappa values as a degree of agreement beyond the level of chance was as follows: 0 < Kappa ≤ 0.2, slight; 0.2 < Kappa ≤ 0.4, fair; 0.4 < Kappa ≤ 0.6, moderate; 0.6 < Kappa ≤ 0.8, substantial; 0.8 < Kappa ≤ 1.0, almost perfect. For ordinal variables, we used the weighted kappa to express the degree of agreement. The larger the Kappa value is, the better the agreement is. For the evaluation of the CatLet score, we divided it into tertiles: ≤14, 15–22, >22 according to our previous study.^[1] All tests were two-sided. A value of P < 0.05 was considered significant for all statistical tests.

Results

For evaluation of the inter-observer variability of the CatLet score, a total of 54 patients were included for potential analysis, among whom 24 patients were excluded because of stenting history, normal CAG results, chronic total occlusion, incomplete imaging data, or failure to undergo the CAG exams. A total of 30 patients thus met the inclusion criteria for final analysis. For evaluation of the intra-observer variability of the CatLet score, a total of 66 patients were included for potential analysis, among whom 17 patients were excluded because of the above-mentioned reasons. A total of 49 patients thus met the inclusion criteria for final analysis. Figure 1 showed the flow diagram of patient enrollment.

Figure 1.

Flow diagram of patient selection for inter-observer study (A) and for intra-observer study (B). AMI: Acute myocardial infarction; CAG: Coronary angiography.

Clinical baseline data

In both intra-observer and inter-observer groups, AMI patients were more likely to be with the aged, male, hypertension, and smoking although there were no significant differences between these two groups. Table 1 presented the baseline characteristics and their proportions in corresponding groups.

Table 1.

Clinical baseline data and risk factors of AMI patients for intra- and inter-observer variability analysis.

Clinical data	Intra-observer (n = 49)	Inter-observer (n = 30)	Likelihood-ratio χ2	P
Age >65 years	27 (55.10)	21 (70.00)	1.76	0.19
Sex, male	40 (81.63)	26 (86.67)	0.35	0.56
Diabetes	9 (18.37)	6 (20.00)	0.03	0.86
Hypertension	29 (59.18)	17 (56.67)	0.05	0.83
Smoking	35 (71.43)	17 (56.67)	1.78	0.18
Alcohol use	19 (38.78)	8 (26.67)	1.24	0.27

Values were presented as n (%). AMI: Acute myocardial infarction.

Inter-observer variability of the CatLet score

The mean CatLet score calculated by observer 1 was 23.2 ± 16.2 (range 2–74), while that calculated by observer 2 was 24.7 ± 16.4 (range 4–70). The weighted kappa value for evaluation of the inter-observer variability was 0.86 (0.54–1.00, Z = 5.20, P < 0.001; tertile partitioning, ≤14, 15–22, and >22) [Figure 2A]. Regarding the adverse characteristics pertinent to the lesions and the dominance parameters, the kappa values for evaluation of the inter-observer variability were shown in Table 2.

Figure 2.

Inter- and intra-observer reproducibility of the CatLet score. Both (A) inter- observer (observer 1: 23.2 ± 16.2 [range 2–74], observer 2: 24.7 ± 16.4 [range 4–70]) and (B) intra-observer (initial calculation: 19.7 ± 9.9 [range 5–49]; second calculation: 20.7 ± 10.9 [range 5–56]) reproducibility showed an almost perfect degree of agreement for the scoring values. The values of the CatLet score were shown as mean ± standard deviation. κ: Kappa value; γ: Correlation coefficient.

Table 2.

Intra- and inter-observer kappa values for CatLet score, adverse angiographic characteristics, and dominance parameters.

Parameters	Intra-observer kappa (95% CI)	Z	P	Inter-observer kappa (95% CI)	Z	P
CatLet score, tertile	0.82 (0.59, 1.00)	7.23	<0.001	0.86 (0.54, 1.00)	5.20	<0.001
Total number of lesions	0.87 (0.68, 1.00)	8.99	<0.001	0.80 (0.56, 1.00)	6.47	<0.001
Trifurcation lesions	1.00 (0.78, 1.00)	9.06	<0.001	NA	NA	NA
Bifurcation lesions	0.85 (0.63, 1.00)	7.67	<0.001	0.57 (0.28, 0.85)	3.93	<0.001
Angulation	NA	NA	NA	NA	NA	NA
Aorto ostium	NA	NA	NA	NA	NA	NA
Heavy calcification	0.89 (0.67, 1.00)	8.08	<0.001	0.69 (0.43, 0.96)	5.06	<0.001
Length > 20 mm	0.66 (0.46, 0.86)	6.36	<0.001	NA
Tortuosity	0.65 (0.44, 0.87)	5.92	<0.001	1.00 (0.72, 1.00)	6.93	<0.001
Thrombus	0.77 (0.55, 0.98)	7.04	<0.001	0.54 (0.26, 0.82)	3.78	<0.001
The LAD length	0.85 (0.62, 1.00)	7.11	<0.001	0.69 (0.41, 0.96)	4.91	<0.001
The Dx size	0.54 (0.29, 0.80)	4.23	<0.001	0.22 (0.06, 0.51)	1.56	0.060
The RCA dominance	0.62 (0.44, 0.80)	6.76	<0.001	0.69 (0.48, 0.91)	6.29	<0.001

CatLet: Coronary Artery Tree description and Lesion EvaluaTion; CI: Confidence interval; LAD: Left anterior descending artery; Dx: Diagonal; RCA: Right coronary artery; NA: Not available because of the low number of cases.

Intra-observer variability of the CatLet score

The mean CatLet scores in the initial and second calculations by observer 1 were 19.7 ± 9.9 (range 5–49) and 20.7 ± 10.9 (range 5–56), respectively. The weighted Kappa value for evaluation of the intra-observer variability was 0.82 (0.59–1.00, Z = 7.23, P < 0.001; tertile partitioning, ≤14, 15–22, and >22) [Figure 2B]. Regarding the adverse characteristics pertinent to the lesions and the dominance parameters, the kappa values for evaluation of the intra-observer variability were shown in Table 2.

Discussion

This is a first study to evaluate the inter- and intra-observer reproducibility for the CatLet score. The main findings in the current study were as follows: (i) the reproducibility of the intra- and inter-observer for the CatLet score was almost perfect, (ii) the reproducibility of both inter- and intra-observer variability for the adverse characteristics pertinent to the lesions was at least moderate, and (iii) the reproducibility of the inter- and intra-observer variability for the dominance parameters was at least moderate except for the Dx size between the two different investigators.

Reproducibility of the CatLet score according to the terciles

In the current study, the reproducibility of the inter- and intra-observer for the CatLet score showed almost a perfect degree of agreement. The weighted kappa values for the intra-observer and inter-observer was 0.82 and 0.86, respectively, higher than those for the global SYNTAX score.^[14,15,19] In the SYNTAX score, the weighted kappa values of the inter-observer ranged 0.45 to 0.62, indicating a moderate or substantial degree of the agreement; likewise, the weighted Kappa values for the intra-observer ranged 0.51 to 0.78, also indicating a moderate or substantial degree of agreement.^[13–15] After an advanced training session with the angiographic core laboratory technicians, the reproducibility of the inter-observer and intra-observer for the global SYNTAX score scrambled to an almost perfect degree with kappa values of 0.82 and 0.84, respectively.^[16] Interestingly, the reproducibility of the CatLet score was comparable to that of the SYNTAX score without the advanced training. According to our new reclassification scheme proposed in the CatLet score, six types of RCA and three types of LAD or Dx were categorized, and all of the types progressively changed to reflect the variability in the coronary anatomy. Therefore, misjudgment, if present, usually happened in the two nearest neighboring coronary circulation patterns, and the effect of this misjudgment on the CatLet score was thus minimized. This could in part explain almost the perfect degree of agreement with high kappa values revealed in the current study. In the CatLet score, we only scored the stenosis lesions, and qualitatively evaluated those adverse characteristics, which also contributed to the almost perfect degree of agreement. By contrast, the SYNTAX score failed in reflecting the variability in the coronary anatomy, and the adverse characteristics, mostly not predicting clinical outcomes, were also scored, which led to the lower reproducibility of the SYNTAX score.^[11,14,16]

Reproducibility of the assessment of adverse characteristics pertinent to the stenosis lesions

In the CatLet angiographic scoring system, we did not score those adverse characteristics, such as bifurcation, trifurcation, ostial lesion, and so on, and only qualitatively recorded them. The reproducibility for the number of lesions, bifurcation, heavy calcification, tortuosity, and thrombus was at least moderate albeit a little lower than that of the CatLet score value. These results highlighted the difficulty in assessing the adverse characteristics pertinent to the lesions and also highlighted the possible advantages afforded by the CatLet score in assessing these adverse characteristics over the SYNTAX score. The reproducibility of the CatLet score for evaluating these adverse characteristics was overall better than that of the SYNTAX score. The reproducibility of the inter-observer for evaluation of the bifurcation lesions by the SYNTAX score demonstrated a fair degree of agreement, with kappa values of around 0.40, and struggled to increase to a moderate degree of agreement, with a kappa value of 0.56 after an advanced training session.^[14,16] In the SYNTAX score, the definition of bifurcation lesions was preset for the segment junctions such as 5/6/11, 6/7/9, and 7/8/10 whereas, in the CatLet score, however, the judgment of bifurcation lesions was completely dependent on the actual situations.^[3] The preset situations for bifurcation lesions underestimated the complexity of the actual situations, which could explain the lower reproducibility of the SYNTAX Score for evaluation of the bifurcation lesions.

Reproducibility for the evaluation of the coronary artery circulation patterns

Coronary dominance was traditionally divided into the right or left. In the CatLet angiographic scoring system, however, we put forward a new reclassification scheme according to RCA dominance, LAD length, and Dx size, based on which, 54 coronary circulation patterns were characterized to reflect the variability in the coronary anatomy. The reproducibility of the inter- and intra-observer for RCA dominance was a substantial degree of agreement with Kappa values of 0.62 and 0.69, respectively. For LAD length evaluation, the agreement degree was almost perfect with a kappa value of 0.85 for the intra-observer variability while the agreement degree was substantial with a kappa value of 0.69 for the inter-observer variability. For Dx size evaluation, the agreement degree was moderate with a kappa value of 0.54 for intra-observer variability. Unfortunately, the reproducibility of inter-observer for the Dx size was at best fair with a kappa value of 0.22. The left anterior oblique 40°/cranial 20° view was a standard one to be used to adjudicate the Dx size.^[20] However, this view was usually lacking in primary percutaneous coronary intervention. In this case, observers adjudicated the Dx size by default. One observer adjudicated most Dx size as intermediate while the other, as large. The systematic bias ensued, which were easy to be corrected from a statistical point of view. Correction of these systematic biases will improve the reproducibility of the CatLet score for Dx size evaluation.

Several limitations have to be considered for this study. First, the kappa values represented the agreement level beyond the chance. However, the actual chance agreement was variable and affected by the prevalence of a disease being studied.^[18] The prevalence of disease was relatively stable in a local region and clinical data were blinded to investigators, therefore, this affection was not particularly relevant for this study. Second, the borderline lesion (around 50% diameter stenosis) was a challenge and would greatly affect the scoring results. Less than 35% of borderline lesions were hemodynamically relevant as defined by the fractional flow reserve (FFR) ≤0.8.^[21] Therefore, the FFR-guided lesion evaluation would be expected to improve the reproducibility of the CatLet score. Third, the exclusion of small vessels (1.5–2.0 mm in diameter) also had comparable clinical outcome predictions with the FFR-guided revascularization.^[22,23] The current study, however, still scored the vessels ≥1.5 mm in diameter as in the SYNTAX score so as to compare with this widely used and intensively studied angiographic scoring system. Fourth, only AMI patients were enrolled in this study, and extrapolation of these results to other CAD populations should be cautious. Fifth, the inter- and intra-variability calculated by two different datasets rather than the same dataset may bias their true values. However, we assessed the Kappa values independently, without involving the statistical inference, which would not weaken the main findings revealed in this study. Finally, we failed to obtain the kappa values of angulation, aortal ostium lesion, and lesion length because of the low number of cases. In conclusion, the reproducibility of the CatLet angiographic scoring system for the assessment of the stenosis lesions, their pertinent adverse angiographic characteristics, and the dominance parameters, were substantial or excellent, within an acceptable range. The high reproducibility of the CatLet angiographic scoring system will boost its clinical application to AMI patients.

Funding

This work was in part supported by a grant from the Sci-Tech Development Program of Suzhou City, China (No. SYS2019040).

Conflicts of interest

None.