The SCQ-SCORE: initial validation of a new scoring system for elective stone surgery prioritization in the COVID-19 era

Abstract

Introduction

Endourology waiting lists have increased during the COVID-19 pandemic and prioritization strategies are needed. Some tiered classifications have been put forward aimed at prioritizing patients by using criteria related with clinical severity or social impact of stone disease, yet no quantitative system has been published to date. The objective of this study is to present a new quantitative scoring system for elective stone surgery prioritization and show its intra- and inter-rater reliability.

Material and methods

A scoring system coined ‘SCQ–score’ was set up, which consists of 9 variables: infection (ranges 0–3), obstruction (0–3), indwelling time (0–3), admissions (0–3), symptoms (0–2), ureteral location (0–1), solitary or suboptimal kidney (0–1), chronic kidney disease (0–1) and presence of percutaneous nephrostomy (0–1).

The intra- and inter-rater reliability of the SCQ-score was prospectively validated in 60 consecutive patients on the waiting list, by calculating the intraclass correlation coefficient (ICC).

Results

The SCQ-score demonstrated having an excellent interobserver agreement (ICC >0.75) for the final score and its different domains. After 4 weeks, a second analysis was carried out to measure its intra-rater reliability, which was also excellent. On average, 134.9 ±50 seconds were required to complete the SCQ-score.

Conclusions

The SCQ-score is a new quantitative system to help prioritize elective stone surgeries, which has been shown to be user-friendly and to have an excellent intra- and inter-rater reliability. Initially developed to help during the COVID-19 pandemic, its utility will probably remain of interest in the post-COVID-19 era to ensure a fairer access to stone surgery.

INTRODUCTION

On March 11, the World Health Organization declared the coronavirus disease (COVID-19) a pandemic. During the worst days of the pandemic in Spain, many hospitals could only focus on COVID-19 patients. Many operating rooms (ORs) were turned into intensive care units (ICUs), therefore were only available for high-risk oncological conditions or emergencies. Consequently, surgical waiting list times experienced a considerable increase that put our healthcare system organization at stake [1].

Once recovered from the peak of the disease, as more ORs become available, Urology Departments faced the challenge of rescheduling postponed elective surgical procedures [2].

Among the different conditions on the surgical waiting list, stone disease has a considerable weight. An excessive delay in urolithiasis treatment can lead to infection and sepsis, to an irreversible loss of renal function or the entire renal unit, have a negative impact on quality of life and implies a socioeconomic burden. Therefore, it is paramount to perform timely interventional stone treatment [3, 4, 5].

Lots of tiered classifications have arisen recently aimed at prioritizing urological conditions on the waiting list. In particular, the European Association of Urology (EAU) guidelines office adapted the EAU guidelines to the COVID-19 era, and published their recommendations for different urological conditions, including stone disease [6]. These classifications seek to bring greater equity to the system, where patients with the most serious conditions would be operated on first. Some of these proposals focus on interventional stone treatment, yet no quantitative system has been published in this area to date [7–11].

The objective of this prospective study is to present a new scoring system for elective stone surgery prioritization and show its intra- and inter-rater reliability.

MATERIAL AND METHODS

Urologists posted to the Endourology Unit of the University Hospital Complex of Santiago de Compostela (DPF, CFB, JNO and RDG) decided to elaborate a quantitative system to help prioritize stone patients on the waiting list for the post-COVID-19 period. DPF has been dedicated to these procedures since 2008, while CFB, JNO and RDG have around 3 years of experience in endourology. The study was approved by the local ethics committee and follows the principles of the Declaration of Helsinki.

We initially carried out an extensive review of the literature using PubMed to identify the possible consequences of a delay in urolithiasis treatment, along with major clinical or social criteria to be considered when prioritizing patients on the waiting list for stone treatment. A literature review was also conducted to identify how these waiting list patients are prioritized in other centers.

Several meetings were held during the lockdown period using Jitsi Meet videoconferencing solution (https://meet.jit.si/). During the first meeting, we discussed a previously drawn up list of variables. Using a Delphi consensus process, we decided to keep those we considered to be the most important when prioritizing a stone case on the waiting list. Some sociodemographic variables such as age, were not considered appropriate to be included in the score. By consensus, different weights were given to each variable, depending on the importance of the item for surgical priority.

The feasibility of the first version of the scoring system was assessed in 20 waiting list patients. Access to the medical records was possible due to the electronic medical database used in our region.

During the second meeting, all the problems found when applying the scoring system were shared and discussed. We reached an agreement on which items should be kept and discarded those that were not possible to be assessed by the medical records, were mutually exclusive or at risk of being duplicated. The weights of the different variables were adjusted again by consensus. A pilot analysis was carried out in 40 patients, to once more assess its feasibility and reliability. The interobserver agreement was measured calculating the intraclass correlation coefficient (ICC) for each variable. The different rates were again considered and modified where necessary. A third and last online meeting was held. The results of the previous analysis were shared with the panel. In those variables where a weak interobserver agreement was found, the definition was reviewed so as to make it clearer and reduce the variability in its interpretation. Again, weights were discussed and recalibrated. The final version of the scoring system was set up, which we decided to coin ‘SCQ-score’. SCQ is the 3-letter location code used for Santiago de Compostela in the International Air Transport Association (IATA) language [12]. The flowchart of the method used to design the scoring system is depicted in Figure 1.

Figure 1.

Flowchart diagram of the SCQ-score design.

The SCQ–score consists of 9 variables: infection at inclusion or while on waiting list (ranges 0–3), upper urinary tract obstruction (ranges 0–3), urinary diversion indwelling time (ranges 0–3), admissions related to the stone case while on the waiting list (ranges 0–3), symptoms at inclusion or while on the waiting list (ranges 0–2), ureteral location of the stone (ranges 0–1), solitary or suboptimal kidney (ranges 0–1), chronic kidney disease (ranges 0–1) and patients with percutaneous nephrostomy (ranges 0–1). The final score is calculated for each patient by adding the weights assigned to the different factors. The SCQ-score ranges from 0 to 18, with the highest scores representing the highest surgical priority. The SCQ-score is shown and explained in more detail in Table 1 and illustrated with examples in Figure 2.

Table 1.

The SCQ-score

1	INFECTION at inclusion or while on waiting list (Y/N)	N = 0 Y = 3
2	UUT OBSTRUCTION while on waiting list (Y/N)	N = 0 Y = 3
3	URINARY DIVERSION INDWELLING TIME (DJ/PCN) A: No stent B: 0–3 months C: 3–6 months D: >6 months	A = 0 B = 1 C = 2 D = 3
4	ADMISSIONS stone-case related while on waiting list (Y/N)	N = 0 Y = 3
5	SYMPTOMS at inclusion or while on waiting list (Y/N)	N = 0 Y = 2
6	URETERAL LOCATION of the stone (Y/N)	N = 0 Y = 1
7	SOLITARY or SUBOPTIMAL KIDNEY (DRF <35%) (Y/N)	N = 0 Y = 1
8	CHRONIC KIDNEY DISEASE (eGFR < 60 ml/min) (Y/N)	N = 0 Y = 1
9	PERCUTANEOUS NEPHROSTOMY (Y/N)	N = 0 Y = 1
SCQ-SCORE		0–18

1. Infection at inclusion or while on waiting list: sepsis, pyelonephritis, pyonephrosis (lower UTIs are excluded).
2. UUT obstruction: moderate to severe dilation of, at least, a renal calyx without urinary diversion or despite having a stent. All the imaging tests available should be reviewed.
3. Urinary diversion indwelling time (double J or percutaneous nephrostomy): since its first placement, in months.
4. New admissions to the hospital while on waiting list, related to the stone case.
5. Symptoms at inclusion or while on waiting list, related to the stone case (stent symptoms included): hematuria, loin pain, recurrent lower UTIs, dysuria. Includes patients needing sick leave or are care-dependent due to the stone burden.
6. Stone located in the ureter.
7. Stone in a solitary kidney, either anatomical or functional (DRF <15%) or in a suboptimal kidney (DRF < 35%, if no DMSA scan available, then answer NO)
8. Chronic kidney disease (CKD): eGFR below 60 ml/min for 3 months or more. eGFR calculated using the CKD-EPI equation, available at https://www.senefro.org/modules.php?name=calcfg
9. PCN: patient bearing a percutaneous nephrostomy

Y/N – yes/no; DJ – double J; DRF – differential renal function; UUT – upper urinary tract; UTIs – urinary tract infections; CKD-EPI – Chronic Kidney Disease Epidemiology Collaboration

Figure 2.

Practical application of the SCQ-score in three different cases of our series.

Case 1. Solitary stone in lower pole. Asymptomatic, no obstruction or infection. No admissions while on waiting list. Normal renal function. No stents.

Case 2. Ureteric stone. Pyonephrosis that required double J stent placement 4 months ago. Now asymptomatic. GFR 50 ml/min.

Case 3. Renal pelvis stone. Sepsis requiring double J placement 7 months ago. New admission with double J malfunction (obstruction), a nephrostomy was placed. Very symptomatic.

This final version of the SCQ-score was applied to 60 consecutive patients on our waiting list, to assess its inter-rater reliability. Score completion time was measured by all the observers. The analysis was repeated 4 weeks later in the same patients to calculate its test-retest reliability.

The participants’ demographic and clinical characteristics were analyzed using descriptive statistics. The results were presented as mean ±SD or n (%). Light's kappa for agreement and their corresponding 95% confidence intervals were calculated to assess inter-rater reliability for each item of the SCQ-score [13]. Test-retest reliability was also assessed for each item. For the total SCQ-score, two-way random-effects models with absolute agreement were applied to calculate the intraobserver and interobserver reliability, using the ICC, as described by Eliasziw et al. [14]. To avoid the normality assumption, the associated two-sided 95% confidence intervals for the reliability coefficients were obtained using the non-parametric percentile bootstrap method [15], with 1000 replicates. Absolute consistency was quantified using the SEM. We used the following criteria to interpret the ICC: 0.00–0.39, poor; 0.40–0.59, fair; 0.60–0.74, good; and 0.75–1.00, excellent [16].

All statistical analyses were carried out in R version 3.5.1 using the packages ‘irr’ and ‘psy’. These packages are freely available at http://cran.r-project.org. Sample size for reliability was based on the one-way random effects model, which tends to provide more liberal estimates compared to those based on the two-way models. Assuming a true ICC of 0.8, evaluation of 60 patients 2 times by 4 readers would yield an approximately 90% chance of obtaining a lower bound for the two-sided 95% CI for an ICC greater than 0.6 [17].

RESULTS

The characteristics of the patients and stone-cases included in the final analysis are described in Table 2. In the first assessment, the interobserver agreement was excellent (ICC >0.75) for all the items of the SCQ-score and its final score (Table 3).

Table 2.

Characteristics of the patients and stone-cases included in the SCQ-score analysis (n = 60)

Age, years	60.6 ±15.2
No. female (%)	36 (60)
No. left kidney (%)	30 (50)
BMI, kg/m2	30 ±6.7
ASA score	2.1 ±0.8
Stone burden (mm2)	135 [50, 317]
Hounsfield units	1057 [671, 1396]
Stone location  No. ureter (%)  No. renal (%)  No. renal and ureter (%)	11 (18) 42 (70) 7 (12)
Guy’s stone score¶  No. grade I (%)  No. grade II (%)  No. grade III (%)  No. grade IV (%)	14 (29) 25 (51) 6 (12) 4 (8)
No. with UUT obstruction (%)	22 (37)
No. with double J stent (%)	22 (37)
No. with nephrostomy tube (%)	3 (5)
No. with infection at inclusion or while on waiting list (%)	5 (8)
No. symptomatic (%)	37 (62)
No. admitted while on waiting list (%)	7 (12)
No. suboptimal or solitary kidney (%)	5 (8)
No. chronic kidney disease (%)	5 (8)

Results are expressed in mean ± SD or median [p25, p75]; BMI – body mass index; ASA – American Society of Anaesthesiologists; UUT– upper urinary tract; ¶ – Guy’s stone score was calculated only for renal stones

Table 3.

Interobserver agreement in the different variables included in the SCQ-score

Variables	ICC (95% CI)
Infection at inclusion or while on waiting list	0.867 (0.811–0.911)
UUT obstruction while on waiting list	0.798 (0.720–0.862)
Urinary diversion indwelling time	0.984 (0.976–0.990)
Stone-burden related admissions	0.811 (0.737–0.872)
Symptoms at inclusion or while on waiting list	0.756 (0.667–0.832)
Ureteral location of the stone	0.961 (0.942–0.974)
Solitary or suboptimal kidney	0.921 (0.886–0.948)
Chronic kidney disease	1.000 (1.000–1.000)
Percutaneous nephrostomy	0.779 (0.696–0.849)
SCQ-score	0.879 (0.827–0.919)

ICC– intraclass correlation coefficient; UUT – upper urinary tract

Four weeks later, the test-retest reliability was measured. The SCQ-score showed an excellent intra-rater agreement for all the investigators. The variable in which the highest agreement was found was infection at inclusion or while on the waiting list, with an ICC >0.90 for the 4 investigators, followed by the indwelling time variable. These results are shown in Table 4. On average, 134.9 ±50 seconds were required to complete the SCQ-score (148.1 ±51.9 seconds for investigator 1, 109.7 ±40.9 seconds for investigator 2, 129.7 ±46.2 seconds for investigator 3 and 152.3 ±55.5 seconds for investigator 4).

Table 4.

Test-retest reliability of the different variables included in the SCQ-score

Variables	ICC (95% CI)
	Investigator 1	Investigator 2	Investigator 3	Investigator 4
Infection at inclusion or while on waiting list	0.902 (0.840–0.940)	0.902 (0.840–0.940)	0.915 (0.862–0.948)	0.915 (0.862–0.948)
UUT obstruction while on waiting list	0.953 (0.923–0.972)	0.688 (0.528–0.801)	0.762 (0.631–0.851)	0.816 (0.711–0.886)
Urinary diversion indwelling time	0.843 (0.939–0.978)	0.953 (0.922–0.972)	0.991 (0.985–0.994)	0.932 (0.889–0.959)
Stone–burden related admissions	1.000 (1.000–1.000)	0.649 (0.475–0.774)	0.851 (0.762–0.908)	1.000 (1.000–1.000)
Symptoms at inclusion or while on waiting list	0.867 (0.786–0.918)	0.732 (0.589–0.831)	0.764 (0.634–0.852)	0.790 (0.671–0.869)
Ureteral location of the stone	0.794 (0.677–0.871)	0.863 (0.781–0.916)	0.791 (0.673–0.869)	0.752 (0.616–0.844)
Solitary or suboptimal kidney	1.000 (1.000–1.000)	0.902 (0.840–0.940)	0.713 (0.561–0.818)	0.788 (0.669–0.868)
Chronic kidney disease	1.000 (1.000–1.000)	0.815 (0.708–0.885)	0.915 (0.862–0.948)	0.843 (0.751–0.903)
Percutaneous nephrostomy	0.794 (0.678–0.872)	0.794 (0.678–0.872)	0.794 (0.677–0.872)	1.000 (1.000–1.000)
SCQ-score	0.930 (0.886–0.958)	0.858 (0.773–0.913)	0.853 (0.766–0.910)	0.862 (0.780–0.916)

ICC – intraclass correlation coefficient; UUT – upper urinary tract

DISCUSSION

From March 14, 2020 to May 1, 2020, all the endourological surgeries except the emergencies were cancelled in our hospital due to the COVID-19 outbreak. In the aftermath of the hot phase of this pandemic, our waiting list for stone surgery was considerably increased and fewer operating rooms were available for elective stone treatment than before the crisis. Hence, we thought that a priority analysis of these cases, including clinical criteria beyond the mere concept of time, was mandatory.

In our region, a 3-tiered classification is used to establish surgical priority: level 1 (under 30 days for surgery), level 2 (1 to 3 months) and level 3 (up to 6 months). Stone cases are mainly included in priority groups 2 and 3. In each group, patients are ordered according to time waited, following the ‘first-in, first-out’ principle [18].

By using this vague description, it is uncertain that all our stone patients are receiving their needed services in a timely manner, just as it is impossible to ensure that those cases with the highest urgency are served first. Thus, a better and standardized method to stratify the priority of these patients is paramount, in order to give fairness, equity and transparency to the system accessibility for elective stone interventions.

This change of organizational model has some legal implications, since some patients that were close to be scheduled might, according to the new clinical priority policy, now have their surgery delayed. Therefore, it is important to involve the hospital administrators in the validation of this new standardized system, as well as to collaborate with associations of patients to further explain the benefits of its implementation [19].

In response to the COVID-19 pandemic and its collateral effects on OR accessibility, many groups and societies have published their recommendations and algorithms for elective stone surgery triaging. Overall, these encompass the same criteria we have included in the SCQ-score: obstructing vs. non-obstructing stones, ureteral vs. renal location, patients with stents, stones with infection, in a solitary kidney or with renal function impairment, and lithiasis causing symptoms. However, none of them use a point-count system to stratify the need for surgery [6–11].

The method we have used to create and validate the SCQ-score is akin to that previously reported by groups from other specialties and conditions. Overall, it was initiated with a thorough literature review to identify those criteria that would reflect the benefit expected from elective surgery. Subsequently, a panel of experts discussed the adequacy of the different items and its relative weights in the final score. Then, the pilot scoring system was applied to a set of patients to analyze its feasibility. After a new discussion and revision of the tool, its reliability was assessed in a larger cohort, to finally establish the definitive scoring system. The resultant score should represent the degree of benefit expected from surgery [20–26].

We ultimately decided to include 9 variables in our scoring system that have shown to correlate well with the clinical severity or social impact of stone disease, ensuring its face validity. Items that were not easy to calculate or obtain straightaway from the electronic medical records were discarded, in order to develop a quick and practical tool for prioritization. Factors that could be controversial when prioritizing patients on the waiting list, such as age or cost, were not included either [27, 28].

We reached an agreement in giving the highest weight in the SCQ-score to cases with obstruction, upper urinary tract infection, to those who needed admissions while on the waiting list or with stents for more than six months. The rationale behind this decision is based on our experience and literature review, and it is in accordance to other classifications of priority. Infection is the most relevant complication in stone disease, which can be life-threatening in some cases [29]. Upper urinary tract obstruction due to stone disease can also be a serious situation when urine above the level of obstruction becomes infected and, depending on its degree and time of evolution can lead to a complete and irreversible loss of the kidney [30, 31, 32]. In situations of obstructive uropathy, urinary diversion is temporarily guaranteed with stents, yet they can malfunction due to encrustation, or put the patient at a greater risk of sepsis, which is directly related to the indwelling time [33, 34, 35].

We decided to allot two points in the scoring system to those symptomatic patients due to the calculus. Patients bearing stones can suffer from different symptoms that, depending on their severity, can interfere with normal daily activities needing sick leave or a caregiver [36].

Finally, one additional point was assigned to cases with ureteral stones, calculi in solitary or suboptimal kidneys, with chronic kidney disease and with nephrostomy tubes. Ureteral calculi can cause silent obstruction that could lead to a loss in renal function and, for us, these must be in a higher risk group than that of stones located in the kidney. Considering their decreased functional reserve, solitary, suboptimal and patients with preexisting renal insufficiency might need special attention [37]. Finally, nephrostomy catheters are known to be at a higher risk of accidental dislodgement requiring replacement [38]. The SCQ-score resulted in having an excellent interobserver agreement for all the variables and its final score. Regarding the test-retest reliability of the final score, it demonstrated to be excellent for all the investigators [39].

Our study has some limitations that deserve commenting on. By using this tool, it is likely that the cases with the lowest score will have difficulties in having their surgery scheduled. One possible solution to this drawback is to adjust the priority score by waiting times, as some authors have proposed [40]. The weight we have assigned by consensus to each variable is a consequence of an agreement, in which our local practice and culture may have swayed the decision. Although we have based this judgment on a thorough review of the literature and our experience, we have to acknowledge that other groups would have allotted different weights to some criteria or included different ones. Moreover, it is likely that implementing this scoring system in other regions or countries will be difficult, mainly for cultural or organizational reasons. One of the limitations of scores using linear models is that two patients with identical scores can actually be different in clinical priority, for reasons that cannot be discriminated by the scoring system only. Hence, the results of the SCQ-score should have an important but not absolute weight in the decision-making process.

Finally, prioritization of patients on the waiting list must be a dynamic process, since the conditions of the patients may vary across time. Therefore, maintaining a regular audit of the list and updating the score is mandatory. The waiting list and priorities are reviewed on a monthly basis, trying to keep waiting times for non-emergent stone surgery between 3 and 6 months, if possible.

To the best of our knowledge, the SCQ-score is the first quantitative scoring system for elective stone surgery triaging that has been validated in the literature. Once this system is implemented, it should be easier to justify why a patient has to wait longer for surgery, going beyond the classic criteria of time spent on the list. One of the drawbacks of some priority systems is that they are time-consuming. However, the SCQ-score has shown to be a user-friendly tool, which can be completed using electronic medical records in less than 3 minutes and it has demonstrated to be reliable among investigators with different degrees of experience. We have to acknowledge that the different items included and their weights were a result of an expert consensus. Hence, future studies with larger series of patients, the prospective application of our tool by other groups on different samples and the analysis of the correlation between the SCQ-score implementation and surgery outcomes will help to improve this scoring system and demonstrate its external validity.

CONCLUSIONS

The SCQ-score is a new quantitative system to help prioritize elective stone surgeries. Although it was initially developed to help during the COVID-19 pandemic, in which the waiting lists have increased due to a more restricted access to operating rooms, its utility will probably remain of interest in the post-COVID-19 era to ensure a fairer access to surgery for patients with stone disease. This scoring system has demonstrated to be quick and easy to apply and has proved to have a good inter- and intraobserver reliability. Future applications in other endourology units will help assess its external validity.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.