Clinician Versus Nomogram Predicted Estimates of Kidney Stone Recurrence Risk

Abstract

Purpose: Kidney stone recurrence rates vary between patients. A patient's risk informs the frequency and intensity of preventative interventions. Clinicians routinely use clinical experience to estimate risk. We sought to compare clinician estimated recurrence risk with the recurrence of kidney stones (ROKS) nomogram.

Materials and Methods: We surveyed members of the Endourological Society with clinical expertise in kidney stones. Respondents estimated the risk of recurrence for patients in three clinical vignettes corresponding to low, intermediate, and high recurrence risk from the nomogram. Clinician estimates were compared with ROKS estimates.

Results: The majority of the 318 respondents were from North America (n = 127, 40%). The most commonly estimated recurrence was 50% at 5 years. The respondents' estimates were significantly different from the ROKS predicted recurrence rate for all cases (Case 1, 50% vs 93% p < 0.0001; Case 2, 50% vs 60% p < 0.0001; Case 3, 60% vs 22% p < 0.0001). The ROKS predicted estimates ranged from 22% to 93%, whereas the median urologist-derived 5-year risk estimates for each case ranged from 50% to 60%. The median range of estimates by respondents across cases was 20%, narrower than the 71% for the ROKS nomogram. The majority of respondents (95%) do not use nomograms in practice, mostly because of lack of awareness of useful nomograms (59%).

Conclusions: This study suggests that clinicians may not be able to distinguish those with high and low recurrence risk when compared with peers and when compared with a nomogram. Clinical decision support tools are needed to enable clinicians to better estimate stone recurrence risk.

Introduction

Kidney stones are prevalent,¹ costly,² and can have multiple symptomatic recurrences over a lifetime.³ Estimates of kidney stone recurrence rates range from 30% to 50% at 10 years.^4,5 This represents a high recurrence rate overall, but individual patients with kidney stones may have vastly different recurrence rates.⁶

Inaccurate risk estimation among patients at lower risk for recurrence can result in more frequent testing that is burdensome, and may lead to overuse of preventative dietary and pharmacologic interventions with associated side effects. Conversely, inaccurate risk estimation among patients at a higher risk of recurrence may lead to undertreatment with preventative interventions, contributing to future stone recurrence events. Therefore, individualized estimation of stone recurrence risk, if accurate, allows the clinician to tailor the preventative interventions to the risk.

Nomograms have been published to improve the accuracy of recurrence risk estimation.^4,6 The most widely used of these is the recurrence of kidney stones (ROKS) nomogram, which estimates recurrence based on 11 items in the medical history.⁴ However, many clinicians rely on experience and clinical gestalt to estimate an individual patient's risk of recurrence. The commonly cited estimate of 50% recurrence at 10 years applies to a select population of stone formers and may mislead clinicians when addressing individual patients.⁴ The effect of integration of nomograms into clinical workflow is not known to date, and no comparison has been performed between nomogram and clinicians' estimates of kidney stone recurrence rates. In addition, the impact of an individual patient's stone recurrence risk on how clinicians perceive appropriateness of intervention and surveillance has not been assessed. Therefore, we sought to characterize stone recurrence estimates using nomograms compared with clinical judgment.

Materials and Methods

Urologists with clinical expertise in kidney stone disease were surveyed in April and May 2020 after obtaining institutional review board approval (IRB#200435). An electronic questionnaire was sent to the international e-mail list of Endourological Society members with expertise in kidney stone disease. Demographic information on respondents was collected, including geographic location, year of residency graduation, and any fellowship training related to stone disease. After this, three hypothetical clinical vignettes were provided (Supplementary Appendix SA1), each detailing a patient's presentation with a kidney stone, and including relevant medical history. Enough information was included in each vignette to fully calculate the recurrence risk at 5 years using the ROKS nomogram, although respondents were not prompted to do so. Respondents were then asked to estimate the risk of recurrence of a symptomatic kidney stone episode within 5 years from 0% to 100% in 10% increments.

Respondents were then surveyed on their use of nomograms in clinical practice, type of nomogram used, if applicable, and reason(s) for not using nomograms. Finally, respondents were asked how their clinical management would differ for patients with a high compared with a low risk of kidney stone recurrence. Statistical analysis was performed using a one sample Wilcoxon rank sum test comparing respondent estimates with the ROKS estimate (RStudio).⁴

Results

The survey was filled by 318 respondents, with 261 complete and 57 incomplete surveys. The survey response rate was 13% based on the mailing list size estimate provided by the Endourology Society of 2500. The demographic information of respondents is outlined in Table 1. The majority of respondents were from North America (n = 127, 40%), followed by Europe (n = 67, 21%), Asia (n = 38, 12%), and South America (n = 37, 11%). North American respondents were distributed across all American Urology Association (AUA) sections, with a maximum of 26 (20%) in the South Central section and a minimum of 8 (6%) in the New York and Mid-Atlantic sections. The majority of respondents were in an academic/university practice (n = 193, 61%), with 92 (29%) in solo/community/private practice and 30 (10%) working in a government facility. Most respondents graduated during or after 2001 (n = 194, 62%) and had fellowship training in stone disease after residency (n = 195, 63%). More than half (n = 171, 55%) of respondents dedicate at least 50% of their practice to stone disease.

Table 1.

Demographic Characteristics of Survey Respondents

Demographic information	No. (%)
Total survey respondents	318
Complete surveys	261 (82)
Incomplete surveys	57 (18)
No case estimates provided	22 (7)
Some case estimates missing	18 (6)
Nomogram/impact on practice incomplete	11 (3)
Blank survey	3 (1)
Demographic information incomplete	3 (1)
Location of practice
North America	127 (40)
Europe	67 (21)
Asia	38 (12)
South America	37 (12)
Middle East	26 (8)
Africa	13 (4)
Other	7 (2)
AUA section membership (if applicable)
South Central	26 (21)
Northeastern	21 (17)
Western	21 (17)
North Central	20 (16)
Southeastern	14 (11)
New England	9 (7)
New York	8 (6)
Mid-Atlantic	8 (6)
Practice environment
Academic/university	193 (61)
Solo/community/private practice	92 (29)
Government facility	30 (10)
Year of residency graduation
2011–2019	84 (27)
2001–2010	103 (33)
1990–2000	83 (27)
<1990	35 (11)
Current resident	7 (2)
Fellowship training in stone disease
Yes	195 (63)
No	117 (38)
Percentage of practice dedicated to stone disease
>50%	171 (55)
20–50%	121 (39)
<20%	20 (6)

ROKS = recurrence of kidney stones.

Respondent and ROKS nomogram estimates of symptomatic kidney stone recurrence within 5 years for each of the cases are shown in Figure 1. The most common clinician estimated risk of recurrence for all three cases was 50% at 5 years. In all cases, the respondents' estimates were significantly different than the ROKS predicted recurrence rate (Case 1, 50% [interquartile range; IQR 30%–50%] vs 93% p < 0.0001; Case 2, 50% [IQR 40%–70%] vs 60% p < 0.0001; Case 3, 60% [IQR 40%–70%] vs 22% p < 0.0001).

FIG. 1.

Estimated chance of recurrence of symptomatic kidney stone episode within 5 years by number of respondents. Each panel represents 1 of the 3 numbered cases. Means were Case 1, 48%; Case 2, 53%; Case 3, 58%. ROKS nomogram estimates (red arrows) were Case 1, 93%; Case 2, 60%; Case 3, 22%. All differences between respondent and ROKS estimates were significant, p < 0.0001. ROKS = recurrence of kidney stones. Color images are available online.

The difference between the maximum and minimum estimate of risk for each respondent was calculated to determine the range of risk estimates for each individual clinician (Fig. 2). Nearly all of the respondents who answered multiple cases (n = 271, 99%) had a narrower range in risk estimation than the ROKS nomogram's range of 71% for the clinical vignettes. The median range of estimates for clinicians was 20% (IQR 20%–40%). Twenty-two respondents (8%) estimated the same risk for all patients.

FIG. 2.

Absolute difference between maximum and minimum estimated recurrence rate between cases for each participant, grouped by size of estimated range. Absolute difference between ROKS nomogram estimates of recurrence rate between cases labeled in red. ROKS = recurrence of kidney stones. Color images are available online.

Respondents' self-reported use of nomograms in clinical practice is given in Table 2. Overall, 14 respondents (5%) reported using nomograms in practice, of whom 12 used the ROKS nomogram. Of the remaining two respondents, one used a personal nomogram and one did not provide the nomogram name. When queried on reasons for not using a nomogram, the most common reason was lack of known useful nomograms (n = 150, 59%), and the second most common reason was the opinion that nomograms would not be useful (n = 69, 27%).

Table 2.

Nomogram Use and Potential Use in Clinical Practice

Survey question	No. (%)
Nomogram use in clinical practice?
Yes	14 (5)
No	254 (95)
Type of nomogram used?
ROKS	11 (79)
Other	3 (21)
Reason(s) nomograms are not used
Too much work	47 (19)
Will not be useful	69 (27)
Unaware of useful nomograms	150 (59)
Other	8 (3)
Knowing a patient's risk of stone recurrence would change your clinical practice in
metabolic testing (e.g., 24 hours urine)
Yes	221 (83)
No	45 (17)
Frequency of surveillance imaging
Yes	226 (85)
No	40 (15)
Modality of surveillance imaging
Yes	147 (55)
No	119 (45)
Office visit frequency
Yes	205 (77)
No	61 (23)

Respondents reported that their intensity of work-up and treatment would differ among patients if they had a high vs low recurrence risk (Table 2). For example, a patient with an 80% risk of recurrence would have a more intense work-up and treatment than a patient with a 20% risk of recurrence. Specifically, metabolic testing would differ for 221 (83%) of respondents, frequency of surveillance imaging for 226 (85%), modality of surveillance imaging for 147 (55%), and frequency of office visits for 205 (77%).

Discussion

Overall, these data show several important findings. First, most urologists (95%) with expertise in kidney stone disease primarily use clinical judgment to estimate the risk of stone recurrence rather than relying on nomograms. Only 5% of respondents use nomograms clinically to predict recurrence risk for patients. This is notable, since clinical judgment compared with nomogram estimates was markedly different. Second, clinicians may not be able to discriminate between those patients with high and low recurrence risk. In the clinical scenarios, the nomogram predicted estimates ranged from 22% to 93%, whereas the median urologist-derived 5-year risk estimates for each case were narrowly from 50% to 60%. These estimates were nearly identical for fellowship and nonfellowship trained urologists, with a median 50% recurrence risk estimated by fellowship and nonfellowship urologists in all cases except Case 3. In this case, fellowship trained clinicians were less accurate (median estimate 60%, IQR 40%–70%) than non-fellowship trained clinicians (median 50%, IQR 50%–72.5%) with respect to concordance with the ROKS estimate (22%). The similarity between fellowship and non-fellowship groups may be because of the high percentage of stone practice in both groups, and membership in the Endourological Society. Third, at the least, these data show that clinicians do not agree on recurrence risk at the individual patient level. The clinical estimates of kidney stone recurrence varied substantially among urologists for each case. Clinician estimates ranged from 10% to 100% for Cases 1 and 2, and from 10% to 90% for Case 3. Overall, these findings have clinical relevance, since the clinicians reported that they would alter their investigations and surveillance based on perceived recurrence risk. For example, challenges in estimating recurrence risk may, in part, explain variation in rates of follow-up surveillance^7,8 and preventative treatment.⁹

There are barriers to broader dissemination and use of predictive nomograms among urologists. One issue is the perceived or actual time burden of nomogram use. In this study, 19% (n = 47) of survey respondents were of the opinion that nomograms are too much work for routine use. Nomograms that require opening a separate window on the computer and manually entering patient data can be time consuming and prone to error; these also violate the principle of Five Rights of Clinical Decision Support (CDS), which recommends providing the right information to the right people, in the right intervention formats, through the right channels, at the right points in workflow.^10,11 Instead, groups have shown success with integrating nomograms directly into electronic health records (EHRs) in fields such as pediatrics and psychiatry.^12,13 However, widespread adoption of EHR integrated CDS tools, such as nomograms, is difficult, as the burden of integration is often placed on each individual institution. Strategies to improve uptake include sharing design process, configuration, and code between institutions.¹⁴

A further barrier identified in this study is uncertainty as to the utility of nomogram estimates of kidney stone recurrence. A majority of respondents reported having no knowledge of useful nomograms (n = 150, 59%) and many reported a perceived lack of nomogram usefulness (n = 69, 27%). The ROKS nomogram (2014 edition) comprised 11 questions used to estimate risk of kidney stone recurrence (Supplementary Appendix SA2).⁴ Whereas nomograms often suffer from a lack of external validation,¹⁵ the ROKS nomogram has been externally examined. Using a different patient population, the ROKS nomogram was assessed and showed moderate predictive value for stone recurrence rates with an area under the curve (AUC) of 0.61 at 5 years.¹⁶ However, the authors used the outcome of surgical intervention for stone recurrence, rather than overall symptomatic stone recurrence as was originally intended for ROKS.¹⁶ The ROKS nomogram was also prospectively validated on a patient population in the same area it was created, and the AUC was 0.69 for symptomatic recurrence and 0.79 for any recurrence.¹⁷ Future research is needed to refine and validate usable nomograms across different populations for clinical use.

Another finding from this study is that clinicians report that stone recurrence risk estimates impact clinical care. This is not surprising, but nevertheless necessary to demonstrate if investments are made toward developing CDS tools. The majority (85%) of survey respondents indicated that they would alter the frequency of surveillance imaging for a given patient if they had a low compared with high recurrence risk. Whether or not metabolic testing such as a 24-hour urine collection would be ordered differed for 83% of respondents based on a patient's risk. It is likely that more accurate estimation of risk would help clinicians to determine an individual patient's need for such an investigation. The frequency and modality of follow-up imaging and the frequency of office visits would also be altered by the majority of respondents based on a patient's recurrence risk (Table 2). This would have an impact on the total cost of investigating and treating stone disease.

The methodology of this study has several limitations. The respondents to this survey, who were mostly fellowship trained endourologists, may not represent the views of clinicians from other backgrounds in managing patients with kidney stones. The response rate of 13% is similar to other endourology surveys,¹⁸ and may indicate the time limitations of endourologists. The membership of the Endourological Society in general is self-selected, and further selected among those who responded to the survey invitation. The definition of kidney stone recurrence was one of many possible definitions used in the literature, which could affect clinician estimates or prevent generalizability of these results to other types of asymptomatic stone recurrence.¹⁹ In addition, it is not known which estimates, those derived from clinical judgment or those from the ROKS nomogram, more accurately reflect the true recurrence risk. Although the ROKS nomogram was derived from a large longitudinal cohort study, the AUC ranges from 0.61 to 0.79 in two separate external validations. Since the original ROKS nomogram was released in 2014, there has been a subsequent update in 2018.^4,6 In this study, we chose the initial version because it has fewer questions, and we expected clinicians to have had more time to adopt its use.

Conclusions

Since recurrence risk varies across patients, it is necessary to accurately individualize and personalize care for a given patient. This study shows that clinicians and nomograms are not concordant for predicting stone recurrence risk. Furthermore, clinicians do not appear to discriminate among those at high and low risk for recurrence, and there is disagreement among clinicians on recurrence risk at the individual patient level. Clinicians reported that the perceived risk of ROKS affects the intensity of follow-up and surveillance offered to patients. Therefore, the discrepancies between risk estimates have implications for patient care. The barriers to adopting nomograms need further investigation, and prompt the need for future study of how to implement clinical support tools to integrate these tools into clinical workflow.

Abbreviations Used

AUA

American Urological Association

CDS

clinical decision support

EHRs

electronic health records

IQR

interquartile range

ROKS

recurrence of kidney stones

Authors' Contributions

Each author has made a substantial contribution to the article. All authors have read and approved the final article.

Disclaimer

The article or portions thereof are not under consideration by another journal or electronic publication and have not been previously published.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article. The project described was supported by CTSA award No. UL1TR000445 from the National Center for Advancing Translational Sciences. Its contents are solely the responsibility of the authors and do not necessarily represent official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.

Supplementary Material

Supplementary Appendix SA1

Supplementary Appendix SA2