Due to a poor evidence base arising from a dearth of high-quality comparative effectiveness studies, there is substantial uncertainty regarding optimal surgical treatments for patients across the spectrum of malignant and benign urologic disease. Additionally, most existing research lacks information on patients’ experiences after surgery. Without sufficient information on patient-prioritized outcomes after surgery, patients, caregivers, and surgeons lack the necessary evidence to make shared, informed, and individualized treatment decisions.
This uncertainty about optimal surgical treatments exists for patients of all ages; accordingly, this commentary applies to any urologic (or surgical) disease and procedure in any age group. However, this ambiguity is compounded by the relative rarity of pediatric disease and unique considerations for children, such as the need for ancillary procedures, risk of anesthesia, and a greater event horizon for long-term post-procedural sequelae. For children, these evidence gaps lead to extrapolations of adult treatment guidelines, adaptations which insufficiently address the unique considerations of children and their caregivers.
Despite the mandate for high-level evidence in urologic disease, recent critiques have recognized the limited role of procedural-based randomized controlled trials (RCTs).1 In particular, the highly-controlled environment of an RCT has limited “ability to assess the individualized effect of treatment, as can result from differences in surgical techniques.”2 Furthermore, the relative rarity of pathologies, as referenced above, presents additional challenges to achieving necessary sample sizes to detect clinically meaningful differences. Additional concerns about RCTs include:
Lack of clinical equipoise. Clinical equipoise implies uncertainty regarding the comparative effectiveness of interventions and serves as the ethical and practical basis for RCTs. Equipoise is challenged in trials of surgical interventions, where treatment choice is heavily influenced by surgeons’ experience, training, and local practices. Consequently, surgeons (and patients) may be unwilling to have surgical decisions determined by chance alone, leading to poor enrollment, the most common cause of failed surgical trials in Urology.3
Procedural learning curve. As opposed to static pharmaceutical formulations, surgical procedures evolve, sometimes rapidly, due to surgeon acquisition of experience and proficiency. Consequently, evaluating a nascent procedure may detect a greater frequency of complications or fail to detect differences in outcomes due to insufficient surgeon experience.4 To mitigate risk of complications and probability of Type 2 errors, typically only highly experienced surgeons and centers participate in surgical RCTs.4 Accordingly, results are difficult to generalize to real-world clinical care.
Changing technology. Surgery leads rapid technological innovation and adoption; such modifications may not be allowable within RCTs. Therefore, surgical RCTs may produce results that are outdated by time the study is published.
Consequently, the greatest obstacle to improving surgical outcomes is the lack of systems able to identify effective treatments that improve patient-centered outcomes and incorporate new knowledge into clinical practice across a diverse healthcare landscape. In contrast to rigid RCT protocols, such systems must respond to evolving practice patterns and technology while maintaining scientific rigor.
To this end, we propose a conceptual framework for comparative effectiveness studies of urologic procedural interventions – the surgical improvement cycle – realized through learning health systems (Figure). Learning health systems are designed to pragmatically collect and analyze real-world data, which subsequently returns to end-users (e.g. urologists) and directly benefits patients.5 A model of continuous knowledge generation is well-suited for urologic surgery, where discrete outcomes often occur shortly after interventions. The periodicity of the surgical improvement cycle varies based on patient volume, time to outcome ascertainment, and the logistics of analyzing data and returning reports to end-users. Surgical learning health systems may be situated within individual hospitals or structured across a broader collaborative.
Figure.
The organizational structure of PKIDS provides a conceptual framework for a surgical improvement cycle, which is centered around the patient. The process starts at surgical intervention with real-time measurement of outcomes and subsequent analysis of technical variation and outcomes that are ultimately returned back to the surgeon for continuous improvement.
We present, as an example, the Pediatric KIDney Stone (PKIDS) Care Improvement Network: a community of patients, caregivers, and clinicians at 26 pediatric healthcare systems throughout North America that conducts studies to improve the lives of youth with kidney stones. PKIDS is conducting a PCORI-funded prospective cohort study comparing stone clearance and the lived experience after ureteroscopy, shockwave lithotripsy, and percutaneous nephrolithotomy (NCT 04285658). Embedded within the clinical care of more than 100 surgeons and designed to leverage the natural variation of their treatment preferences, this study will generate real-world evidence to improve pre-, intra-, and post-operative decisions surrounding the surgical management of nephrolithiasis.
We propose the following design elements as essential to surgical observational studies to reduce bias and confounding:
Prospective and Integrated Data Collection: Prospective collection of data that could influence the choice of surgical intervention and outcomes is the essential to reduce unmeasured confounding and selection bias. Recording these data at the point of care, ideally integrated with the electronic health record, decreases missingness and recall bias while reducing the burden on clinician and research team members. Using kidney stones as an example, elements that impact the type of intervention and outcomes such as stone clearance and the lived patient experience after surgery include: (A) Patient characteristics (e.g. stone size/location, prior experience with surgery, pre-operative pain interference and anxiety). (B) Surgeon characteristics (e.g. treatment preferences, procedural experience).6 (C) Hospital characteristics (e.g. equipment availability, surgical volume). (D) Healthcare access (e.g. payers, geography).
Iterative Process Evaluation: Tracking of intra-operative processes ensures that quality of care remains constant across the collaborative (e.g. post-operative imaging, use of surgical antibiotic prophylaxis). In addition, prospective recording of key technical aspects for each procedure facilitates continuous process improvement, generating knowledge about the outcomes resulting from how a procedure is performed.
Patient-Centered Design: Health care systems must, at their core, value the patient. To that end, PKIDS partners with patients living with kidney stones and their caregivers as members of the research team. These patient partners inform research questions addressed, study design, implementation, and results dissemination to ensure network alignment with a mission towards improving care that will be meaningful to patients.
Ultimately, surgical learning health systems are better poised to address the research and clinical needs for procedural-based care than RCTs. By establishing a collaborative within a diverse and engaged network, clinical equipoise is generated while reducing bias and confounding by prospectively measuring natural surgeon, patient, and disease variation. Pragmatic study designs and adaptable approaches to prospective data collection minimize barriers to patient enrollment and surgeon participation, and enable longitudinal evaluation of surgical techniques and technology in evolution. Embedding data capture within clinical documentation allows for near real-time feedback to surgeons on many domains (e.g. guideline adherence; individual and aggregate outcomes) and comparative assessment across the network. Procedural learning curves are no longer a limitation, but rather an element of care that can be improved through network participation. Knowledge synthesis and propagation is thus realized within healthcare systems treating thousands of patients, thereby shortening the time from study launch to patient impact to an order of months rather than years.7 Surgical learning health systems require invested individuals willing to improve themselves in a collaborative rather than competitive fashion and a sustainable infrastructure for knowledge generation and application. These challenges necessitate a sea-change in surgical culture and re-assessment of traditional funding mechanisms, as sustainable learning health systems must demonstrate fluidity and responsiveness to external pressures. However, the return on investment to participating urologists and, more importantly, their patients, are enormous.
Acknowledgement
Gregory Tasian and Jonathan Ellison are the Director and Associate Director of the the Pediatric KIDney Stone (PKIDS) Care Improvement Network, respectively. Research reported in this article was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (CER-2018C3-14778).
The views, statements, opinions in this editorial are solely the responsibility of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee.”
The following PKIDS investigators contributed to the design of PKIDS, drafted and/or revised the manuscript, and approved the final manuscript. These individuals are members of the PKIDS Data Coordinating Center, or principal investigators or co-investigators at the 26 sites in PKIDS: Christopher B. Forrest, MD, PhD, Arun K. Srinivasan, MBBS, MRCS, Michelle Denburg, MD, MSCE, Susan J. Back, MD, Amy Kratchman, Jing Huang, PhD, Matthew Lorenzo, Christina B. Ching, MD, William Robert DeFoor, Jr., MD, MPH, Ahmad H. Bani-Hani, MD, Pamela I. Ellsworth, MD, Andrew A. Stec, MD, Thomas S. Lendvay, MD, Nicolás Fernández, MD, PhD, Kyle Rove, MD, Caleb P. Nelson, MD, MPH, Michael P. Kurtz, MD, MPH, Douglas E. Coplen, MD, Christopher E. Bayne, MD, Pankaj P. Dangle, MD, Aaron Krill, MD, Linda A. Baker, MD, Nicolette K. Janzen, MD, Renea Sturm, MD, Abby S. Taylor, MD, Wolfgang H. Cerwinka, MD, Eric D. Nelson, MD, Justin B. Ziemba, MD, David I. Chu, MD, MSCE, Pasquale Casale, MD, MHA, Rosalia Misseri, MD, Kate H. Kraft, MD, Bhalaajee Meenakshi-Sundaram, MD, and Armando Lorenzo, MD.
References
- 1.Wallis CJD, Detsky AS, Fan E: Establishing the Effectiveness of Procedural Interventions: The Limited Role of Randomized Trials. JAMA, 2018 [DOI] [PubMed] [Google Scholar]
- 2.Frieden TR: Evidence for Health Decision Making - Beyond Randomized, Controlled Trials. N Engl J Med, 377: 465, 2017 [DOI] [PubMed] [Google Scholar]
- 3.Bandari J, Theisen KM, Maganty A. et al. : Clinical Trials in Urology: Predictors of Successes and Failures. J Urol, 204: 805, 2020 [DOI] [PubMed] [Google Scholar]
- 4.Combes A, Hajage D, Capellier G. et al. : Extracorporeal Membrane Oxygenation for Severe Acute Respiratory Distress Syndrome. N Engl J Med, 378: 1965, 2018 [DOI] [PubMed] [Google Scholar]
- 5.Krapohl GL, Hemmila MR, Hendren S. et al. : Building, scaling, and sustaining a learning health system for surgical quality improvement: A toolkit. Learn Health Syst, 4: e10215, 2020 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Smith J: Role of surgeon volume in radical prostatectomy outcomes - Hu JC, Gold KF, Pashos CL, Mehta SS, Litwin MS, Departments of Urology and Health Services, University of California, Los Angeles, David Geffen School of Medicine and School of Public Health, Los Angeles, CA. J Clin Oncol 2003:21:401–405. URO, 21: 480, 2003 [DOI] [PubMed] [Google Scholar]
- 7.Medicine I. o.: Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press, p. 436, 2013 [PubMed] [Google Scholar]