Abstract
Health systems today have increasing opportunities and imperatives to conduct delivery science, which is applied research that evaluates clinical or organizational practices that systems can implement or encourage. Examples include research on eliminating racial/ethnic disparities in hypertension management and on identifying the types of patients who can successfully use video visits. Clinical leaders and researchers often face barriers to delivery science, including limited funding, insufficient leadership support, lack of engagement between operational and research leaders, limited pools of research expertise, and lack of pathways to identify and develop ideas. We describe five key strategies we employed to address these barriers and develop a portfolio of delivery science programs in Kaiser Permanente Northern California. This portfolio now includes small and medium-sized grant programs, training programs for postdoctoral research fellows and experienced physician researchers, and a dedicated team that partners with clinicians to develop high-priority ideas and conduct small projects. Most of our approaches are consistent with frameworks used to develop delivery science by other health systems; some are innovative. Most of these strategies are adaptable by other health systems prepared to make long-range organizational commitments to mechanisms that foster partnerships between clinical leaders and researchers.
KEY WORDS: implementation research, health care delivery, program evaluation, medical informatics, physician decision support
BACKGROUND
Recent developments have accelerated the impetus for health care organizations to evolve as learning systems.1 Nationwide adoption of electronic health records and shifts in financing and management have increased the opportunity and imperative to conduct delivery science, which we define as applied research that evaluates clinical or organizational practices that health systems can implement or encourage.1–3 This field includes effectiveness research and implementation research, which encompasses evaluation of quality improvement interventions.4–6 Examples of delivery science questions include:
How can we eliminate racial/ethnic disparities in hypertension management?
What types of patients can successfully use video visits to substitute for in-person office visits?
Clinicians and researchers nationwide face barriers to conducting delivery science, including limited funding, insufficient leadership support, lack of engagement between research and operational leaders, limited access to research expertise by clinicians, and confusion about how to prioritize opportunities.7, 8 We have first-hand experience with these barriers as leaders within Kaiser Permanente Northern California (KPNC), an integrated system with more than four million members. Despite having a robust research group, we saw longstanding frustration from both clinical leaders and researchers that we had not optimized the potential of research in our system.
Like others who have established delivery science portfolios, we recognized that increasing partnerships between clinical leaders and researchers would increase the quality of our performance improvement work and enable research to have higher impact.9–12 To address this need, we developed new programs to support delivery science. For example, to address the questions above, our projects:
Created an algorithm on thiazide dosing that led to a region-wide quality metric for reducing blood pressure disparities
Generated findings to inform the operational implementation of video visits13
This paper is for clinical leaders and researchers who wish to foster delivery science. We describe five key strategies: (1) committing organizational funding, (2) engaging leadership, (3) integrating research with clinical operations, (4) creating reservoirs of expertise, and (5) devising structured pathways for multidisciplinary idea development. We also examine the generalizability and limitations of our experience, comparing it with models in other health systems.5, 9, 10
COMMITTING ORGANIZATIONAL FUNDING
Research groups in health systems often depend on external funding. In 2010 before our efforts, 88% of our research group’s $96 million in annual expenditures were from external sponsors, predominantly the National Institutes of Health. External sponsors’ priorities often did not facilitate the research our organization needed. In 2012, three conditions converged to catalyze new internal investment in delivery science: KPNC’s finances were robust; a new research group director (T.L.) was recruited; and our executives were forward-thinking, viewing delivery science as a means of improving care and enhancing reputation.
To ensure support for delivery science, our executives have committed sustained funding for five interrelated programs that comprise our delivery science portfolio (Table 1).14 We initially established two funding programs to stimulate collaboration between embedded researchers and clinicians: the Rapid Analysis Program for small projects and the Delivery Science Research Program for medium-sized projects. Subsequently, we developed the Rapid Analytics Unit, a dedicated team that enables executives to commission high-priority small projects. We have also established training programs for postdoctoral fellows and experienced physician researchers. These long-term commitments have been critical in developing our delivery science pipeline.
Table 1.
Programs in the Delivery Science Portfolio, Kaiser Permanente Northern California
| Program | Function or type of work supported | Specifications | Number of projects or persons supported |
|---|---|---|---|
| Rapid Analysis Program | Grants for small projects and feasibility analyses | Up to 6 months and up to $30,000 using only existing data | 8 projects completed; 2 in progress |
| Delivery Science Research Program | Grants for medium-sized projects | Up to 24 months and up to $250,000 | 11 projects completed; 28 in progress |
| Rapid Analytics Unit | Dedicated team that does work commissioned by executives, including consultation, project design, analytics, and small projects | Up to 12 months, using only existing data | 18 projects completed; 9 in progress |
| Delivery Science Fellowship Program | Training for postdoctoral research fellows | 2 years of full-time support for fellows, plus tuition for a masters in public health if appropriate | 6 fellows at any given time; 8 have graduated as of summer 2018 |
| Physician Researcher Program | Training for physicians with 2 or more years of attending-level experience | 4 years of protected effort for research at 20–40% FTE for physicians, plus $250,000 in research project support | 6 physicians in the first cohort |
ENGAGING LEADERSHIP
Our efforts have been supported and guided by the executives of The Permanente Medical Group, who lead 9000 physicians in 21 hospital-based medical centers and more than 250 outpatient clinics. Through a series of one-on-one and small group meetings, we were able to educate leaders about the needs for funding, development of internal expertise, and time and persistent effort required to enable research to contribute to operational improvement.
Guidance from executives has been critical to enable prioritization, the key to effectively investing the finite funding and researcher bandwidth available. To elicit executive input, we have used both ad hoc and systematic approaches, including emails to test ideas and small group meetings to refine them. We hold quarterly large group meetings of executives and research leaders to review ideas and set priorities. Criteria for a project to be designated high-priority are that it will (1) have operational impact in a strategic priority area, (2) build relationships among operational leaders and researchers, (3) develop a method that generalizes to other clinical situations, and (4) enhance our national reputation through publication or dissemination.
INTEGRATING RESEARCH WITH CLINICAL OPERATIONS
Forming partnerships among researchers, clinicians, and health system leaders is crucial, since few individuals have the full range of skills needed for delivery science.7, 11, 12, 15 To stimulate collaboration, we have required that each project be co-led by a clinician and researcher. For example, an internist leading region-wide quality efforts and a research scientist wanted to evaluate antihypertensive medication treatment among African-Americans. Our Rapid Analysis Program supported their collaborative project, which led to implementation of a new quality measure to reduce disparities in thiazide dosing.16
In some cases, our programs have catalyzed new clinician-researcher partnerships. For example, a pulmonologist and a physician technology leader wanted to evaluate a new protocol for triaging and tracking pulmonary nodules identified on computed tomography for lung cancer screening. They sought collaboration from a lung cancer epidemiologist who helped design a project, supported by our Delivery Science Research Program, that led to better standardization of care.
We found that these funding programs could not meet all delivery science needs. Many clinicians need experienced researchers to help them develop feasible project plans.17 To provide a mechanism more agile than a grant program, we developed the Rapid Analytics Unit, a dedicated team that conducts small projects commissioned by executives. The unit includes three senior researchers from our embedded research group, a project manager, and several programmer/analysts.
This unit’s work has contributed to effective implementation in many specialties. For example, our oncology chiefs developed systems to enable regional experts to offer consultation to local oncologists treating testicular and other uncommon cancers. The group needed algorithms to rapidly identify newly diagnosed patients, so regional experts could contact local oncologists. The Rapid Analytics Unit, with our cancer registry experts, created case-finding algorithms using discrete data elements and natural language processing.
CREATING RESERVOIRS OF EXPERTISE
To expand the talent pool for delivery science, we have established two training programs. The Delivery Science Research Fellowship Program includes six postdoctoral fellows who receive 2 years of training in research and quality improvement. Each fellow completes two or more projects and participates in work-in-progress and skills seminars. Fellows’ backgrounds include medicine, nursing, epidemiology, and genetics. Their research topics include reducing care gaps for cancer survivors, mental health in complex patients, and implementation of genetic testing.
The Physician Researcher Program leverages the talents of experienced physicians who are capable of leading both research and implementation. It provides structured training and research project support. The six physicians in the program’s first cohort are devoting 20 to 40% of their effort to research for at least 4 years, with the rest in clinical practice. Projects include a study of outcomes of asymptomatic carotid stenosis and development of clinical decision support for emergency management of atrial fibrillation.
DEVISING PATHWAYS FOR IDEA DEVELOPMENT
Good delivery science ideas emanate from a variety of people, including clinicians, researchers, and operational leaders. Executives and clinical chiefs are well-positioned to identify strategic priorities and usually need others to develop and execute projects.
We have devised structured pathways to stimulate multidisciplinary idea development using both bottom-up and top-down approaches. In the bottom-up approach, we issue a region-wide request for applications to the Delivery Science Research Program twice a year. We ask executives, physicians-in-chief of medical centers, chairs of region-wide specialty groups, and other leaders to disseminate the request. Letters of intent are discussed in an executive meeting, after which only ideas likely to be funded are invited to develop full proposals.
The second pathway uses a top-down approach in which executives first identify clinical areas that are ready for delivery science development. Criteria for readiness include having a chief group with a strategic plan for enhancing care delivery and having one or more clinicians with ability to lead a project. In a typical development process, the regional chair of a specialty invites ideas from clinical leaders, designates a champion to work with a senior researcher to develop a set of feasible study ideas, and works with their executive and others to select priorities. This process has the advantage of systematically involving many key stakeholders from its inception.
MEASURES OF SUCCESS
Our delivery science portfolio’s primary objective is to generate evidence that improves care delivery and resulting health outcomes. Each project commits to creating deliverables relevant to implementation and is assessed at its end and 12 months later. Because many projects aim toward improving processes, with more distal effects on clinical outcomes, we have used process measures to evaluate success (Table 2). These include whether a project (1) improved a region-wide delivery practice, (2) generated information that indicated the next step in care improvement, (3) created a useful analytic approach or tool, (4) identified barriers to effective care delivery, (5) identified barriers to generating evidence, such as data gaps, (6) built relationships between key clinicians and researchers, or (7) enhanced organizational reputation through publication.
Table 2.
Examples of Projects and Their Impact, Delivery Science Portfolio, Kaiser Permanente Northern California
| Program/project title | Work accomplished | Measures of success1; specifics |
|---|---|---|
| Rapid Analysis Program | ||
| Electronic secure message reminders in patients with diabetes | Found that patients opened electronic secure messages at higher rates than expected, and the volume of messages sent was relatively low | Generated information that indicated the next step in care improvement: the quality department is designing approaches to encourage more use of such messages |
| Gaps in follow-up surveillance for survivors of early-stage colorectal cancer | Found that the proportion of eligible patients with follow-up testing varied substantially across medical centers | Generated information that indicated the next step in care improvement: the regional team is pilot testing ways to close the observed gaps |
| Delivery Science Research Program | ||
| Real-time decision support for pulmonary embolism in the ED | Evaluated data capture methods and incorporated KP-specific mortality point estimates in a risk stratification decision support tool | Improved a region-wide delivery practice: enhanced a risk stratification tool now in use in EDs region-wide; results published in Ann Intern Med[18] |
| Robotic hysterectomy for benign gynecologic disease—pilot study | Developed outcome measures and described variation in case complexity | Generated information that indicated the next step in care improvement: robotic and conventional laparoscopic procedure outcomes studied in >7000 patients; report shared with regional leaders to help shape referral protocols |
| Rapid Analytics Unit | ||
| Familial hypercholesterolemia—identifying patients at risk | Created algorithms to identify cases of potential familial hypercholesterolemia based on laboratory results for referral | Generated information that indicated the next step in care improvement: reports from the algorithm are now being used to identify patients for evaluation |
| Adnexal masses identified on ultrasound | Conducted analyses of utilization and outcomes of women with adnexal masses managed via a new clinical protocol for risk classification based on ultrasound | Improved a region-wide delivery practice: supported the implementation of the new clinical protocol and led to a subsequent evaluation phase; results published in Obstet Gynecol[19] |
1Measures of success included (1) improved a region-wide delivery practice, (2) generated information that indicated the next step in care improvement, (3) created a useful analytic approach or tool, (4) identified barriers to effective care delivery, (5) identified barriers to generating evidence, such as data gaps, (6) built relationships between key clinicians and researchers, or (7) enhanced organizational reputation through publication or dissemination
Some projects have yielded financial return on investment, for example, by averting needless ultrasounds for low-risk ovarian masses19 or finding that African-American patients with hepatitis C do not require extended treatment courses,20 leading to estimated savings of several million dollars annually. Still, our portfolio’s central objective is effective and efficient care delivery, rather than cost savings.
More broadly, our efforts have shifted our organization’s culture toward increasing collaboration between operations and research, a vital condition for learning health systems.4, 6, 10, 21 Operational leaders have become more conversant with important methodologic issues such as regression to the mean, while researchers have a deeper understanding of operational needs. Researchers have been increasingly able to influence implementation efforts to adopt high-quality evaluation designs.
The evolution of our delivery science portfolio has uncovered new challenges. The volume of ideas has increased, necessitating better structures for developing and prioritizing ideas. We found that some projects encountered recurring types of barriers, leading us to devise ways of proactively identifying such barriers before settling project plans.
COMPARISONS AND LIMITATIONS
Our work aligns with published frameworks for delivery science, especially the operating models described by King.5, 9–11 Many projects supported by our Rapid Analysis Program and Delivery Science Research Program fall within King’s clinician and researcher program evaluation model. Our Physician Researcher Program parallels King’s clinician-researcher skill development model, but goes beyond other efforts by dedicating organizational funding for mid-career physicians to conduct both research and implementation. Our Rapid Analytics Unit is relatively innovative in providing a utility that can quickly respond to executives’ requests to develop high-priority delivery science ideas and conduct studies.
We acknowledge that not every health system has the financial means and leadership support that catalyzed our efforts. Other factors underlying our success include Kaiser Permanente’s integration, cohesive culture, embedded research group, and information systems, which facilitate interchange between research and operations. Integrated systems, including the Veterans Administration,5, 11 Intermountain Healthcare,22 other Kaiser Permanente regions,10, 23 and the UK,24 may offer especially supportive environments for delivery science due to incentives being aligned between payers and providers. These systems have used similar strategies, including commitment of internal funding, mechanisms to stimulate partnerships, and idea development structures.5, 10, 11
Other types of health systems, ranging from academic medical centers to a children’s rehabilitation center, have succeeded in developing delivery science programs.9, 25, 26 For smaller or less integrated settings, the strategies we employed may require changes in scale or structure. For example, establishing a funding program to stimulate clinician-researcher collaboration can be initiated with modest funding. Other strategies, such as creating training programs, may require funding beyond the means of smaller organizations but could be attained through collaboration with external partners.
We conclude that delivery science can be fostered through strategies that address key barriers and stimulate clinician-researcher partnerships. Continuing progress in this field will stem from organizations evaluating the factors leading to success and failure and sharing knowledge to facilitate our collective understanding of how to best develop learning health systems.
Acknowledgments
We are grateful to the executive leaders and physicians of The Permanente Medical Group (TPMG), including Robert Pearl, MD (Executive Director and CEO, 1999-2017), Richard Isaacs, MD (current Executive Director and CEO), and Yi-Fen Irene Chen, MD (current Associate Executive Director for quality and research) for their strong support and guidance of our delivery science portfolio. We acknowledge Richard Grant, MD, MPH, Julie Schmittdiel, PhD, Douglas Corley, MD, PhD (current Director of Delivery Science and Applied Research), Andy Avins, MD, MPH, and Lisa Herrinton, PhD, for their capable leadership of the programs described here. We thank Steven Schroeder, MD, Grace Lee, MD, MPH, Hal Sox, MD, and the anonymous reviewers for insightful suggestions on shaping this paper. This portfolio’s development has relied on the skillful contributions of our research scientists, project managers, and programmer/analysts, especially Beth Dameron, MS, Suzanne Furuya, MPH, MBA, and Stephanie Prausnitz, MPH.
Funding Information
This work was funded by TPMG, Kaiser Permanente Northern California.
Compliance with Ethical Standards
Conflict of Interest
The authors are employees of TPMG and have no other relationships or conflicts of interest to declare.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Whicher D. Accelerating Progress Toward Continuous Learning: A National Academy of Medicine Meeting Summary. Washington, DC: National Academy of Medicine; 2017. https://www.ahrq.gov/sites/default/files/wysiwyg/professionals/systems/learning-health-systems/Learning-Health-Systems-Meeting-Summary.pdf. Accessed 5 Jan 2019.
- 2.Jena AB, Stevens W, McWilliams JM. Turning evidence into practice under payment reform: the new frontier of translational science. J Gen Intern Med. 2014;29(11):1542–1545. doi: 10.1007/s11606-014-2979-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Lieu TA, Platt R. Applied Research and Development in Health Care - Time for a Frameshift. N Engl J Med. 2017;376(8):710–713. doi: 10.1056/NEJMp1611611. [DOI] [PubMed] [Google Scholar]
- 4.Rubenstein LV, Pugh J. Strategies for promoting organizational and practice change by advancing implementation research. J Gen Intern Med. 2006;21(Suppl 2):S58–64. doi: 10.1007/s11606-006-0276-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Atkins D, Kilbourne AM, Shulkin D. Moving From Discovery to System-Wide Change: The Role of Research in a Learning Health Care System: Experience from Three Decades of Health Systems Research in the Veterans Health Administration. Annu Rev Public Health. 2017;38:467–487. doi: 10.1146/annurev-publhealth-031816-044255. [DOI] [PubMed] [Google Scholar]
- 6.Fisher ES, Shortell SM, Savitz LA. Implementation Science: A Potential Catalyst for Delivery System Reform. JAMA. 2016;315(4):339–340. doi: 10.1001/jama.2015.17949. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bindman AB, Pronovost PJ, Asch DA. Funding Innovation in a Learning Health Care System. JAMA. 2018;319(2):119–120. doi: 10.1001/jama.2017.18205. [DOI] [PubMed] [Google Scholar]
- 8.Kilbourne AM, Atkins D. Partner or perish: VA health services and the emerging bi-directional paradigm. J Gen Intern Med. 2014;29(Suppl 4):817–819. doi: 10.1007/s11606-014-3050-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.King G, Currie M, Smith L, Servais M, McDougall J. A framework of operating models for interdisciplinary research programs in clinical service organizations. Eval Program Plann. 2008;31(2):160–173. doi: 10.1016/j.evalprogplan.2008.01.003. [DOI] [PubMed] [Google Scholar]
- 10.Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207–210. doi: 10.7326/0003-4819-157-3-201208070-00012. [DOI] [PubMed] [Google Scholar]
- 11.Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating Research Impact in a Learning Health Care System: VA’s Quality Enhancement Research Initiative in the Choice Act Era. Med Care. 2017;55(7 Suppl 1):S4–S12. [DOI] [PMC free article] [PubMed]
- 12.Selby JV, Slutsky JR. Practicing partnered research. J Gen Intern Med. 2014;29(Suppl 4):814–816. doi: 10.1007/s11606-014-3046-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Reed ME, Parikh R, Huang J, Ballard DW, Barr I, Wargon C. Real-Time Patient-Provider Video Telemedicine Integrated with Clinical Care. N Engl J Med. 2018;379(15):1478–1479. [DOI] [PubMed]
- 14.Division of Research, Kaiser Permanente Northern California. Delivery Science Portfolio. https://divisionofresearch.kaiserpermanente.org/deliveryscience/deliveryscience/. Accessed 8 Sept 2018.
- 15.Wu RR, Kinsinger LS, Provenzale D, et al. Implementation of new clinical programs in the VHA healthcare system: the importance of early collaboration between clinical leadership and research. J Gen Intern Med. 2014;29(Suppl 4):825–830. doi: 10.1007/s11606-014-3026-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Schmittdiel JA, Dlott R, Young JD, Rothmann MB, Dyer W, Adams AS. The Delivery Science Rapid Analysis Program: A Research and Operational Partnership at Kaiser Permanente Northern California. Learn Health Syst. 2017;1(4). 10.1002/lrh2.10035. [DOI] [PMC free article] [PubMed]
- 17.Schmittdiel JA, Grant RW. Crossing the Research to Quality Chasm: A Checklist for Researchers and Clinical Leadership Partners. J Gen Intern Med. 2018;33(1):9–10. doi: 10.1007/s11606-017-4189-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Vinson DR, Mark DG, Chettipally UK, et al. Increasing Safe Outpatient Management of Emergency Department Patients With Pulmonary Embolism. Ann Intern Med 2018;169(12):855–865. 10.7326/M18-1206. [DOI] [PubMed]
- 19.Suh-Burgmann EJ, Flanagan T, Osinski T, Alavi M, Herrinton L. Prospective Validation of a Standardized Ultrasonography-Based Ovarian Cancer Risk Assessment System. Obstet Gynecol. 2018; 10.1097/AOG.0000000000002939. [DOI] [PubMed]
- 20.Marcus JL, Hurley LB, Chamberland S, et al. No Difference in Effectiveness of 8 vs 12 Weeks of Ledipasvir and Sofosbuvir for Treatment of Hepatitis C in Black Patients. Clin Gastroenterol Hepatol. 2018;16(6):927–935. doi: 10.1016/j.cgh.2018.03.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Hochman M, Asch SM. Disruptive Models in Primary Care: Caring for High-Needs, High-Cost Populations. J Gen Intern Med. 2017;32(4):392–397. doi: 10.1007/s11606-016-3945-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.James BC, Poulsen GP. The Case for Capitation. Harv Bus Rev. 2016;94(7-8):102–111, 134. [PubMed]
- 23.Alem AC, Gould MK. 2017 Annual Report of the Care Improvement Research Team. Pasadena, CA: Kaiser Permanente Southern California Department of Research and Evaluation: 2018. https://www.kp-scalresearch.org/documents/CIRT_2017_Annual_Report.pdf.
- 24.National Institute for Health Research. NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs). https://www.nihr.ac.uk/about-us/how-we-are-managed/our-structure/infrastructure/collaborations-for-leadership-in-applied-health-research-and-care.htm. Accessed 8 Sept 2018. [DOI] [PMC free article] [PubMed]
- 25.Penn Medicine. Center for Health Care Innovation. https://healthcareinnovation.upenn.edu/. Accessed 8 Sept 2018.
- 26.Gonzales R, Moriates C, Lau C, et al. Caring Wisely: A Program to Support Frontline Clinicians and Staff in Improving Healthcare Delivery and Reducing Costs. J Hosp Med. 2017;12(8):662–667. doi: 10.12788/jhm.2786. [DOI] [PubMed] [Google Scholar]