Abstract
Background
An ambulatory safety net (ASN) is an innovative organizational intervention for addressing patient safety related to missed and delayed diagnoses of abnormal test results. ASNs consist of a set of tools, reports and registries, and associated workflows to create a high-reliability system for abnormal test result management.
Methods
Two ASNs implemented at an academic medical center are described, one focusing on colon cancer and the other on lung cancer. Data from electronic registries and chart reviews were used to evaluate the effectiveness of the ASNs, which were defined as follows: colon cancer—the proportion of patients who were scheduled for or completed a colonoscopy following safety net team outreach to the patient; lung cancer—the proportion of patients for whom the safety net was able to identify and implement appropriate follow-up, as defined by scheduled or completed chest CT.
Results
The effectiveness of the colon cancer ASN was 44.0%, and the effectiveness of the lung cancer ASN was 56.9%. The ASNs led to the development of registries to address patient safety, fostered collaboration among interdisciplinary teams of clinicians and administrative staff, and created new workflows for patient outreach and tracking.
Conclusion
Two ASNs were successfully implemented at an academic medical center to address missed and delayed recognition of abnormal test results related to colon cancer and lung cancer. The ASNs are providing a framework for development of additional safety nets in the organization.
Introduction
Over the last decade, there has been a growing interest in patient safety issues in the ambulatory setting. This interest can be traced in part to several seminal articles that called for a focus on patient safety in the ambulatory setting in contrast to the then dominant focus on patient safety in the inpatient setting.1–3 A technical brief identified five patient safety issues in the ambulatory setting: medication safety, diagnosis, transitions among providers in ambulatory settings, referrals from one provider to another, and management of test results.4 Studies have focused on assessing the rates of these safety issues as well as interventions for addressing the issues. In one study, Singh and colleagues estimated the rate of outpatient diagnostic errors at 5.08% or about 12 million US adults every year.5 In another study, Giardina and colleagues employed root cause analysis to identify factors contributing to delayed recognition of abnormal test results.6 Some of these factors included a lack of coordination in follow-up planning for test results, inadequate tracking of abnormal test results, and the absence of a system to track patients in need of short term follow-up.
Other studies in ambulatory safety have focused on interventions to address safety issues. Schiff and colleagues describe the Massachusetts PROMISES project which focused on test results, referrals, and medication management. This project relied on multiple strategies to address safety issues including educational sessions, materials and a PDSA (Plan-Do-Study-Act) approach.7 The concept of a safety net as an intervention to address ambulatory safety was developed at Kaiser Permanente Southern California (KPSC).8 The KPSC program was the first in the country to use electronic health information and electronic clinical surveillance to address a range of care gaps including missed and delayed diagnoses. However, beyond these studies there have been few interventions to promote ambulatory patients safety, particularly around missed and delayed cancer diagnoses.
The overall objective of this paper is to describe two ASN interventions at a large tertiary academic medical center. The ASNs were implemented to prevent missed and delayed diagnoses and to build a system that closed the loop from the point of test-ordering to the completion of guideline-based recommended follow-up care. We developed two ASNs: the colon cancer safety net, focusing on patients at risk for colon cancer based on prior colonoscopy procedures; and the lung cancer safety net, focusing on incidental lung nodules detected on CT scans.
Methods
Study Setting and Ethics
We implemented the two ASNs at an AMC in Boston. The AMC, which consists of a 747-bed tertiary care hospital and 174 ambulatory practices, had 1,800,926 ambulatory visits in 2018 and employs more than 12,000 people—of whom approximately 3,000 are physicians. It has a fully integrated electronic health record (EHR) and computerized order entry system (Epic; Epic Systems Corporation, Verona, Wisconsin). The Partners HealthCare Institutional Review Board approved the project. CRICO, the malpractice risk insurer, funded the project to assist with developing a model ASN program with the long-term goal of obtaining AMC funding for sustainability.
Organization for ASN interventions
The organization for the ASN interventions varied by the type of safety net. For the colon cancer safety net, the organizational setting consisted of primary care, gastroenterology (GI), and colorectal surgery, while the setting for the lung cancer safety net consisted of primary care and radiology. In addition, an ASN project team provided project management support and facilitated the implementation of the safety nets. This project team included two central resources (a medical director and a project coordinator) and local resources (multidisciplinary teams of stakeholders, including frontline clinicians, physician leadership, administrative leadership, and nursing leadership).
ASN interventions
Working Groups
The interventions related to the two safety nets consisted of working groups, creation of patient reports, workflow redesign, and patient tracking and outreach (Table 1). A key component of both safety nets was the creation of small working groups. These working groups consisted of clinicians, managers, and staff from the various departments involved in the safety nets such as primary care, pulmonary, radiology, GI, and population health. The colon cancer safety net working group met on a monthly basis over the course of several years while the lung cancer safety net working group met for a six-month period followed by ad hoc monthly meetings. All meetings were coordinated and managed by the ASN project team.
Table 1.
Key components of Ambulatory Safety Nets (ASNs)
| Colon Cancer ASN | Lung Cancer ASN | |
|---|---|---|
| Patient identification | • Patients with colonoscopy and abnormal pathology now due to return for repeat colonoscopy | • Patients with incidental lung nodules noted on chest CT scans |
| Reports and registries | • Registry within EMR with functionality to send letters | • Application of natural language processing (NLP) tool to text of radiology reports and validation through chart review |
| Workflow redesign | • Gastroenterologists: continuously update the health maintenance modifier in the EMR; process of updating EMR occurs days after they actually perform colonoscopy once the pathology returns and is not synchronous with their process of sending letters to patients • ASN: runs the registry, conducts chart review, emails back and forth with gastroenterologists on cases where there is a discrepancy with the health maintenance modifier, creates a final list of at-risk patients, and communicates this list to endoscopy scheduling staff • Colonoscopy administrative team: runs the registry of at-risk patients, prints and mail letter to patients, sends patient information to outside PCP offices to obtain order for colonoscopy, and needs to coordinate scheduling of colonoscopy |
• PCPs: review lists of incidental lung nodule patients, send letter or call patients, order CT scan for follow-up, and communicate with ASN patient navigator • Radiology team: reviews any CT scan reports identified by NLP that do not have clear follow-up recommendations • ASN: conducts detailed chart reviews on all patients without follow-up, organizes overdue patient lists by PCP and practice, emails PCPs and practice leadership with overdue patients up to three times |
| Patient outreach and tracking | • ASN Patient Navigator calls patients overdue for colonoscopy • Separate database to track completion of follow-up |
• Separate database to track completion of follow-up |
EHR, electronic health record; NLP, natural language processing, PCP, primary care provider
Reports and Registries
Another important component of the interventions was the development of reports and registries linked to the EHR. For the colon cancer safety net, a registry was developed that captured patients who previously had a colonoscopy with abnormal pathology and were due for a repeat colonoscopy. This report relied on a coded field called “colonoscopy health maintenance,” which is the return interval EHR field that GI physicians were encouraged to update following a colonoscopy so that patient follow-up occurred in the correct time period. The registry did not use the direct result that was indicated in the pathology report but instead relied on the return interval placed by the GI physician or primary care provider (PCP) based on the result of the colonoscopy, the pathology report, and patient risk factors. The creation and validation of this registry took six months and required iteration and input from numerous clinical and administrative stakeholders across our health system.
For the lung cancer safety net, a natural language processing (NLP) tool was used to develop reports of patients with incidental lung nodules at the AMC for 2016 and 2017 who required further follow-up imaging or evaluation. We used manual chart review to validate the accuracy of the reports. In addition, we narrowed the list to include only patients with a primary care physician at the AMC and who were not already followed by our oncology teams or in our lung nodule clinic by thoracic surgery staff. This process took close to six months due to nuances in how incidental lung nodules were mentioned in radiology reports, the changing guidelines for management of incidental pulmonary nodules, and the multiple data and information technology teams required to narrow our target patient population.9
Patient Tracking and Outreach
After the reports and registries were developed, the final component of the interventions was patient tracking and outreach. For the colon cancer safety net, the ASN project team ran the report monthly for patients overdue for a colonoscopy. Chart review was then conducted to ensure that the correct patients received the outreach, particularly regarding the accuracy of the modifier. After chart review was performed, bulk letters were sent by the endoscopy staff to eligible patients. The message was sent electronically to patents with an activated patient portal account, whereas patients without the portal were sent hard-copy letters. In our initial outreach efforts from January 2018 through July 2018, two outreach attempts were conducted, primarily as electronic patient portal messages or hard-copy letters. In August 2018 a patient navigator was hired as part of the ASN project team, and we added phone calls as the third outreach attempt. Figure 1 shows the work flow for the colon cancer safety net. Figure 2 shows the efforts of the patient navigator after we added phone calls to our process.
Figure 1. Workflow for colon cancer safety net.
Figure 1 outlines the workflow for the safety net patient navigator: running the colon cancer safety net registry, conducting chart review, communicating with the gastroenterologist when needed, coordinating patient lists with the endoscopy team, calling patients overdue for colonoscopy, and tracking follow-up in the database.
Figure 2. Volume of colonoscopies scheduled and completed with colon cancer safety net =efforts.
Figure 2 demonstrates the impact of the colon cancer safety net efforts, beginning in January 2018 with sending patients letters, followed by adding a phone call in August 2018 and culminating in a total of 113 scheduled colonoscopies and 84 completed colonoscopies.
For the lung cancer safety net, the outreach was to the patients’ PCPs and practice leadership so that the providers could review the patient’s history, determine if a follow-up chest CT was indicated, and—if so—schedule the chest CT procedure. The individual lists were sent to the PCPs via secure e-mail, with a copy to the practice leadership. Up to three monthly reminders were sent requesting that the PCPs communicate back to the ASN project team with the outcome for those patients. Table 1 summarizes the key components of both the colon cancer and lung cancer ASNs.
Effectiveness of the interventions
The colon cancer safety net focused on patients who had a prior colonoscopy at the hospital and who were due for a repeat colonoscopy based on the health maintenance modifier set in the EHR. We defined effectiveness as the proportion of patients who were either scheduled for or who completed a colonoscopy following the safety net’s outreach to the patient. Of the 252 patients identified for the colon cancer safety net, 111 patients (44.0%) either scheduled or completed a colonoscopy.
The lung cancer safety net focused on patients who had a primary care provider at the academic medical center, did not have a provider at a cancer center affiliated with the academic medical center, and did not have a follow-up chest CT within six to twelve months of the initial CT scan. We defined effectiveness as the proportion of patients for whom the safety net was able to identify and implement appropriate follow-up, as defined by scheduled or completed chest CT. Of the 123 patients identified for the safety net, 70 (56.9%) patients received appropriate follow up as a result of the safety net’s outreach to the PCP.
Qualitative Data
We conducted structured interviews with five physicians and three administrative directors involved in the ASN interventions about the implementation process, success of the interventions, and sustainability of the interventions. Participants agreed that the ASNs addressed patient safety related to missed and delayed diagnoses through follow-up and redundancy. Participants used the terms “thoughtful” and “collaborative” to describe the overall adoption of the safety nets as well the adoption strategy which minimized disruption on clinicians. Participants overall agreed that the safety net fit the mission of the practice and department.
Discussion
In this paper we described two safety net interventions focusing on missed and delayed results related to colon and lung cancer. Our interventions consisted of multi-disciplinary teams, development of reports and registries, workgroup meetings to discuss intervention progress, and patient tracking and outreach.
The concept and development of our two safety nets borrowed heavily on the pioneering work done by the KPSC Outpatient Safety Net Program.8, 10 As in the KPSC program we chose not to disrupt normal workflow or utilize real time alerts and decision support that would be disruptive to the provider. Like the KPSC program, we found local leadership support and a just culture framework to be crucial to the successful implementation of the two safety nets.
Our safety nets differ from those at KPSC in several ways. First our safety nets were launched through a grant program and resources for implementation of the safety nets were covered through the grant. Thus, the safety nets only partially relied on institutional resources like the KPSC program. Second, unlike the KPSC program which relied on a top-down centralized approach, our safety nets were developed and implemented through a bottom-up approach. They were small in scope and we net worked locally with stakeholders from different departments for their implementation.
Third, unlike the KPSC program, our setting was not an integrated delivery system. In fact, the colon cancer safety net initially focused on patients who did not have a PCP at our organization, as these patients were believed to be at greatest risk of not receiving follow-up. Subsequently, we expanded our focus to include all patients. We are also following two additional subsets of patients through the safety net—patients with rectal bleeding and patients with iron deficiency anemia—as part of a pilot program with four of our primary care practices. This ability to pilot and test small changes prior to widespread scaling across our primary care practices has been essential in the development of our model of ASNs.
We believe that safety nets, such as the ones deployed in this study, provide a useful intervention to address ambulatory safety and that such interventions need to be explored in greater detail. In general, interventions addressing ambulatory safety related to missed and delayed diagnoses have been understudied in contrast to the inpatient setting.4,11,12 Recognizing this need The Joint Commission Perspectives recently called for comments on processes for timely reporting and follow-up on all test results and for all patients.13 Safety nets like the ones described in this study can provide an innovative approach by incorporating multiple, cross-cutting factors (including interdisciplinary teams), facilitating infrastructure development such as reporting tools and registries, and promoting a culture of safety.1,4,14–18
The successful implementation of the safety nets resulted in our organization recognizing their value and supporting them after the grant funding ended. Some of the duties of the ASN project team, such as management of reports and registries, were transitioned to operational teams at the AMC. Other components of the ASN are codeveloped and comanaged by the safety net team and local departments. For example, a new pilot program codeveloped by the safety net team and the radiology department identifies patients with an incidental lung nodule in real time and alerts the ordering clinician but then shifts the burden of scheduling the test and tracking results to the radiology team. Furthermore, the organization has provided funding for (1) maintenance of the colon cancer and lung cancer ASNs and (2) for development of new ASNs, through an additional ASN safety net navigator and an ASN program manager. Our new ASNs will focus on (1) reducing medication errors through a pharmacist-led medication safety program and (2) reducing missed and delayed diagnoses of cervical, breast, and prostate cancer. These resources help to ensure sustainability and scalability at the conclusion of grant funding through a framework for expansion of the safety nets as well as development of additional safety nets in the organization.
Limitations
This study was conducted in one AMC located in the northeast United States. The findings may not be generalizable to other regions or settings such as community centers, although a number of physicians at our organization do practice in the community setting. We did not have a control group as part of this project and did not gather any data on health outcomes in this pilot study. Our definition of success of the ASNs was a scheduled or completed procedure. For example, when we performed chart review on the reasons patients in the colon cancer safety net did not have a scheduled or completed colonoscopy, we found that there were patients who had moved away or no longer received care within our system. In our lung cancer safety net, we found that our NLP tool was quite sensitive but not as specific in that it often identified nodular opacities or very small nodules that did not necessarily warrant any further follow-up. We were also unable to incorporate patient perspectives into the development and implementation of our safety nets. Finally, it is possible that patients could have scheduled their follow-up care at their next office visit with their PCP or specialist.
Lessons Learned
We identified the following key lessons with respect to safety nets as a strategy for addressing ambulatory patient safety:
Safety net concept: The concept of a safety net as described here, consisting of tools such as reports and registries and associated work flows, can be implemented in a variety of settings, including those without a dedicated ambulatory patient safety program or infrastructure in place.
Scaling up: Our ability to pilot safety nets and test small changes in a few ambulatory practices prior to widespread scaling was essential in the development of our model of ASNs.
Multidisciplinary stakeholder buy-in and workflow redesign: The development and implementation of safety nets will take time (our safety nets took almost two years to reach a maintenance stage), and this may be particularly true when the development is not centrally driven but is, rather, a local, bottom-up approach like the one we adopted.
Information technology registries and tools: The development of reports and registries for electronic surveillance will be a key contributing factor to the implementation of safety nets. Although this may seem intuitive, it may not be simple even in the presence of an advanced EHR such as the one used at our organization. The tools for electronic surveillance needed to be combined with time-consuming and labor-intensive chart reviews and a natural language processing tool to obtain data.
Leadership buy-in and support for ambulatory patient safety: Leadership backing will be critical to the implementation and sustainability of safety nets. Raising awareness of ambulatory patient safety vulnerabilities, such as missed and delayed cancer diagnoses, to hospital leadership is critical
Conclusions
We successfully implemented a colon cancer ASN and a lung cancer ASN at an academic medical center to address missed and delayed diagnoses. The effectiveness of the colon cancer safety net was 44.0%, and the effectiveness of the lung cancer safety net was 56.9%. We are currently creating a more automated incidental lung nodule ASN, and we are piloting rectal bleeding and iron-deficiency anemia as part of our colon cancer ASN,19 In addition, we are building several new ASNs on medication safety, cervical cancer, breast cancer, and prostate cancer. Future research should focus on developing more reliable ASN systems to address the myriad of abnormal test results that are generated as part of routine patient care and also on designing proactive electronic surveillance systems to identify, mitigate, and reduce safety risks related to diagnostic error.
Acknowledgment
The authors thank Stacy Wiegand for her role as patient navigator conducting patient outreach and tracking and helping with data collection.
Funding
This study was funded by the CRICO Risk Management Foundation through fund # 115553. This study is partly funded by Grant R01HS024722 from the Agency for Healthcare Research and Quality (AHRQ).
Contributor Information
Srinivas Emani, Division of General Internal Medicine, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, Laboratory of Computer Science, Massachusetts General Hospital, Boston, MA.
Thomas D. Sequist, Division of General Internal Medicine, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA; Partners Healthcare System, Boston, MA.
Ronilda Lacson, Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA.
Ramin Khorasani, Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA.
Kunal Jajoo, Division of Gastroenterology, Department of Medicine, Brigham and Women’s Hospital, Boston, MA.
Laura Holtz, Department of Quality and Safety, Dana Farber Cancer Institute, Boston, MA.
Sonali Desai, Department of Medicine, Brigham and Women’s Hospital, Boston, MA.
References
- 1.Gandhi TK, Lee TH. Patient safety beyond the hospital. N Engl J Med. 2010;363(11):1001–2. [DOI] [PubMed] [Google Scholar]
- 2.Wynia MK, Classen DC. Improving ambulatory patient safety: learning from the last decade, moving ahead in the next. JAMA. 2011;306(22):2504–5. [DOI] [PubMed] [Google Scholar]
- 3.Sarkar U et al. Refocusing the lens: patient safety in ambulatory chronic disease care. Jt Comm J Qual Patient Saf. 2009;35:377–383, 341. [DOI] [PubMed] [Google Scholar]
- 4.Sarkar U et al. Pragmatic Insights on Patient Safety Priorities and Intervention Strategies in Ambulatory Settings. Jt Comm J Qual Patient Saf. 2017;43(12): 661–670. [DOI] [PubMed] [Google Scholar]
- 5.Singh H, Meyer AN, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. 2014;23:727–731. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Giardina TD et al. Root cause analysis reports help identify common factors in delayed diagnosis and treatment of outpatients. Health Aff (Millwood). 2013. August;32(8):1368–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Schiff GD et al. Addressing Ambulatory Safety and Malpractice: The Massachusetts PROMISES Project. Health Serv Res. 2016. December;51 Suppl 3:2634–2641 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Danforth KN, Smith AE, Loo RK, Jacobsen SJ, Mittman BS, Kanter MH, and the KPSC Outpatient Safety Net Program. Electronic Clinical Surveillance to Improve Outpatient Care: Diverse Applications within an Integrated Delivery System. eGEMs (Generating Evidence & Methods to improve patient outcomes). 2014; 2;1:Article 9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.MacMahon H, Naidich DP, Goo JM, Lee KS, Leung AN, Mayo JR, Mehta AC, Ohno Y, Powell CA, Prokop M, Rubin GD. Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology. 2017. February 23;284(1):228–43. [DOI] [PubMed] [Google Scholar]
- 10.Sim JJ, Rutkowski MP, Selevan DC, et al. Kaiser Permanente creatinine safety program: A mechanism to ensure widespread detection and care for chronic kidney disease. Am J Med 2015. November;128(11):1204–11. [DOI] [PubMed] [Google Scholar]
- 11.Whitehead NS et al. Interventions to improve follow-up of laboratory test results pending at discharge: A systematic review. J Hosp Med. 2018. February; 13:631–636. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Darragh PJ et al. (2018) A systematic review of interventions to follow-up test results pending at discharge. J Gen Intern Med. 2018. May; 33(5):750–758. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.The Joint Commission Perspectives. Request for comments: Processes and procedures for follow up on tests and diagnostic results. Jt Comm J Qual Patient Saf. 2018; 38(11): 1–2. [Google Scholar]
- 14.Meyer AND, Murphy DR, Singh H. Communicating Findings of Delayed Diagnostic Evaluation to Primary Care Providers. J Am Board Fam Med 2016;29:469–473. [DOI] [PubMed] [Google Scholar]
- 15.Roy CL, Rothschild JM, Dighe AS, Schiff GD, Graydon-Baker E, Lenoci-Edwards J, Dwyer C, Khorasani R, Gandhi TK. An initiative to improve the management of clinically significant test results in a large health care network. The Joint Commission Journal on Quality and Patient Safety. 2013. November 1;39(11):517–27. [DOI] [PubMed] [Google Scholar]
- 16.Sarkar U, Simchowitz B, Bonacum D, Strull W, Lopez A, Rotteau L, Shojania KG. A qualitative analysis of physician perspectives on missed and delayed outpatient diagnosis: the focus on system-related factors. The Joint Commission Journal on Quality and Patient Safety. 2014. October 1;40(10):461–AP1. [DOI] [PubMed] [Google Scholar]
- 17.Clarity C, Sarkar U, Lee J, Handley MA, Goldman LE. Clinician Perspectives on the Management of Abnormal Subcritical Tests in an Urban Academic Safety-Net Health Care System. The Joint Commission Journal on Quality and Patient Safety. 2017. October 1;43(10):517–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Murphy DR, Meyer AN, Sittig DF, Meeks DW, Thomas EJ, Singh H. Application of electronic trigger tools to identify targets for improving diagnostic safety. BMJ Qual Saf. 2019. February 1;28(2):151–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hammer MM, Kapoor N, Desai SP, Sivashanker KS, Lacson R, Demers JP, Khorasani R. Adoption of a Closed-Loop Communication Tool to Establish and Execute a Collaborative Follow-Up Plan for Incidental Pulmonary Nodules. AJR Am J Roentgenol. 2019. February 19; 1–5. [DOI] [PMC free article] [PubMed] [Google Scholar]