Electronic Clinical Surveillance to Improve Outpatient Care: Diverse Applications within an Integrated Delivery System

Abstract

Efforts to improve patient safety have largely focused on inpatient or emergency settings, but the importance of patient safety in ambulatory care is increasingly being recognized as a key component of overall health care quality. Care gaps in outpatient settings may include missed diagnoses, medication errors, or insufficient monitoring of patients with chronic conditions or on certain medications. Further, care gaps may occur across a wide range of clinical conditions. We report here an innovative approach to improve patient safety in ambulatory settings – the Kaiser Permanente Southern California (KPSC) Outpatient Safety Net Program – which leverages electronic health information to efficiently identify and address a variety of potential care gaps across different clinical conditions.

Between 2006 and 2012, the KPSC Outpatient Safety Net Program implemented 24 distinct electronic clinical surveillance programs, which routinely scan the electronic health record to identify patients with a particular condition or event. For example, electronic clinical surveillance may be used to scan for harmful medication interactions or potentially missed diagnoses (e.g., abnormal test results without evidence of subsequent care). Keys to the success of the program include strong leadership support, a proactive clinical culture, the blame-free nature of the program, and the availability of electronic health information. The Outpatient Safety Net Program framework may be adopted by other organizations, including those who have electronic health information but not an electronic health record. In the future, the creation of a forum to share electronic clinical surveillance programs across organizations may facilitate more rapid improvements in outpatient safety.

Introduction

The goal of this paper is to describe an innovative approach for addressing care gaps in ambulatory settings. The Kaiser Permanente Southern California (KPSC) Outpatient Safety Net Program is a collection of electronic clinical surveillance (ECS) tools that routinely scan electronic health information to identify potential care gaps and flag them for review and resolution by health care providers. We present here the overall framework, as well as examples of specific ECS tools, so that other organizations may learn from our experiences and adapt the framework for use in their settings if desired.

We also aim to stimulate the development of new ECS tools by describing key considerations for developing and implementing these tools. ECS does not require an electronic health record (EHR); it simply requires that the key pieces of information required by any particular ECS tool (e.g., lab results) are available electronically. ECS may be used to address a wide range of potential quality gaps, as illustrated by the diverse ways it has been used in our setting. We present the origins, development approach, key features, and implementation considerations of the KPSC Outpatient Safety Net Program as an example of innovative use of electronic health data to support high-quality, efficient care.

Background

EHRs have been championed as a means to improve health care quality and safety while also improving efficiency and reducing costs.1,2 Evidence suggests, however, that an EHR alone may be insufficient to address quality gaps.1,2 In 2009, the Health Information Technology for Economic and Clinical Health (HITECH) Act sought to encourage the adoption and meaningful use of EHRs by medical professionals and hospitals through financial incentives.1,3 Meaningful use of EHRs includes relatively straightforward activities, such as e-prescribing and computerized provider order entry, as well as more complex applications, such as clinical decision support and medication monitoring.1,4

The majority of patient safety and quality monitoring and improvement efforts leveraging EHRs have focused on inpatient care or acute, immediate care gaps, with less attention to care gaps in the outpatient setting.5–10 The landmark IOM report To Err is Human: Building a Safer Health System was virtually silent on safety issues in ambulatory settings.11 The consequences of care gaps in outpatient care may be equally serious, however, even if they are not as immediately apparent. For instance, a delay in diagnosing a patient’s cancer may mean the difference between a treatable and fatal cancer. Diagnostic errors related to failure to follow up clinically significant results are also a common source of medicolegal claims.10,12 Recently there have been calls to focus greater attention on patient safety in ambulatory settings,5–7 where most of the health care in the United States is delivered in an estimated 1 billion physician office visits each year.6,8,13,14

Care gaps in ambulatory care may result from several factors. Communication breakdowns contribute to outpatient care gaps, just as they contribute to inpatient care gaps, but are further complicated by the diffuse nature of care delivery.6 The inherent delay in the availability of key pieces of information (e.g., radiology or laboratory results that aren’t received until after a patient leaves the provider’s office) creates communication challenges in the outpatient setting that may only infrequently occur during inpatient care.6,7,10 Additionally, outpatient settings have a lower signal-to-noise ratio.6,10 That is, in ambulatory settings a greater number of healthy patients present with the same or similar symptoms as patients with serious health conditions, which may contribute to missed diagnoses or delayed follow-up care. Patients also play a greater role in their own care in outpatient settings.5–7,10 For instance, patients may contribute to delays in the follow-up of abnormal results by not responding promptly to their providers’ attempts to schedule a follow-up appointment.

In both inpatient and outpatient settings, identifying potential care gaps in a timely, reliable manner for which harms can be mitigated against is critical to improving quality and safety. Errors of commission, rather than omission, are greater in inpatient care than ambulatory care settings,5 and the speed at which errors need to be corrected is greater in inpatient settings. As a result, quality improvement (QI) efforts in inpatient settings may require a stronger emphasis on the prevention of errors through popup reminders or best practice alerts, as well as systems that will rapidly detect and correct any errors that do occur. In contrast, in outpatient settings, some care gaps may develop slowly or may not be immediately detectable, instead only emerging or becoming evident after some time. For instance, missed or delayed follow-up of an abnormal test result is not an immediately observable event; it can only be detected after allowing for the usual time in which follow-up of results occurs. Further, an immediate alert may not be sufficient to address the care gaps, as prompt notification of providers and patients of results represents a necessary but not sufficient step to ensure appropriate follow-up care.

We report here on an innovative approach that uses electronic health information and multidisciplinary teams to identify and resolve potential care gaps in outpatient settings. A distinctive feature of our approach is its role as a redundant backup system rather than as a system that focuses on providing an intervention during the clinical encounter in which initial care is provided. The KPSC Outpatient Safety Net Program uses ECS to routinely scan electronic health information for care gaps and to provide a “safety net,” in recognition of the fact that there may be missed care opportunities even with the best efforts to deliver high-quality care. Our approach takes advantage of the lower urgency of response in outpatient settings than in inpatient settings, and thus the system is implemented in a way that does not disrupt normal work flow or initial care, as opposed to real-time decision support or best practice alerts.

The KPSC Outpatient Safety Net Program is also designed to be blame-free. KPSC leadership has emphasized that the purpose of the program is to improve care quality – including cases in which the provider did not do anything wrong – and there is no attempt to track or review events at the physician level as part of their evaluation or metrics monitoring. Below, we present details on the development and implementation of ECS tools within our setting so that others may consider this framework as one way to target ambulatory care gaps in their organizations.

Origins of the Electronic Clinical Surveillance “Safety Net” Programs

ECS involves scanning electronic health information on a regular basis to identify patients with a specific condition or event. This approach was originally developed to identify adverse drug events for research purposes, and was later applied in practice-oriented systems designed to identify and address adverse drug events.15–17 In order for ECS to be effective in improving outpatient care, it requires that there is enough lag time, between an error and potential harm, to intervene and mitigate the potential harm. It also requires that the key pieces of data needed to identify potential care gaps are available electronically, although they do not have to be part of an EHR. ECS has a potential role to play in addressing diverse care gaps in outpatient settings: it can be used to scan for abnormal results lacking follow-up, which may not have been communicated back to the appropriate provider; to help flag patients with potential missed diagnoses or medication interactions; or to prompt reminders to the patient about needed care (e.g., a needed laboratory test or follow-up appointment).

Setting

KPSC is an integrated health care delivery system serving approximately 3.7 million members in 14 hospitals and over 200 medical offices. Patients interact with over 6,000 physicians, as well as numerous other health care professionals, during their approximately 12 million annual ambulatory encounters. KPSC provides comprehensive health care to its members, including primary care, preventive health services, optometry, obstetrics, and emergency and hospital services. Additionally, KPSC operates its own laboratory and pharmacies. QI activities across medical offices are prioritized and coordinated by a regional Quality & Clinical Analysis group, which also leads other regionwide activities such as guideline development, technology assessment, continuing medical education, member health education, and health research.

In 2006, KPSC began using an Epic-based EHR. Although KPSC had electronic health information systems in place prior to implementation of the Epic EHR, the EHR presented new opportunities for quality and safety improvement by integrating information on diverse aspects of care into a single record. For instance, shortly after adopting the EHR, KPSC launched its proactive office encounter, in which point-of-care reminders regarding needed follow-up care are triggered by any patient encounter.18 Thus, a patient being seen for an eye exam may be reminded of and scheduled for an overdue mammogram by a receptionist at an optometry clinic. This type of use of the EHR is one example of the “Complete Care” culture within KPSC.19 The multifactorial, team-based, proactive Complete Care approach to health care delivery seeks to comprehensively address members’ health by encouraging care coordination across providers and conditions. It also emphasizes the role of all employees in care delivery and the ability for care improvements to be achieved by staff other than physicians when their scope of practice allows.

Development of a Formal Outpatient Safety Program

Recognizing that the vast majority (>98 percent) of its patient contact occurs in the outpatient setting, leadership within KPSC launched the first of several ambulatory ECS programs in 2006. The development of a centralized program to address ambulatory safety issues was initially driven by two very different types of challenges: high-volume and low-volume occurrences. High-volume diagnostic or screening tests presented challenges due to the large number of results requiring monitoring and possible follow-up. The potential for delayed or missed follow-up of an abnormal result was further increased by the fact that providers might not be able to reach patients on the first try, thus requiring multiple attempts to schedule a follow-up appointment. By comparison, it was difficult to design an effective QI strategy to educate individual physicians or even local medical centers to identify infrequent safety issues. Additionally, there was not always a single provider or group with primary responsibility for the potential care gap. Thus, rare but serious safety concerns presented a different kind of challenge for which to design appropriate QI strategies.

These two types of potential care gaps were addressed by six initial ECS programs. In 2006, KPSC launched the prostate-specific antigen (PSA) Safety Net with the goal of improving timely follow-up of men with elevated serum PSA levels. PSA testing is commonly used to screen for prostate cancer, and the PSA Safety Net scanned EHRs daily to identify men with elevated serum PSA levels but without a record of follow-up care (i.e., no urology or oncology appointments subsequent to the elevated serum PSA level). Existing care managers contacted men flagged by the electronic PSA Safety Net tool to schedule expedited appointments.

An advantage of the Safety Net tool was that it could be accessed whenever time was available for panel management activities, as opposed to a pop-up reminder that required immediate action and might be more disruptive to a health provider’s workflow. Subsequently, similar ECS tools were created to improve follow-up of abnormal Pap smear results and fecal immunochemical test results, which are used to screen for cervical and colorectal cancer, respectively. Since the 1980s, Pap smear results have been tracked and followed up on by a dysplasia coordinator within KPSC. However, the new Safety Net provided automated, daily updating of patient lists and a new electronic tracking tool for managing follow-up care.

Two additional ECS programs were initiated that focused on rare health issues: (1) medication monitoring for patients on Digoxin and antiseizure medications, and (2) reevaluation of high-dose acetaminophen prescriptions. These programs were established to facilitate appropriate monitoring of patients on long-term medications and to reduce the risk of medication overuse. Additionally, a Safety Net was launched to improve timely follow-up of abnormal serum creatinine results in order to facilitate prompt diagnosis of chronic kidney disease. While creatinine testing is common, the decision was made to run this Safety Net program at discrete intervals (rather than daily) out of convenience since retesting did not need to be done immediately.

The implementation of these six initial programs demonstrated to leadership the feasibility and flexibility of the ECS approach, and in 2009 the KPSC Outpatient Safety Net Program9 was formally created to oversee the ECS programs focused on ambulatory care within KPSC. The Southern California Permanente Medical Group (SCPMG) Regional Medical Director of Quality & Clinical Analysis (MHK) sponsored the Outpatient Safety Net Program. The Safety Nets address a wide range of clinical areas (e.g., cancer, chronic kidney disease, diabetes) as well as a variety of quality issues (e.g., improved follow-up of tests, reduction of drug-drug and disease-drug interactions). They also utilize different aspects of electronic health information, involve different health care providers, and are run on different schedules. Their commonality is the use of electronic health information to address quality and safety gaps in outpatient care by using ECS.

Process to Develop and Launch New Electronic Clinical Surveillance Programs

KPSC has developed an efficient process for creating and launching new ECS programs (or Safety Nets), which could be adapted by other organizations and are enumerated in this section. A small core of centralized (regional) staff facilitate the development, launch, and maintenance of the Safety Nets. Within a 7-year period (2006–2012), 24 Safety Nets were developed and implemented (Table 1).

Table 1.

24 Electronic Clinical Surveillance Programs Implemented through the Outpatient Safety Net Program to Improve Ambulatory Care Safety and Quality

	Program Goal
Diagnosis Detection
Cancer
Cervical Cancer - Pap Smear	Improve timely follow-up* of abnormal Pap smears.
Colorectal Cancer – Fecal Immunochemical Tests (FIT)	Improve timely follow-up* of positive FIT tests.
Colorectal Cancer – Colonoscopy	Increase use of colonoscopy among members with (presumed) iron deficiency anemia or a history of rectal bleeding, and without a colonoscopy in the past 10 years.
Prostate Cancer – Prostate-Specific Antigen (PSA)	Improve timely follow-up* of elevated serum PSA levels.
Kidney Disease
Chronic Kidney Disease -- Creatinine	Increase repeat creatinine testing within 90 days of an initial abnormal creatinine result to facilitate prompt diagnosis of Chronic Kidney Disease (CKD).
Hepatitis C Virus Infection
Hepatitis C Virus Infection	Improve timely follow-up* of a positive hepatitis C antibody result.
Medication and Lab Monitoring
Potentially Harmful Interactions
Statin Medications
Amiodarone + Simvastatin	Minimize safety risks by medication conversion.
Gemfibrozil + any Statin	Minimize rhabdomyolysis risk by converting or stopping medications.
Antidepressants or Sleep Medications
Dementia + Antidepressants	Reduce drug-disease interactions through medication conversion.
Falls + Antidepressants or Sleep Medications	Reduce drug-disease interactions through medication conversion.
Monitoring
ACE/ARBs + Diuretics *	Increase annual monitoring (electrolytes and serum creatinine).
Anticonvulsants*	Increase annual monitoring (drug levels).
Digoxin*	Increase annual monitoring (potassium and serum creatinine).
Amiodarone	Improve follow-up of abnormal lab results to monitor for liver, thyroid, or pulmonary toxicity.
Overdue Labs in Primary Care	Increase the number of completed labs via patient reminder letters regarding labs more than 30 days overdue.
Other Medication Issues
ACE/ARB candidates	Increase appropriate hypertension medication use among members with diabetes.
Acetaminophen (APAP)	Decrease overuse of acetaminophen for members taking >4gm daily by converting prescription or referring to pain management clinics.
Digoxin Dose Adjustment	Decrease potential for toxicity.
Other Necessary Follow-Up Care
Abdominal Aortic Aneurysm (AAA)	Improve care by (1) point-of-care screening reminders, (2) follow-up based on clinical practice guidelines, and (3) tracking to improve timely referral to radiology or vascular surgery.
Chronic Kidney Disease (CKD) Progression	Identify members with rapid deterioration of renal function to ensure timely and appropriate follow-up.
Chlamydia Infection	Improve timely follow-up of positive chlamydia tests.
Newborn Hearing Screen	Ensure timely and appropriate follow-up of a failed hearing screening among newborns.
Post Splenectomy Immunization	Increase vaccinations among asplenia patients missing one or more recommended immunizations.
Unintended Pregnancy Follow-up	Increase birth control counseling following an elective termination of pregnancy.

TSH = thyroid-stimulating hormone

^*The definition of “timely follow-up” varies by the type of care gap being targeted by the Safety Net Program, and is jointly determined by leadership, clinicians, and other care providers.

^**Combined into a single medication monitoring tool at the end of 2012.

Overview of ECS Development Process

The idea for a new ECS Safety Net program may originate from a variety of sources, including internal quality imperatives and leadership priorities, peer-reviewed publications, information from federal agencies (e.g., FDA), frontline staff suggestions, and medicolegal claims that identify potential safety gaps. Ideas for new Safety Nets are first presented to KPSC Quality leadership for approval of the general concept (unless the idea originated with quality leadership). If approved, a Subject Matter Expert and Stakeholder (SMES) Panel is convened to develop a detailed plan of how the ECS program would work if approved and implemented. The SMES panel includes core regional staff, who are involved in all the Safety Nets, as well as frontline providers and others with content-specific knowledge (Figure 1).

Figure 1.

Process for Developing and Launching New Electronic Clinical Surveillance Programs through the Outpatient Safety Net Program

An early step in the detailed plan development is a review of the clinical evidence by the Evidence-Based Medicine (EBM) Services Unit, a group within KPSC that conducts evidence reviews to support the development of clinical guidelines, technology assessment, and other clinical and operational decision-making efforts. Thus, both expert clinical opinion within KPSC and the broader literature are considered in the development of a Safety Net.

Depending on the electronic health information needed for the ECS tool, an assessment of data quality and investigations of the data may be undertaken at this time. For ECS tools that use routine information, such as laboratory test results, no specific validation is done by the KPSC Outpatient Safety Net Program (although, e.g., the laboratory has its own quality control procedures). With more complex problems, such as when criteria for intervening are unclear or natural language processing is being used, chart reviews are performed in order to develop and validate the ECS algorithm. After the launch of a new Safety Net, providers provide an indirect measure of data quality, because they will report back if the ECS does not seem to work properly.

Once a detailed plan has been developed for the proposed new ECS program, KPSC Quality leadership reviews it, offers feedback, and (potentially) approves the plan for implementation. After this review and approval, an email is sent to all providers who might be affected by the implementation of the Safety Net Program to solicit their feedback. The Safety Net might be revised further based on feedback, or might be launched immediately if no feedback is received. Generally, a new ECS program can be launched within a few months.

Criteria and Considerations for ECS Safety Net Programs

The criteria that leadership use to decide whether to proceed with the development of a Safety Net includes an assessment of both the content and the feasibility of the proposed ECS tool to identify, confirm, and correct potential safety issues (Table 2).

Table 2.

Criteria Used by KPSC Leadership to Assess Potential Topics for Outpatient Safety Net Program

Criteria Used by KPSC Leadership to Assess Potential Topics for Outpatient Safety Net Program
Clinical Impact	Will it improve patient safety or quality? If yes, – How many people will be affected? – What is the severity of the potential safety gap?
Identification	Is the potential safety gap readily identifiable using existing electronic health data? That is: – Is the information required discretely coded? (e.g., captured by diagnostic or procedure codes, pharmacy data) – If the information required is not discretely coded, is it noted consistently in a way that makes natural language processing a reasonable option? If the potential safety problem is not readily identifiable, is it important enough that revisions to existing documentation should be considered?
Follow-up	How many patients might be identified & what follow-up would be necessary? – If the Safety Net is likely to identify a large number of patients, the importance of automated systems becomes more important (e.g., automated mailings to patients reminding them of labs) – If the number of patients identified is smaller, it may be feasible to have central review (e.g.. by a pharamcist or expert clinician) – In considering the capacity for follow-up, existing care processes and resources to resolve safety issues in a timely fashion are considered (e.g., care managers, automated direct mailings to patients**)
Other Operational Considerations	Is anyone already working on addressing this safety issue? Does the issue fall solely within one group’s domain? (and thus may be better-suited to intervention by that group rather than by the centralized approach of the Safety Nets)

^*The KPSC Outpatient Safety Net focuses on medication safety and diagnosis detection and follow-up. However, these are not listed among the criteria above because the Safety Net framework can be used to address a wide range of potential care gaps.

^**After quality gaps are identified, an intervention is required to resolve the quality gap in a timely fashion. These interventions have included: automated outreach to patients via mailed letter or Interactive Voice Recognition (IVR); notification of physicians with recommendations; use of case managers; use of pharmacists to review medications manually for flagged patients; batch order of lab tests with automated notification of the patient; and manual tracking and follow-up of other safety issues. “High volume” results require more automated solutions, or more staff, than “low volume” results; while there is no set definition for what constitutes high- versus low-volume issues, institutions will want to consider both the identification and resolution in assessing feasibility and solutions.

Clinical Impact

A potential Safety Net is first evaluated for its importance in terms of its potential impact on patient safety. This is the only critical factor – if intervening won’t improve patient safety or quality, there is no reason to implement ECS. Operational factors are then considered, such as the ability to identify safety issues using electronic health information and the ability to mitigate these hazards expediently.

Identification

The feasibility of identifying a potential care gap generally requires that the potential care gap can be detected using discretely coded data (e.g., coded by ICD9, CPT, or other discrete data fields). In rare instances, a natural language processing algorithm to identify potential safety issues from free text notes in pathology, radiology, or clinical visit reports may be considered. If the potential care gap still cannot be easily identified, modifications to clinical documentation might be considered.

Follow-up

In assessing the capacity for follow-up care, both the number of patients identified and existing resources are considered. If an important safety problem is identified that affects a large number of people, greater importance is placed on developing an automated solution (e.g., automated generation of letters to remind patients to come in for follow-up labs). Existing care managers are used to follow up some of the elevated lab tests, minimizing the physician workload. Additionally, the Safety Nets utilize a system to batch order laboratory tests that was initially created as part of KPSC’s chronic disease management processes. Batch ordering allows large volumes of follow-up test orders to be placed at a single time provided that agreed-upon criteria are met, which have been approved and signed off on by primary care providers in advance.

While the definition of “high-volume” and “low-volume” issues may vary across organizations in terms of absolute cut-points, the concept should have meaning within an organization in terms of the impact on resources and how the ECS is implemented. For instance, a high-volume issue may warrant more frequent running of the ECS (even in the absence of a safety reason to do so) in order to even out the workload, and it may require more automated solutions. In contrast, low-volume safety issues might benefit from less frequent running of the ECS to make efficient use of someone’s time when reviewing potential safety issues (assuming there is no need, for safety reasons, to run it more frequently) and may make it more feasible to have a specialist review patient records to evaluate the potential safety issue.

We present two examples to illustrate the difference in magnitude of potential safety issues identified in our setting. The KPSC Safety Net monitoring labs for patients on ACEs/ARBs and diuretics is an example of a higher-volume ECS tool that lends itself to an automated solution (e.g., automated reminder letters). During a 2-year period, over 250,000 patients were identified who needed lab monitoring. Following identification by the Safety Net, approximately 131,500 had their labs repeated, 16 percent of which yielded abnormal results.20 In contrast, over a 2.5 year period, the hepatitis C Safety Net identified 614 patients who had a positive hepatitis C antibody test without a confirmatory test. The Safety Net program resulted in 482 receiving a confirmatory test, 416 of whom were newly diagnosed with a hepatitis C virus infection.20 Automated letters could still be used, but the lower number of patients identified also makes it more feasible to have care managers make follow-up calls.

Other Considerations and Factors

Additional factors that are considered are whether other groups within the organization are already working on addressing the issue, in which case the Safety Net may not be necessary, or whether the problem falls clearly within the domain of a single group, and thus might be better addressed by that single group than by the centralized approach of the Safety Nets.

Leadership plays a key role in the development and success of the KPSC Outpatient Safety Net Program. The program was intentionally designed to be blame-free, in order to increase acceptance by physicians and other health care providers. Thus, the focus is on identifying patients who may need review or follow-up care – without trying to determine the cause for missed care – and then providing any necessary follow-up care. The KPSC Outpatient Safety Net Program operates under the assumption that some patients will be missed in routine care no matter how diligent the provider or system, and therefore uses a “safety net” to identify and resolve potential care gaps. No tracking of Safety Net results at the physician level is done as part of metrics or a clinician’s review.

Leadership emphasizes that the goal of the KPSC Outpatient Safety Net Program is to improve patient quality for our system, and not to track individual performance. Additionally, leadership is open about the purpose of the Outpatient Safety Net Program and shares findings on how many potential care gaps are identified and resolved with both internal and external audiences, which may contribute to the blame-free, safety oriented culture. Further, leadership seeks to implement as much of the work centrally as possible, rather than increasing the already busy workload of frontline physicians. In some cases, the clinical workload may even be reduced by the Safety Net (e.g., switching to reminding patients directly about overdue labs, rather than notifying providers).

Discussion

The KPSC Outpatient Safety Net Program represents an innovative application of health information technology (HIT) to identify and address a broad range of patient safety issues in ambulatory settings. This approach leverages the power of EHRs and a proactive clinical culture to scan for potential QI opportunities and intervene to improve patient care. The ECS tools that are part of this program target different types of quality gaps across several clinical content areas, illustrating one of the diverse ways in which electronic health information may be leveraged to improve safety and quality.

ECS tools may be used to complement computerized reminder systems and other real-time HIT applications and provide a second, automated look at patient data to identify potential care gaps. In some cases, ECS may fully identify patients requiring care, and the role of staff on the frontlines or in back offices is to provide the requisite follow-up care. In other cases, ECS may serve as an efficient way to identify potential misses for review by a clinician or pharmacist, who will then determine what additional care, if any, is needed. In both scenarios, the automated scanning of patient data is a key step toward improving care that requires relatively few human resources.

ECS is different from decision support or other prompts that are triggered during the normal work flow: rather than rely on point-of-care or other real-time reminders, the KPSC Outpatient Safety Net uses the complementary approach of ECS to identify and address potential care gaps. The program also includes a wide range of caregivers and staff rather than focusing exclusively on the role of physicians.

The KPSC Outpatient Safety Net Program builds upon and extends previous work and provides a framework that can be adapted by other organizations to address the quality issues in ambulatory care that are of highest priority to them. Prior research on potential errors in ambulatory care has commonly focused on medication safety or diagnostic and communication errors, including studies of the follow-up of abnormal results. Several existing studies have relied on malpractice claims or small patient samples; their findings may highlight important care gaps, but also may not be representative of diagnostic errors overall.10 Additionally, interventions to improve care have tended to focus on provider notification/reminder systems and other decision-support tools.10,12 In contrast, the KPSC Outpatient Safety Net Program casts a wider net by scanning electronic data for potentially missed care, regardless of cause.

Factors contributing to the success of the KPSC Outpatient Safety Net Program include strong leadership support; a provider culture that emphasizes the importance of a proactive, team-based approach to care (the Complete Care culture19); and an integrated delivery system in which laboratory results, office visits, and other information can be readily combined. However, the ECS tools could be adopted outside an integrated delivery system, even in the absence of an EHR, so long as an organization has electronic data available on the key Safety Net components. For instance, as long as serum PSA test results and claims data regarding follow-up office visits are available electronically, another organization could implement the PSA Safety Net even outside of an integrated delivery system and without an EHR.

On average, a new ECS tool or Safety Net has been launched every few months at KPSC. While some other organizations may require time to generate leadership support or develop infrastructure, the same algorithms employed by the KPSC ECS tools can be applied or adapted elsewhere. As organizations develop new ECS tools, publications and evaluations of them may help more broadly promote advancements in outpatient care safety. The establishment of a forum to share ideas and algorithms across organizations could help expedite national improvements in patient safety.

One limitation of our work on the KPSC Outpatient Safety Net Program is that the impact on patient harms has not been rigorously evaluated. For instance, while internal data have been examined regarding how many medication interactions have been identified, a rigorous assessment of the number of adverse outcomes (which would result from not just the existence of a medication error, but from actual harm occurring) avoided by identifying potential interactions has not been conducted. Additionally, while the blame-free nature of the Safety Nets likely contributes to their success and support from frontline clinical staff, there may be missed opportunities to understand causes of care gaps. However, this may also be considered a strength: a single Safety Net system may be an effective way to address potential safety issues with diverse underlying causes. Whether delayed care is due to provider or patient delay, for instance, the system will identify the delayed care and trigger follow-up. As KPSC continues to develop new Safety Nets, we will determine how best to evaluate and monitor their effectiveness.

Conclusion

The approach used by the KPSC Outpatient Safety Net Program for identifying and addressing potential care gaps in outpatient settings is flexible and readily adaptable by other organizations. While its implementation benefitted from our integrated care model and advanced EHR, these are not required elements so long as certain key data elements are available electronically. The framework for the ECS Safety Net Programs is flexible for addressing a wide range of potential quality issues across different clinical areas; and as other health care delivery systems develop and implement similar tools, information sharing of algorithms across organizations may help contribute more broadly to improvements in outpatient safety.