Characterizing barriers to closing cross-institutional referral loops:Workflow and information flow analysis

Abstract

The specialty referral process consists of primary care clinicians referring patients to specialty consultants. This care transition requires effective care coordination and health information exchange between care teams; however, breakdowns in workflow and information flow impede “closing the referral loop” and delay or prevent referrers from receiving the consultant’s “visit notes,” particularly in cross-institutional referrals. This study aimed to describe and map the referral process as it occurs in clinics and identify and characterize work system barriers affecting its performance. Referrers and consultants were interviewed about their perceived workflows, barriers, and clinical outcomes to inform a workflow analysis.

1. Introduction

1.1. Specialty Referral Process

The specialty referral process in the United States healthcare system consists of a primary care clinician referring a patient to a specialist consultant. The ensuing consultation is performed and documented through the consultant’s “visit notes,” which include the appointment results and consultant’s recommendations. The referral process is increasingly common within the U.S. healthcare system; approximately 105 million specialty referrals occurred in 2009, marking a significant increase from 40.6 million annually a decade earlier (IHI/NPSF, 2017).

The referral process has three principle actors: the referrer (in many cases, the primary care team), the consultant (or specialist), and the patient (Weiner, Savoy, & Barker, 2020). This is reflected in Fig. 1, which depicts the high-level idealized referral process workflow. Here, the referrer initiates the referral; the consultant evaluates the patient, generates recommendations, and transmits them to the referrer, and the referrer ultimately addresses ongoing clinical needs. The description of the patient as an actively engaged entity may include patient caregivers or family members who support the patient in interactions with clinicians. The patient is involved throughout the work process and may engage in work tasks themselves, including health information exchange (HIE) tasks supporting coordination between care teams during care transitions (B. S. Caldwell, Heiden, & Jahn Holbrook, 2021).

Fig. 1:

A high-level, patient-centered description of the referral process, characterized by referral initiation by a referrer, consultant activation, and the referrer response. The consultant activation process contains the actual consultation appointment and resulting recommendations and documentation and closing the referral loop by successfully communicating these outcomes to the referrer. The steps that are the focus of this paper are shown in the white boxes. Figure adapted from (Weiner et al., 2020).

Within the referral process, HIE between the referrer and consultant care teams is paramount. However, one study found that the referral process progressed through the consultation appointment and “closing the referral loop” (i.e., the referrer gaining access to the consultant’s visit notes) in less than 35% of cases (M. P. Patel et al., 2018). It is critical for referrers to have access to these visit notes to inform ongoing treatment. Poor or incomplete information exchange during care transitions can result in primary care clinicians needing to dedicate time to determine why care changes were made or making care decisions based on assumptions (Fylan et al., 2019; Johnson et al., 2012). As care teams become increasingly distributed and multidisciplinary, care coordination among clinicians and patients is critical to ensuring care quality. Clinicians who consistently experience care coordination challenges are significantly more likely to report negative impacts on the quality of care they are able to deliver (O’Malley & Reschovsky, 2011). Impaired information flows during care transitions such as the referral process can result in degraded or delayed care (Epstein, 1995; Horwitz, Moin, Krumholz, Wang, & Bradley, 2008; Jahn & Caldwell, 2018; Johnson et al., 2012; Pezzolesi et al., 2010), unnecessary repeated testing (Epstein, 1995), patient safety risks (Heiden & Caldwell, 2018; IHI/NPSF, 2017), patient frustration (Forrest et al., 2000; Ju, Guanzon, DeWitt, Liu, & Heit, 2020), and clinician frustration (Gandhi et al., 2000; Savoy et al., 2018).

Cross-institutional care coordination is plagued by additional challenges related to limited overlap in health information technology (HIT) systems (Fylan et al., 2019). In this context, a healthcare institution is a network providing a comprehensive range of care, including primary care, a diverse array of specialty services, emergency services, and inpatient care including intensive care. Although within-institution (or “internal” and cross-institutional referrals have similar workflows, cross-institutional referrals have higher risks of communication and coordination breakdowns in closing the referral loop due to interoperability deficiencies in the electronic health records (EHRs) used by different healthcare institutions (IHI/NPSF, 2017; S. A. Patel et al., 2020), among other factors. Although over 80% of consultants report always or mostly sending visit notes to referrers, only 62% of referrers report receiving the documentation as expected (O’Malley & Reschovsky, 2011); past research suggests this may be because cross-institutional information is not easily accessible (Phipps, Morris, Blakeman, & Ashcroft, 2017). Examining the gaps between the clinical referral process “as designed” and the organizational reality of referrals “as operating” can help identify barriers that hinder closing the referral loop.

1.2. Sociotechnical Work System Models

The process of “closing the referral loop” can be conceptualized as a process performed by the larger referral and consultation work system, which is itself a complex sociotechnical system. Thus, application of sociotechnical work system models developed for use in healthcare systems can be valuable in investigating, identifying, and describing systematic barriers to successfully closing the loop. Such models may include the Community Health Integration through Pharmacy Process and Ergonomics Redesign (CHIPPER) framework (Jahn & Caldwell, 2018), the Chronic Care Model (CCM; Wagner, 1998), and the Systems Engineering Initiative for Patient Safety (SEIPS) framework (Carayon et al., 2006; Holden et al., 2013; Werner et al., 2020). Although these models approach healthcare systems from different perspectives, they emphasize similar underlying work system factors.

SEIPS describes the work system, relevant processes, and the outcomes of those processes being completed in the work system. Although the SEIPS framework has evolved (e.g., SEIPS 2.0, SEIPS 3.0, SEIPS 101), the framework’s work system components include: person, task, tools and technology, organization, and immediate internal environment (Carayon et al., 2006; Carayon, Wooldridge, Hoonakker, Hundt, & Kelly, 2020; Holden & Carayon, 2021; Holden et al., 2013). Subsequent iterations recognized the external environment created by societal, economic, and policy factors beyond the boundaries of an organization (Holden et al., 2013) that is important when considering cross-institutional referrals.

SEIPS has been applied to describing a diverse collection of healthcare work systems, processes, and outcomes, including medication prescription in primary care and intensive care settings (Holden et al., 2013; Keating, McKinley, & Safdar, 2020), evaluation of consumer health informatics interventions (Martinez, Marquard, Saver, Garber, & Preusse, 2017), and analysis of novel remote care approaches to respond to the COVID-19 pandemic (Parmasad, Keating, Carayon, & Safdar, 2021). SEIPS frameworks are frequently utilized to identify and describe barriers and facilitators to work system goals (Carman, Fray, & Waterson, 2021; Parmasad et al., 2021; Werner et al., 2020; Wooldridge et al., 2019; Wooldridge, Carayon, Hundt, & Hoonakker, 2017). SEIPS 2.0 has been specifically applied to characterizing the barriers and facilitators affecting care transitions across a broad range of complex healthcare system features and challenges, such as hospital discharges (Carman et al., 2021; Werner et al., 2020). These care transitions, like those within the specialty referral process, require a high level of care coordination to prioritize patient safety. The SEIPS model is applied here as an operational example to investigate factors that impair effective coordination and care transitions.

1.3. Objectives

This study has two primary objectives. First, the study aims to describe the current process for closing the cross-institutional referral loop as it occurs in practice. Specifically of interest were the tasks associated with the consultation appointment and closing the referral loop processes. Second, the study seeks to identify and characterize barriers to that process by quantity, severity, likelihood of occurrence, and affected work system factors. (Steps 4–6 in Fig. 1). This characterization can guide future intervention efforts to focus on relevant aspects of the work system to address frequent and influential barriers that impact the referral workflow and patient care outcomes.

2. Materials and Methods

This study represents a secondary analysis of a larger study investigating the differences between cross-institutional and internal referrals, the design of which will be described below.

2.1. Participants

Administrators associated with two large medical institutions (see definition in 1.1) in Indiana provided contact information for referrers and consultants. Referrers were recruited from medical centers across Indiana, including urban, suburban, and rural areas; consultants were recruited from specialty clinics and tertiary hospitals in urban Indianapolis. A list of potential participants was generated following notifications sent out via emails, letters, meetings, and phone calls. Our intent was to recruit 20 clinicians, which is comparable to prior research on achieving thematic saturation in qualitative interviews (Guest, Namey, & Chen, 2020; Lowe, Norris, Farris, & Babbage, 2018). We aimed to enroll participants from diverse backgrounds, including self-reported gender, race, and age demographics. This study was approved by the Institutional Review Board at Indiana University (#10493).

2.2. Semi-structured Interviews

Semi-structured interviews were used to gather data from referrers and consultants about their workflows, barriers to workflow and information exchange, and the impact of those barriers on workflow and care outcomes. The semi-structured design enabled the interview to be guided by a set of consistent core questions while allowing flexibility for follow-up questions and deeper probing where needed. Interview guides for both consultants and referrers consisting of 10–11 questions were developed based on the SEIPS 2.0 framework (see Appendix A). These questions emphasize workflow factors such as the participant’s role in the referral process, differences between internal and cross-institutional referrals, and sources of patient information. Additionally, the interview guides included questions regarding clinical examples of care coordination and communication breakdowns and the impact of referral process barriers on the quality of care. Participants completed one approximately 45-minute interview with one or more research team members (AS1, AS2, AM, AD, EC, or AMG). Interviews were conducted virtually via Zoom-Health and were audio-recorded with participant consent.

2.3. Data Analysis

A two-phase data analysis approach was used to meet the study’s objectives. First, a workflow analysis defined tasks associated with the consultation appointment and closing the referral loop. Second, barriers to these processes were identified, aggregated, and characterized by quantity, severity, and work system factors.

Prior to data analysis, the recorded interviews were de-identified and transcribed. A rapid qualitative analysis approach (RQA) utilizing the SEIPS factors as a guide allowed for more targeted and efficient analysis (see St. George et al., 2023). Initial codes were established a priori based on the interview guides and SEIPS framework and were composed into an episode profile template that provided prompts for the analysts as an alternative to development and use of code books in the initial analysis. Three researchers (AMG, AM, AS1) assessed this template after independently reviewing the interview transcripts prior to further analysis, resulting in iteration to include emergent factors. Each interview was summarized, and an episode profile was created.

2.3.1. Workflow and Operational Process Description

The interview transcripts were further reviewed in full and in greater depth by another researcher with a human factors background (CMM) with an emphasis on the workflow descriptions of cross-institutional referrers. Detailed workflows from referral initiation to closing the loop were developed for each participant based on the transcripts. The individual workflows were then compared across clinician type and integrated based on common elements. However, only tasks associated with the consultation appointment and closing the referral loop were considered within the scope of this paper. The workflow descriptions generated were then compared to the episode profiles to ensure internal consistency.

2.3.2. Barrier Characterization

Transcripts were reviewed in full detail (by CMM) to identify barriers, which were defined as factors in the referral work system that impair workflow or information flow during the consultation appointment and closing the loop processes. Barriers must apply to the cross-institutional referral process but did not have to be unique to it; some barriers applied to both internal and cross-institutional referrals. We used qualitative analysis software, MaxQDA (VERBI GmbH, 2018), to code the interviews, focusing on reported instances or recurring barriers. Barriers identified from individual interviews were compiled and aggregated across the participants by similarity. Ten aggregated barriers were randomly selected and compared to the RQA outcomes, especially the episode profile content, to establish internal reliability.

These aggregated barriers were first described in terms of the workflow tasks they affect; this assignment was not mutually exclusive, as a single barrier could affect several different tasks within the consultation appointment or closing the loop processes. The barriers were further characterized by the work system factors as described in the SEIPS 2.0 framework (Holden et al., 2013) that represented the primary source of the barrier. This characterization was also not mutually exclusive, since one or more SEIPS work system factors could significantly contribute to a single barrier, reflecting the “pachinko model” description of multiple causal contributions to adverse events in healthcare (Barrett S. Caldwell, 2008). After SEIPS work system factors were initially assigned by one team member (CMM), a consensus meeting was held by two researchers with human factors and systems engineering backgrounds (CMM, AS1) to resolve any disparities.

Barriers were additionally rated by severity and likelihood to occur by a researcher who is a physician-scientist with 30 years of active medical practice experience in primary care (MW). The five-point likelihood (ranging from not likely (1) to extremely likely (5)) and severity ratings used (ranging from no or minor effect (1) through extreme effect (5): see Table 1) ratings were used according to guidance from two distinct settings (Nielsen, 1995; Sheridan-Leos, Schulmeister, & Hartranft, 2006). The heuristic evaluation methodology (Nielsen, 1995) in usability testing, uses this methodology to identify usability issues and characterize their effect severity on interface and user experience tasks (Kennedy, Kerns, Chan, Chaparro, & Fouquet, 2019; Rangraz Jeddi, Nabovati, Bigham, & Farrahi, 2020). Failure mode and effects analysis (FMEA) methodologies, including those used in analysis of healthcare treatment settings (DeRosier, Stalhandske, Bagian, & Nudell, 2002; Faiella et al., 2018; Sheridan-Leos, Schulmeister, & Hartranft, 2006; Thomadsen, Caldwell, & Stitt, 1998; Wetterneck, Skibinski, Schroeder, Roberts, & Carayon, 2004) also use this same five-point rating. However, we did not assume that providers would be aware of either the FMEA or usability contexts of such ratings scales. Instead, the effects of barriers as reported during the interviews were estimated by the medical subject matter expert (MW). The severity and likelihood ratings were mutually exclusive; each barrier had only one assigned likelihood and severity rating.

Table 1:

Definition of barrier severity levels, informed by the structure of models by Dumas and Redish (1999), Rubin and Chisnell (2008), and Kennedy et al. (2019), and further adapted to the context of the closing of the cross-institutional referral loop process by the 18 interviews conducted within the study.

Severity Level (rank)	Description
No issue (0)	The element does not negatively affect workflow or information flow.
Minor (1)	The barrier causes minor delays to an individual subtask or increases clinician workload marginally; may be associated with mild to moderate clinician frustration and burden, with little to no patient frustration or burden; potential for slight decrease in care quality due to minor inefficiencies.
Moderate (2)	The barrier causes significant delays to tasks or subtasks and may cause minor to moderate delays to care; associated with moderate to high clinician frustration and burden and low to moderate patient frustration and burden; may result in less clinician-patient time; potential for decrease in care quality due to moderate inefficiencies.
Major (3)	The barrier prevents task or subtask completion at their originally scheduled times; may result in repeated consultation appointments or testing, significant delay in care, or significant decreases in the quality of care; potential for patient morbidity.
Extreme (4)	The barrier prevents task or subtask completion; associated with severe delays and decreased quality of care; potential for patient mortality.

3. Results

3.1. Participants

Eighteen clinicians (six referrers, twelve consultants) participated in the study. Females (56% of total sample) represented 83% of referrers and 42% of consultants¹. Most participants (n=11) identified as white; however, Asian (n=4), multiracial (n=2), and Black (n=1) individuals were represented. Referring specialties included geriatrics (n=2), internal medicine (n=3), and primary care (n=1). Consulting specialties were neurology or pediatric neurology (n=5), cardiology (n=2), gastroenterology (n=4), and oncology (n=1). Participants held positions with any of four healthcare institutions (three public and one private) in the midwestern United States (see Table 2); eight participants held positions at two or more of these organizations.

Table 2:

Breakdown of participant affiliation with four healthcare institutions in the midwestern United States, specifically Indiana. Participants held a position with a minimum of one of the institutions (de-identified as A, B, C, and D). Eight participants were affiliated with two or more institutions and three participants were affiliated (currently or recently) with three of the institutions. Institutions A and B were those through which dissemination of recruitment materials occurred.

Institution Alias	n (referrers)	% (referrers)1	n (consultants)	% (consultants)1
Institution A	5	83%	6	50%
Institution B	2	33%	11	92%
Institution C	1	17%	2	17%
Institution D	0	0%	3	25%

3.2. Supplemental Workflow Processes

The consultation appointment and closing the referral loop processes provide preliminary, high-level insight into the cross-institutional referral workflow. Fig. 2 and Fig. 3 depict referral workflow tasks as they occur in practice (“as operating”), expanding on the process flow diagrams found elsewhere in literature (e.g., Fig. 1). Thus, these diagrams reflect uncertainty and the results of unsuccessful task completion, as well as multiple parallel or alternative methods to accomplish a single goal (e.g., information retrieval by the referrer team). Additional “hidden” tasks were uncovered by this more detailed task analysis, including:

• The consultant prepares for the patient’s appointment by reviewing the available records for the patient, focusing on the reason for their referral, their medical history, and any relevant diagnostic test results (CA-1).

• The consultant sends the visit notes to the referrer care team (CL-1). The visit notes are transmitted either (1) from EHR to EHR, (2) directly via phone, email, fax, or postal mail, or (3) indirectly via the patient themselves. This process is substantially more complex for cross-institutional referrals, as “internal” referral visit notes are accessible directly within a shared EHR tool.

• If notes are not successfully received, the referrer may need to engage in information search tasks (CL-0). This may involve requesting records from the consultant or using other HIT tools, such as external EHR or HIE systems to gather the needed information. This task may require multiple attempts to contact the consultant team or accessing several different systems depending on where (if anywhere), the visit notes are available. Often, this set of tasks is triggered by the referrer learning from the patient that the consultation appointment took place, but they never received the visit notes.

Fig. 2:

Detailed workflow associated with the consultation appointment process, with tasks labeled according to their corresponding tasks as described in Table 2. Consultant documentation of visit notes and other appointment outcomes (CA-3) may be triggered an additional time upon receiving test results if the consultant orders further tests. This is especially true when the testing process requires significant time to schedule and complete. Depending on whether the original referrer’s information is included in the test order, these results may also be provided directly to the referrer. This order process was beyond the scope of this paper, though it is depicted as a high-level process.

Fig. 3:

Detailed workflow associated with the closing the referral loop process; tasks are labeled according to their corresponding labels as described in Table 2. *These barriers are aggregated below into a single workflow step referred to as “receiving consultation results.” **Note that in this case “external EHR” is an EHR that belongs to an institution other than the patient’s primary health network. In cases where this access is possible, the referrer or a supporting team member must have a cross-appointment with the “external” institution.

The number of barriers associated with impairing workflow or information flow throughout the consultation appointment and closing the referral loop processes is shown in Table 3. We identified a total of 47 aggregated barriers (see Appendix B), 43 of which were unique and 4 of which were similar but occurred in different tasks performed by different actors, yielding 86% (43 of 50 individual barriers) reliability with the episode profiles. Seventeen barriers were associated with the consultation appointment itself, with most barriers pertaining to the consultant’s preparation for the appointment. Twenty-five barriers were identified within the closing the referral loop process; roughly 70% of these barriers were related to the consultant team sending visit notes or the referrer engaging in information searching tasks to find the visit notes.

Table 3:

Workflow tasks associated with the cross-institutional referral process, beginning at the consultation appointment process and extending through the end of the closing of the referral loop process. Tasks are defined and described in terms of the number of barriers that affect that task.

Higher-level Process	Workflow Tasks and Definitions	Number of Barriers
Consultation appointment	Consultant prepares for the appointment (CA-1): The consultant reviews the referral order and available medical records. Information is exchanged between the consultant and their electronic health record (EHR).	6
	Consultant evaluates the patient (CA-2): The consultant evaluates the patient. Information is exchanged between the patient and consultant.	4
	Consultant documents visit notes (CA-3): The consultant documents their consultation notes (results and recommendations based on the appointment), which may include initial documentation subtasks (on paper or via dictation) and entering the notes into the consultant’s EHR. Information is exchanged between the consultant and their EHR.	4
	Consultant orders further tests (CA-4): The task in which the consultant creates a diagnostic test order. This acts as a recursive referral function. Information is exchanged between the consultant and their EHR.	3
	Total number of consultation appointment barriers	17
Closing the loop/referrer information gathering	Referrer team requests or searches for visit notes (CL-0): The referrer team requests the consultation results from the consultant, often via phone call, but sometimes via fax. Information is exchanged between the referrer team and the consultant team. OR the referrer team accesses and performs search tasks within HIEs or external EHRs to identify the visit notes. Information is exchanged OR the referrer team requests the appointment outcomes and any documentation the patient has from the patient themselves. Information is exchanged between the referrer team and the patient.	12
	Consultant sends visit notes to the referrer (CL-1): The consultant tasks associated with initiating the transmission of consultation records (usually within the EHR) to the referrer. Information is exchanged between the consultant team and their EHR.	11
	System transmits visit notes (CL-2): The consultation records are transmitted (usually through fax or mail, but sometimes from the consultant’s EHR to the referrer’s EHR). Information flows from one institution to another.	2
	Referrer team processes the visit notes (CL-3): The referrer team receives visit notes, either from the consultant directly, the patient, external EHRs, or HIEs. The receipt of these visit notes must be detected by the referrer team. These visit notes or appointment outcomes are entered into the referrer’s EHR or otherwise connected to the patient’s file. Information is exchanged between the referrer team and their own EHR.	4
	Referrer reviews visit notes (CL-4): The referrer reviews the visit notes and consultation appointment outcomes for understanding and determines their next steps regarding patient care.	1
	Total number of referrer information gathering barriers	25

3.3. Characterizing Barriers by SEIPS Factors, Severity, and Likelihood

Barriers were further described in terms of their severity, likelihood, and causal SEIPS work system factors (see Table 4). The majority of barriers (n=31, 66.0%) were associated with tools and technology factors, including physical tools (e.g., paper notes), communication tools (e.g., fax, phone), and digital interfaces (e.g., EHRs, HIEs, secure chat). Organizational factors such as policies, norms, and culture (n=16, 34.0%), and task factors like complexity and ambiguity (n=13, 27.7%), contributed to at least a quarter of the barriers each. No barriers in the dataset were related to the physical environment. Low-severity barriers were prevalent, but 10 barriers (21.2%) were considered high-severity, defined as major or extreme. Barrier occurrence likelihood was distributed throughout the dataset.

Table 4:

Breakdown of number of barriers and percentage (%) of total barriers (n=47) associated with each SEIPS work system factor, severity level, and likelihood rating. Totals do not add to exactly 100% due to rounding.

Barrier Characterization Dimension	Number of Barriers	% of Barriers
SEIPSa Work System Factor
Person	5	10.6%
Task	13	27.7%
Tools & Technology	31	66.0%
Organization	16	34.0%
Internal Environment	0	0%
External Environment	7	14.9%
Severity
Minor	21	44.7%
Moderate	16	34.0%
Major	6	12.8%
Extreme	4	8.5%
Likelihood
Not likely	8	17%
Somewhat likely	12	26%
Moderately likely	9	19%
Very likely	8	17%
Extremely likely	10	21%

^aSystems Engineering Initiative for Patient Safety framework

Further insights can be gained by considering the relationships between the barrier description dimensions. Barrier severity (Fig. 4) and likelihood (Fig. 5) can be broken down by workflow task and causal SEIPS factors. Tools and technology factors are heavily represented in both high-severity (60%) and high-likelihood (72%) barriers, with the greatest concentration occurring when the consultant team sends the visit notes (CL-1) and when the referrer team requests and searches for visit notes (CL-0). Organization factors were related to approximately half of the high-likelihood barriers but were not heavily represented in high-severity barriers. While high-likelihood barriers (i.e., very or extremely likely) are distributed throughout the workflow, high-severity barriers affect sending of visit notes (CL-1), requesting and gathering of visit notes (CL-0), and the consultation appointment itself (CA-2).

Fig. 4:

Number of barriers of each severity level (minor, moderate, major, and extreme) attributed to each task within the consultation appointment (CA-1 through CA-4) and closing the referral loop (CL-1 through CL-5) processes, further broken down by primary work system factor contributor(s) according to the SEIPS 2.0 model.

Fig. 5.

Number of barriers of each likelihood level (not likely, somewhat likely, moderately likely, very likely, and extremely likely) attributed to each task within the consultation appointment (CA-1 through CA-4) and closing the referral loop (CL-1 through CL-5) processes, further broken down by primary work system factor contributor(s) according to the SEIPS 2.0 model.

Joint analysis of severity and likelihood ratings (Table 5) reveals that there are no high-severity, high-likelihood barriers. Instead, two primary groups of interest emerge: high-severity, low-likelihood barriers and low-severity, high-likelihood barriers.

Table 5:

The number of barriers associated with each severity and likelihood rating. Two barrier types of interest are shaded, the high-severity, low likelihood barriers (bottom left) and the low-severity, high-likelihood barriers (top right).

	Likelihood Ratings
Severity Levels	Not likely	Somewhat likely	Moderately likely	Very likely	Extremely likely	Total
Minor	2	2	3	6	8	21
Moderate	1	5	6	2	2	16
Major	3	3	0	0	0	6
Extreme	2	2	0	0	0	4
Total	8	12	9	8	10	48

3.4. Barriers of Interest: Two Groups

High-severity, low likelihood barriers identified by participants (see Table 6) are highlighted due to the magnitude of their potential adverse impact, despite their low likelihood rating. Over half of the nine unique barriers matching this description are associated with tools and technology factors, with other work system factors being represented uniformly throughout. The consequences associated with many of these barriers, as noted by the rater (MW), include the process being incomplete and severe effects on care quality or timeliness. Barriers during the consultation appointment itself include possible patient-consultant language barriers and the complexity of virtual appointments. Organizational EHR access constraints and government regulations may inhibit the referrer finding visit notes. Finally, inaccurate or unavailable referrer contact information, time and staffing requirements of contacting external referrers, and the lack of error or confirmation messages when sending visit notes impair consultants’ efforts to proactively provide the visit notes.

Table 6:

High-severity barriers (severity levels of major or extreme) that influence information flow and workflow. These are described in terms of the task during which the barrier is encountered, the SEIPS factor that best reflects the nature of the barrier and its source, and a representative quote to provide more vivid insight into the barrier.

Barrier Description	Workflow Step(s)	SEIPS Factor	Representative Quote(s)
Language barrier between patient and specialist consultant*	CA-2: Consultant evaluates the patient	Person, Task	“When you have a Hakha Chin kid, you got to find a Hakha Chin interpreter... which we usually can... We have now two interpretative services. So sometimes... like last week, I saw a colleague had a Mayan patient, and couldn’t find an Mayan interpreter in one interpretation service, so we had to go to another” (C6).
Virtual visits are more complex due to the lack of familiarity and number of appointments*	CA-2: Consultant evaluates the patient	Person, Task	“Because all this stuff has been thrust on us in a very short amount of time, and we’re just really learning on the fly. A lot of the stuff. Now we’ve got a new system. Now we have a new virtual system... And so, I have 10 patients scheduled for my Friday clinic next week, and I don’t know how we’re going to get through them all, but we’ll learn” (C6).
Constraints around accessing some EHR systems (including constraints on physical location)	CL-0: Referrer team requests or searches for visit notes	Organization	“The [healthcare institution]’s tougher because the [healthcare institution], at least traditionally, hasn’t had an external website, so you actually had to be on their premises to log in and look up a patient. And these aren’t necessarily routine. I mean, there are occasional patients who have almost all their records at [healthcare institution], and it’s tremendously hard to take care of a patient you know nothing about” (C8).
State and federal regulations for patient privacy	CL-0: Referrer team requests or searches for visit notes CL-1: Consultant sends visit notes to the referrer	External Environment	“I think the reason we can’t see it is because of HIPAA and all these other regulations. I mean, I think if nationally, everybody opened up their EMR, then it really wouldn’t matter where people went, and they could get care wherever was easiest. But so, I mean, I think it’s completely dependent on policy” (R5).
No error message is displayed if visit notes fail to send*	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“So for the external, even in that system, if the address is [wrong] or if the fax number is wrong or if something changed, then there may be a lapse. And I may be thinking I sent it and they didn’t get it. So there’s no feedback loop” (C2).
Lack of a confirmation message that visit notes were successfully received*	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“I hit the button and then I’m like, "Do they ever get it? I have no idea.” Really no idea. We did a project a few years ago where we tried to call referring docs and say, “Hey, we sent you a note, did you get it?” It wasn’t a very successful project, not because that rate was really low, just because it’s hard to get a hold of people. it’s hard to talk to the person that would actually know if they got the note” (C4).
Contact information for referrers may be inaccurate	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“All different providers and keeping that huge list of referring providers up to date and current, and being able to find them when you need to is probably the biggest barrier” (C7). “Just the fact that you don’t have information, you can’t get information to them because the fax number is wrong or the computer has the wrong listing or that person retired a year ago, and they’ve switched providers” (C7).
Contact information for referrers may not be available	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“Today I told a patient... I said I’m gonna change your blood pressure medicine, but I want to call your primary care doctor and talk to him about it to make sure he’s okay... with me doing that. And then, call me back and let me know. Now why should I have to, why should she have to do that? She had no idea what blood pressure medicine probably I’m trying to prescribe, but I don’t have a good way to get your primary care doctor” (C1).
Time and staff requirements are high for communicating and coordinating with external referrers	CL-1: Consultant sends visit notes to the referrer	Tools & Technology, Organization, External Environment	“So, when we’re getting a consultant involved, there is quite a bit of high-level care coordination, especially in the nursing home setting because we’re trying to avoid a hospitalization” (R2). “And those have to be forwarded to us, and then they get scanned into our records system, and that can be anywhere from a semi smooth process to a completely impossible process that requires multiple other steps, me calling the physician’s office, my referral coordinators calling the referring office, et cetera. So it can be a much less smooth and much less complete process” (C11).

^*denotes barriers that are of the highest severity.

Low-severity, high-likelihood barriers are important because of their frequency (see Table 7). Although they may not have severe adverse impacts, these commonly encountered barriers can contribute to frustration and cause inefficiency, especially if multiple are encountered. Over 70% of the high-likelihood barriers were associated with tools and technology factors and half were related to organizational factors. Though the highest percentage of high-likelihood barriers occur when referrers search for visit notes, such barriers are distributed throughout the workflow tasks.

Table 7:

High-likelihood barriers (likelihood levels of very or extremely likely) that influence information flow and workflow. These are described in terms of the task during which the barrier is encountered, the SEIPS factor that best reflects the nature of the barrier and its source, and a representative quote to provide more vivid insight into the barrier.

Barrier Description	Workflow Step(s)	SEIPS Factor	Representative Quote
Paper-based intake forms limit information flow*	CA-1: Consultant prepares for appointment	Tools & Technology	“Well, there are many opportunities for improvement. For example, I don’t have a sheet here, but when a new patient comes to us, they fill out manually this patient intake form. Who’s your referring doctor? What are your medicines? What are your family history and all that. That has to be filled out by hand. Nope, there should be a website that patient can go into, assuming that they have the electronic ability to do that, and they can fill out all of that information ahead of time” (C11).
Insufficient time to review patient records before the appointment*	CA-1: Consultant prepares for appointment	Task, Organization	“From my end, basically, so I don’t typically have enough time to prepare for the next [day].... The thing is, most days there’ll be few no shows here or there. So I’m able to catch up. But on the days like yesterday, I had everybody show up. I was struggling to keep ahead” (C8).
Difficulty finding patient records in the EHR*	CA-1: Consultant prepares for appointment	Tools & Technology	“And then, we want Google-like search functionality to actually find the transthoracic echo that we were looking for, or the cardiac cath data, or whatever. So, we don’t want to hand go through each one of those individually. We want some sort of search function and we want a repository that has all that data in it so that we can run that search function against it” (R3).
Patients may not be able to provide accurate and useful (and needed) medical information and history	CA-2: Consultant evaluates the patient	Task, Tools & Technology, Organization	“And then just, just missed information. Patients just aren’t necessarily the best historians and if you don’t have all the data and you, you know, you don’t necessarily come to the correct conclusion. And so, missing that data [can lead to] massive delays can just lead to the wrong diagnosis” (C1).
Limited time is provided for each appointment*	CA-2: Consultant evaluates the patient	Organization	“It depends on the providers. For me, I have 30 minutes for a new patient and 15 minutes for a return patient” (C8).
Having to review notes written by fellows or residents	CA-3: Consultant documents visit notes	Task, Organization	“So, it’s not just you writing the note, you’re relying on [the resident or fellow] to write the note. And that can sometimes be inaccurate, and that can sometimes get... That’s an extra layer of complexity, for which can screw up communication” (C6).
Insufficient time to enter visit notes (or other appointment outcomes) into the EHR immediately (during or immediately following the appointment)	CA-3: Consultant documents visit notes	Tools & Technology, Organization	“If you’re putting in orders outside or after the encounter and you have any questions for the patient that may come up during or completing the [EHR’s] templates, but the patient’s not there. What do you do then?” (C3) “Right, so if it’s someone who’s a little bit complicated or took a little bit longer, I may not have time to do my note, my dictation. And so in these sorts of situations, I may end up at the end of the day, I may be stuck with doing 10 notes. At the end of the day, after I’ve seen all the patients” (C8)
Inability to access external EHRs and their contact*	CL-0: Referrer team requests or searches for visit notes	Tools & Technology, Organization, External Environment	“I think everybody in the state of Indiana ought to have a [specific EHR] account, which would require them to register for it, but also to check it regularly. But if everybody in the state had an [EHR] account, any report I sent, it’s gone, it’s there immediately. For example, when I’m sending communication back to a referring physician, the primary care doc will show up there. But if I click that, it will immediately tell me John Smith and it says no external access, which means they don’t have [specific EHR]. Well then I’ve got to go to this other process of create provider letter and then if they’re John Smith, then it’s the same process I described before of figuring out which John Smith it is. So put everybody in the state of Indiana on [the specific EHR]” (C11).
Patients who are unable to provide visit notes or an accurate account of the consultation	CL-0: Referrer team requests or searches for visit notes	Person	“[B]ut there’s a fair number of patients who are cognitively impacted in one way or the other, so can’t consistently, reliably tell us what the consultant said, or recommended, or that sort of thing. And we wouldn’t act on... it helps us to know through the patient’s version, but we’re not liable to actually act on ordering tests or that sort of thing without direct communication from the clinician” (R3).
EHR is difficult to navigate	CL-0: Referrer team requests or searches for visit notes	Tools & Technology	“Oftentimes, [the EHR] is not intuitive enough. A lot of people complain about that. It’s... And they change, now, the flow, oftentimes, of how you get labs, and how you get outside labs. And every so often there’s a change that you have to struggle through” (C6).
Time and staff requirements are high for communicated and coordinating with external consultant care teams	CL-0: Referrer team requests or searches for visit notes	Tools & Technology, Organization, External Environment	“And those have to be forwarded to us, and then they get scanned into our records system, and that can be anywhere from a semi smooth process to a completely impossible process that requires multiple other steps, me calling the physician’s office, my referral coordinators calling the referring office, et cetera. So it can be a much less smooth and much less complete process” (C11).
Lack of a secure, asynchronous tool for communicating with external consultant care teams*	CL-0: Referrer team requests or searches for visit notes	Tools & Technology	“They only use [the asynchronous communication tool] for internal communication. So if somebody in Fort Wayne has a question, I can’t use it to communicate with them because they’re not on our kind of platform. So it makes it easy to communicate. We’re using a personal device, it’s a secure communication, but it only works for internal providers. You can’t use it outside of our systems” (C3).
Lack of a secure, asynchronous tool for communicating with external referrer care teams*	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“I think just allowing docs and nurses to talk more freely with each other you know would, would be ideal, but there’s not really a good format for doing that. Then nothing’s been decided upon and so, ultimately, what happens is people communicate in all these you know patched together formats” (R1).
Institution-centered mindset regarding information flow (influencing sending visit notes)	CL-1: Consultant sends visit notes to the referrer	Organization	“And so that institutional policy to keep the texting within the institution inhibits the ability to communicate with those outside makes us much less efficient... It’s institutional mindset at the level of the senior administrators of the hospitals from practice plans or the advice from their legal consultants.... For me personally as somebody who has a national reputation, it makes it incredibly difficult” (C10).
Contact lists or databases for external referrers are poorly structured*	CL-1: Consultant sends visit notes to the referrer	Tools & Technology	“Then for an external provider, let’s say that they were sent by Dr. John Smith. So I go into my list of John Smiths in the electronic record to generate a letter to them. So how many John Smiths do you think are in? And this has been a big, big problem for me. In some cases, I actually have to go in the list, select a number or select a name and look at the area code on their fax to say that John Smith is in [city], where the patient came from versus it’s [area code], so that’s a John Smith in [other city]. This is a very arduous process which can lead to great frustration and the ultimate frustration is that the letter never gets back to the referring physician” (C11).
Different health institutions’ EHRs don’t interface or communicate with each other*	CL-2: System transmits visit notes	Tools & Technology, Organization, External Environment	"With external, most of the time, it’s not going to be available immediately in our EHR because they still don’t talk to each other. So it’ll all be reliant on that faxed information that’s sent over into the chart” (C7). “But as far as I know, at least [with] [EHR 1]. I know it doesn’t talk to other [EHR 1] systems. It certainly doesn’t talk to [EHR 2]. I know [EHR 2] systems can talk to one another, but [EHR 1] doesn’t really talk to other systems” (C3).
Faxing introduces redundant printing and scanning and is prone to errors*	CL-2: System transmits visit notes	Task, Tools & Technology	“And I feel like we could have a better workflow institutionally and I don’t know if there’s a way to set Epic. There should be a way to set up Epic to do this where when we put in an external order, somehow it faxes it to the right number automatically. It seems sort of silly to rely on humans to print out the referral, have an MA grab it off the printer, fax it on another machine” (R6).
Too many meaningless notifications in the EHR	CL-3: Referrer team processes visit notes	Tools & Technology	“[T]hey may or may not send me the note, but honestly, it’s kind of lot of glut in my inbox. So, it’s okay if they don’t send it cause I know I’m gonna go back and look for it anyway” (R1). “The bad part of it is that we get notified a lot and on patients that arenť necessarily ours. For example, for some reason I’m assigned to a lot of patients that are pregnant, but I don’t do OB, but every time one of them has a baby, I get a notification that they were in hospital and had a baby. We get so many inbox messages already. Those are ones I don’t need to get” (R6)

^*denotes barriers that are of the highest severity.

4. Discussion

Based on consultants’ and referrers’ perceptions, tools and technology barriers account for the majority of total, high-severity, and high-likelihood barriers, indicating a need for improved HIT systems and implementation processes. Our results provide contextual evidence and novel characterizations of barriers to closing the cross-institutional referral loop, ultimately providing insight into knowledge gaps associated with the cross-institutional referrals. Barriers to information flow and workflow were described according to causal SEIPS factors, likelihood of occurrence, and severity of their affects on workflow, care quality, and clinician and patient frustration. SEIPS factor characterization provides insight into work system elements that act as sources of barriers. Likelihood and severity ratings inform intervention effort prioritization, with high-severity and high-likelihood barriers indicating high-priority challenges.

4.1. Operational Tasks: Hidden Workload

The specialty referral process described in our research highlights multiple administrative, HIT, and organizational barriers perceived to limit or prohibit effective information sharing, degrade information quality, and increase clinician team workload. One challenge to effective healthcare coordination in the United States is the limited effectiveness of information flow and task coordination among actors (Caldwell, et al., 2021; Jahn and Caldwell, 2018), which also results in a number of unexpected or “hidden” sources of clinician as well as patient workload. The operational tasks referenced in our interviews often resulted in much more work than expected to complete presumably “simple” tasks. An existing study described such workload increases stemming from hidden complexities of internal referral loops (Militello, et al., 2018). Our findings uncover additional complexities during cross-institutional referrals. In conditions of task overload and poor information quality, the potential for decision making errors and patient safety risks increase (Caldwell, 2008b; Thomadsen, et al., 1998; 2003), leading to a need to prioritize reductions in hidden workload.

4.2. Not all Barriers are Perceived as Equal: Severity, Likelihood, and SEIPS work System Factors

As described above, fields ranging from heuristic evaluations of software usability to engineering analysis of critical safety processes recognize that not all system process barriers or deficiencies are equally impactful in terms of system performance outcomes.(Savoy, Patel, Flanagan, Weiner, & Russ, 2017) Both likelihood of occurrence, and severity of impact, create risks to patient safety and other outcomes (DeRosier et al., 2002; Faiella et al., 2018; Sheridan-Leos et al., 2006; Thomadsen et al., 1998; Wetterneck et al., 2004). However, it is not clear how often clinicians or healthcare administrators address these issues when considering work system processes or barriers to effective care coordination (Caldwell, 2008; Caldwell, et al., 2021; Jahn and Caldwell, 2018; Thomadsen, et al., 1998).

It is important to note that approximately 70% of high-likelihood barriers (those most likely to appear and cause some level of impact on care coordination) and 69% of high-severity barriers (those most likely to cause significant or catastrophic impacts on patient outcomes, regardless of likelihood) were HIT-related. This further highlights the importance of effective design, implementation, and integration of HIT tools for both within- and cross-institutional referrals. Barriers associated with unavailable or inaccurate contact information limit awareness of or access to critical patient information, including notes or relevant imaging data (Savoy et al., 2023).

The fragmented nature of the US healthcare system makes such HIT-related barriers more difficult to detect, due to a lack of a coordinated “ownership” or lifecycle management of patient safety resources between healthcare delivery, insurance, regulatory, and technology implementation actors. Nonetheless, multiple high severity barriers were identified in this study which can have critical impact on clinical outcomes, regardless of overall system detectability. A lack of notification regarding updates to medical records, failures in the presentation and resolution of error messages, or inaccurate contact information for referrers, result in critical gaps in information available to clinicians to support patient care decisions. Limited opportunities for clinicians, hospital administrators, or HIT managers to identify and rectify these barriers may exacerbate adverse clinical outcomes.

4.3. “Low-hanging Fruit” for Intervention

A number of barriers identified in this study address organizational norms, practices and policies, including those specific to the referral process. Barriers influencing the referral process include organizational culture and policy elements related to openness for change, notification to personnel in advance of technology upgrades, or process standardization across different departments with different needs. However, a barrier’s existence does not universally indicate that action should be immediately taken to resolve it. Some person, task, organization, and external environment factors impede the workflow and information flow but serve other purposes (e.g., HIPAA regulations limit sharing information but protect patient privacy).

Additional opportunities for “low-hanging fruit” change interventions include elements of administrative and technical acknowledgments and confirmations, which are less tied to policy or regulatory requirements. Examples of such interventions include the following.

• Routinely updated contact information: Although the scope of this analysis was constrained to the process of closing the referral loop in a cross-institutional referral, taking a holistic view of the full referral process is important. Failures within the referral workflow before the consultation appointment can result in significant disruptions or workflow termination (Zimolzak et al., 2022), and some errors cascade through the processes. For example, if contact information for the referring clinician is incomplete or incorrect, the consultant information gathering process will be impeded; visit notes will not be successfully transmitted to the referrer; and any labs or testing ordered by the consultant will also not be available to the referrer.

• Notification availability and customizations: One critical, high-risk barrier is a lack of notifications for important changes to medical records, including consultation and laboratory results. Consultation and laboratory results inform care, but not all such changes are equally urgent for all patients. A lack of notification for important changes to medical records, especially triggered by interfacing with external sources, can have serious consequences for clinical outcomes in high-urgency cases with heightened patient risk or time-sensitivity. However, receiving too many notifications was also identified as a low-severity barrier to information flow during the cross-institutional referral process. This suggests that important notifications may be “lost in the noise” of a barrage of less urgent notifications that are ultimately impairing information flow, which reflects previous work analyzing EHRs (Murphy, Reis, Sittig, & Singh, 2012; Pretolesi, Motnikar, Till, & Uhl, 2023).

• Increase user control: Another high-severity barrier is that errors may not appear when consultation results fail to be sent to an external provider. This failure represents a severe workflow breakdown and increases the referrer team’s workflow by initiating the search and request task. The consultant team is prevented from successfully completing their task due to a lack of feedback, rather than any of the other potential influencing factors. However, even if errors are presented, the referrer team may not take further action, preventing task completion. This is because when the consultation results fail to be sent within a consultant’s standard workflow, they do not see a clear way to address the issue within the constraints of their work environment. Thus, their task of sending the visit notes to the referrer goes uncompleted, and the referrer does not receive the consultation results.

Our characterizations of barriers addressing the cross-institutional referral loop among US care providers provide insights into configurations of tasks and related work system factors that are difficult or prone to error. These characterizations also prioritize opportunities for redesign at the tool and technology and organizational levels across tasks.

4.4. Limitations and Future Work

We acknowledge limitations to this study. Perhaps the most basic of limitations is that of generalizability, internationally and nationwide. The research study, all clinicians interviewed, and all authors of the paper are embedded in the US healthcare landscape. Further, the clinician sample consisted of practitioners from large institutions; so, results may not be generalizable to all institutions. The sample size was relatively small. However, there was a diverse representation of specialties within the sample across two institutions, which enabled the identification of the most impactful or common barriers related to closing cross-institutional referral loops.

The SEIPS 2.0 framework provides initial insight and characterization of these barriers in terms of causal work system factors. In complex systems, behavior is rarely the result of a single factor or point of failure. Future work could investigate the complex interactions that underlie our initial barrier characterization. Additionally, SEIPS 2.0 does not provide an explicit structure to handle how these barriers interact with dynamic or evolving temporal demands (Caldwell & Garrett, 2011; Caldwell, Garrett, & Boustany, 2010; Heiden & Caldwell, 2018; Jahn & Caldwell, 2018). Future work applying other models explicitly addressing temporal processes, such as information foraging theory or CHIPPER (Caldwell & Garrett, 2011; Heiden & Caldwell, 2018; Jahn & Caldwell, 2018), to provide further insight into additional cross-institutional information flows at other time scales of events. Fine-grained study of consultant-initiated patient information gathering processes may require an expansion of targeted participants to include nurses, medical assistants, and office staff who are often primarily responsible for gathering the patient information. The insight from these groups would also allow for the closing of the referral loop process to be described among healthcare teams and associated roles or responsibilities.

5. Conclusion

The complexity of actual cross-institutional referral processes is distinct from that of an idealized version of the processes as designed. Recognizing and assessing the gaps between design and operational conditions is critical to issues and opportunities for improving the quality of workflow, clinical care, and patient safety. We addressed aspects of this gap using a workflow analysis technique that expounds upon previous depictions and highlights tasks that may be subject to cascading errors occurring in clinical referral processes. Specific barriers were identified and associated with tasks and work-system factors (i.e., person, tasks, tools and technology, organization, external environment) that affect closing the referral loop at different severity levels (i.e., minor, moderate, major, extreme). In essence, this effort integrates a design-focused FMEA-based analysis with a healthcare work system framework template described by SEIPS 2.0.

The referrer’s task of accessing visit notes and the consultants’ process for sending consultation notes were considered in the context of a process flow analysis identifying more than ten potential flow barriers that can be critical to closing the referral loop. Of the barriers that were identified, most related to tools and technology, including challenges or deficiencies in availability, usability, and functionality. Five barriers were characterized as high or very high severity, including consultant beliefs about the scope of recommendations desired by the referrer, and a lack of a clear process or policy for responding to error messages occurring when sending consultation visit notes.

HIT-related barriers included consultation notes not being received by the referrer, as well as a lack of error messages when the consultation note sending process fails, and notifications for critical updates to medical records. Additionally, many clinicians reported that the consultation results and recommendations were not consistently available in their HIT tools, including both HIEs and EHRs. We recommend a process for prioritizing and addressing high-severity barriers with identifiable interventions (e.g., creation of notification customizations, implementation of error messaging, and standard procedures for resolving them) and lower cost and effort requirements.

6. Appendix A

6.1. Consultants’ Care Coordinator Interview Guide

This is an interview with [participant ID] on [date].

Today, we are here to learn from you about the differences between internal and external consultation processes. During this interview, the internal consultation process describes scenarios where both the referrer and consultant are in the same healthcare institution and use the same electronic health record system (EHR). An external or cross-institutional consultation process describes scenarios where the referrer and consultant work in different healthcare institutions and do not use the same EHR.

We would like to know how your workflow differs among these two scenarios, especially for the initial consult. We would like to hear your perspective on provider communication, information technology, documentation, and workflow efficiency.

Characterize Workflow Differences

• 1Describe your typical clinical workflow for initial consults. [Internal]
• 2How does your workflow differ when the consult was ordered from external referrers?

  1. Are there the same number of steps/tasks or documentation for both scenarios?

  2. Describe any other software applications or communication processes that are used to support the initial consult.

• 3What kind of information is needed to fully support the initial consult (i.e., clinical question, medication, referrer note, referrer contact information)?

  1. What are the sources of information (e.g., care team, patients, referrer)?

  2. How do you retrieve information needed? What modes are used to share or communicate patient information (fax, phone calls, EHR, HIE – secure emails, etc.)?

• 4How does the workflow, information flow, or communication differ when patients are referred from an external referrer?

  1. What are the sources of information (e.g., care team, patients, referrer)?

  2. How do you retrieve information needed? What modes are used to share or communicate patient information (fax, phone calls, EHR, HIE, secure emails, etc.)?

Clinical Example...

• 5Can you describe a case(s) where a patient from an external referral required a high-level of care coordination?

  1. Are there any referrals, consults, or patient profiles that are more likely to require higher levels of care coordination, especially with external consultants/specialists?

  2. How well was cross-institutional provider communication supported?

  3. What were facilitators and barriers?

• 6Describe any communication breakdowns with referrers or other specialists both within and outside of your institution?

Impact...

• 7How do the differences among internal and external referrals [to you] impact your care delivery or acceptance of referral? (Do you feel that you can deliver the same quality of care to patients from an external referral).
• 8How do organizational policies affect both internal and external consultations/referrals related communication or information sharing?

  1. Are you familiar with communication and information sharing practices or policies at other medical centers, especially as they relate to cross-institutional physician-dyads (referrers and consultants)?

  2. How has that influenced the workflow here?

  3. Do you feel like you can try new things to improve your work processes? If so, elaborate.

• 9What impact, if any, do health information exchange (HIE) technologies have on your clinical work? [Technologies or software that are designed to help information exchange/sharing or provider communication across healthcare institution.]

  1. How can HIE technologies change to best improve your workflow and quality of care?

• 10In what ways do you think technology improvements can reduce patient burden?
• 11Are there any other aspects of the consultation process that you think we should know?

Thank you for participation!

6.2. Referrers’ Care Coordinator Interview Guide

This is an interview with [participant ID] on [date].

1. At which facility or facilities do you currently practice? How long have you worked at the [facility name]?

2. In what year did you receive the degree, such as M.D., that has enabled you to become a licensed health professional?

3. What is your job title?

4. Specialty?

Today we are here to learn from you about the differences between internal and external referral processes. During this interview, the internal referral process describes scenarios where both the referrer and consultant are in the same healthcare institution and use the same electronic health record system (EHR). An external or cross-institutional referral process describes scenarios where the referrer and consultant work in different healthcare institutions and do not use the same EHR. We would like to know how your workflow differs among these two scenarios, especially for the initial follow-up from consult. We would like to hear your perspective on provider communication, documentation, and workflow efficiency.

Characterize Workflow Differences

• 1Describe your typical clinical workflow for ordering specialty referrals.
• 2How does your workflow differ if the referral is to external specialists?

  1. Are there the same number of steps/tasks or documentation for both scenarios?

  2. Describe any other software applications or communication processes that are used to support the referrals.

• 3Switching to your workflow after the consultation, please describe your workflow when following-up with patients after a specialty referral?

  1. What kind of information is needed to fully support your follow-up with a patient after a consultation (i.e., medication changes, diagnostic plan changes, specialist visit note)?

  2. What are the sources of information (e.g., care team, patients, consultant)?

  3. How do you retrieve information needed? What modes are used to share or communicate patient information (fax, phone calls, EHR, HIE – secure emails, etc.)?

• 4How does the workflow, information flow, or communication differ when patients have visited an external specialist/consultant?

  1. What kind of information is needed to fully support your follow-up with a patient after a consultation (i.e., medication changes, diagnostic plan changes, specialist visit note)?

  2. What are the sources of information (e.g., care team, patients, consultant)?

  3. How do you retrieve information needed? What modes are used to share or communicate patient information (fax, phone calls, EHR, HIE, secure messaging, etc.)?

  4. Do you have to use a system/software to access consultant notes from an external referral?

Clinical Example...

• 5Can you describe a case(s) where following an external consultation a patient required a high-level of care coordination?

  1. Are there any referrals or patient profiles that are more likely to require higher levels of care coordination, especially with external consultants/specialists?

  2. How well was cross-institutional provider communication supported?

  3. What were facilitators and barriers?

• 6Describe any communication breakdowns with specialists both within and outside of your institution?

Impact...

• 7How do the differences among internal and external referrals impact your choice of consultant? (In other words, is one easier than the other and does that influence your choice to make internal/external referrals).
• 8How do organizational policies affect both internal and external referrals related communication or information sharing?

  1. Are you familiar with communication and information sharing practices or policies at other medical centers, especially as they relate to cross-institutional physician-dyads (referrers and consultants)?

  2. Has that influenced the workflow here? How?

  3. Do you feel like you can try new things to improve your work processes?

• 9What impact, if any, do health information exchange (HIE) technologies have on your clinical work?

  1. How can HIE technologies change to best improve your workflow and quality of care?

• 10In what ways do you think technology improvements reduce patient burden?
• 11Are there any other aspects of follow-up after an internal or external process that you think we should know?

Thank you for participation!

Appendix B

Table B.1.

Barriers that affect the cross-institutional referral process broken down by workflow step, SEIPS work system factor, likelihood rating, and severity rating, including representative quotes.

Workflow Step	Barrier Description	SEIPS Factor	Likelihood Rating	Severity Rating	Representative Quote
CA-1: Consultant prepares for the appointment	Difficulty finding patient records in the EHR	Tools & Technology	5	2	“And then, we want Google-like search functionality to actually find the transthoracic echo that we were looking for, or the cardiac cath data, or whatever. So, we don’t want to hand go through each one of those individually. We want some sort of search function and we want a repository that has all that data in it so that we can run that search function against it” (R3).
	Insufficient time to review patient records before the appointment	Task; Organization	5	1	“From my end, basically, so I don’t typically have enough time to prepare for the next [day].... The thing is, most days there’ll be few no shows here or there. So I’m able to catch up. But on the days like yesterday, I had everybody show up. I was struggling to keep ahead” (C8).
	Paper-based intake forms	Tools & Technology	5	1	“Well, there are many opportunities for improvement. For example, I don’t have a sheet here, but when a new patient comes to us, they fill out manually this patient intake form. Who’s your referring doctor? What are your medicines? What are your family history and all that. That has to be filled out by hand. Nope, there should be a website that patient can go into, assuming that they have the electronic ability to do that, and they can fill out all of that information ahead of time” (C11).
	Lack of records at the time of the appointment	Task; Tools & Technology	3	2	“It could require an additional visit. It could require additional record gathering, me looking at the records again, issuing updated notes, deciding at a later time that I need to do additional testing, scheduled tests, et cetera” (C11).
	Scanned records are long, contain irrelevant information, and are not easily navigable	Tools & Technology	3	2	“[T]he biggest hurdle for me is that even when I have good records a lot of times the referring providers will send you, in an effort to be complete, will send you 100 pages of records and it can be all kinds of junk, you know. Like 100 pages that was not relevant. A lot of it’s like random notes from the nurse... And, you have to spend a lot of time sifting through that and it’s very difficult for me personally to sift through that in electronic format when it’s all scanned in as one document. So, then I end up trying to print 100 pages, sort through and find the 10 pages that are relevant. And, so it’s time consuming and wasteful and it’s not organized. The data’s not organized in a way that’s useful and so it is quite painful” (C1).
	Patient records are printed for review	Task; Tools & Technology	2	2	“I print them out from the [EHR], so we waste a lot of paper that way, but I can’t flip through because it’s not user friendly to click through a fax, each individual page, so I print it out. It’s a whole lot easier for me to flip through by hand, and as I’m doing that, I’m doing what I did, getting my note together like I would for the internal referral” (C5). “Once I review those records, I often will print them out. If they are not in [the EHR] because it’s just again hard to view them and manage them all as one pdf. So I often will print out records that are from outside health systems” (C1).
CA-2: Consultant evaluates the patient	Limited time is provided for the appointment	Organization	5	1	“It depends on the providers. For me, I have 30 minutes for a new patient and 15 minutes for a return patient” (C8).
	Patients may not be able to provide accurate and useful (and needed) medical information and history	Task; Tools & Technology; Organization	4	1	“And then just, just missed information. Patients just aren’t necessarily the best historians and if you don’t have all the data and you, you know, you don’t necessarily come to the correct conclusion. And so, missing that data [can lead to] massive delays can just lead to the wrong diagnosis” (C1).
	Language barrier between patients and clinician	Person; Task	1	4	“When you have a Hakha Chin kid, you got to find a Hakha Chin interpreter... which we usually can... We have now two interpretative services. So sometimes... like last week, I saw a colleague had a Mayan patient, and couldn’t find an Mayan interpreter in one interpretation service, so we had to go to another” (C6).
	Virtual visits are more complex due to lack of familiarity and number of appointments	Person; Task	1	4	“Because all this stuff has been thrust on us in a very short amount of time, and we’re just really learning on the fly. A lot of the stuff. Now we’ve got a new system. Now we have a new virtual system... And so, I have 10 patients scheduled for my Friday clinic next week, and I don’t know how we’re going to get through them all, but we’ll learn” (C6).
CA-3: Consultant documents visit notes	Having to review notes written by residents/fellows	Task; Organization	4	1	“So, it’s not just you writing the note, you’re relying on [the resident or fellow] to write the note. And that can sometimes be inaccurate, and that can sometimes get... That’s an extra layer of complexity, for which can screw up communication” (C6).
	Insufficient time to enter notes (or other appointment outcomes) into the EHR immediately (during or immediately after appointment	Tools & Technology; Organization	4	1	“If you’re putting in orders outside or after the encounter and you have any questions for the patient that may come up during or completing the artist’s templates, but the patient’s not there. What do you do then?” (C3) “Right, so if it’s someone who’s a little bit complicated or took a little bit longer, I may not have time to do my note, my dictation. And so in these sorts of situations, I may end up at the end of the day, I may be stuck with doing 10 notes. At the end of the day, after I’ve seen all the patients” (C8)
	Cultural norm and misconception and lack of supporting policies to encourage consultants to share all information (belief that referrers do not want recommendations beyond the scope of the referral order)	Organization	3	2	“Honestly, I often do not ask the other doctor to do stuff that they referred. My impression has been that they don’t want me to, they just want me to take care of the problem. When they sent the patient, they’re not interested in advice, at least that’s my impression, and it may not be true for everyone” (C5).
	Technical failures such as phones failing in the middle of dictation	Tools & Technology	1	1	“Sometimes, like when we were dictating, sometimes the phones would go down” (C6).
CA-4: Consultant orders further tests	Errors when ordering labs not behaving in useful or expected ways	Tools & Technology	2	2	“So, the way it troubles you after that episode is, it’s another five minutes of the system telling you that you cannot order, but it doesn’t even let you pass that step to go into a patient note, even for you to deal with it after the clinic visits. If you ordered 10 labs and out of the 10, five couldn’t be ordered, it will give you that error message for five. So it just doesn’t let you cross that step to do other stuff with a patient” (C2).
	Tests may be reordered in the middle of trying to complete them	Task; Tools & Technology	2	1	“I think that if we don’t have the same EMR, they can reorder. Say we’ve already completed 75% of the workup and then they reorder it, it’s a waste and it’s unfortunate” (R2).
	Disparities in codes for the same labs or tests across different external labs	Organization; External Environment	2	1	“The problem with that is, if I have a lab immunoglobulin profile or vitamin A level, or, I’m failing to remember, a few other rare disease tests, I want to order them, the patient wants to get it elsewhere, but the code is not recognized by the outside lab” (C2).
CL-0: Referrer team requests or searches for visit notes	Lack of a secure, asynchronous tool for communicating with external referrers	Tools & Technology	5	1	“They only use [the asynchronous communication tool] for internal communication. So if somebody in Fort Wayne has a question, I can’t use it to communicate with them because they’re not on our kind of platform. So it makes it easy to communicate. We’re using a personal device, it’s a secure communication, but it only works for internal providers. You can’t use it outside of our systems” (C3).
	Inability to access external EHRs	Tools & Technology; Organization; External Environment	5	1	“I think everybody in the state of Indiana ought to have a [specific EHR] account, which would require them to register for it, but also to check it regularly. But if everybody in the state had an [EHR] account, any report I sent, it’s gone, it’s there immediately. For example, when I’m sending communication back to a referring physician, the primary care doc will show up there. But if I click that, it will immediately tell me John Smith and it says no external access, which means they don’t have [specific EHR]. Well then I’ve got to go to this other process of create provider letter and then if they’re John Smith, then it’s the same process I described before of figuring out which John Smith it is. So put everybody in the state of Indiana on [the specific EHR]” (C11).
	Patients who are unable to provide accurate medical information	Person	4	2	“[B]ut there’s a fair number of patients who are cognitively impacted in one way or the other, so can’t consistently, reliably tell us what the consultant said, or recommended, or that sort of thing. And we wouldn’t act on... it helps us to know through the patient’s version, but we’re not liable to actually act on ordering tests or that sort of thing without direct communication from the clinician” (R3).
	EHR is difficult to navigate	Tools & Technology	4	1	“Oftentimes, [the EHR] is not intuitive enough. A lot of people complain about that. It’s... And they change, now, the flow, oftentimes, of how you get labs, and how you get outside labs. And every so often there’s a change that you have to struggle through” (C6).
	Time and staff requirements of communicating and coordinating with external consultants	Tools & Technology; Organization; External Environment	4	1	“And those have to be forwarded to us, and then they get scanned into our records system, and that can be anywhere from a semi smooth process to a completely impossible process that requires multiple other steps, me calling the physician’s office, my referral coordinators calling the referring office, et cetera. So it can be a much less smooth and much less complete process” (C11).
	HIE and EHR tools not consistently containing needed information (visit notes)	Tools & Technology	3	2	“We may use the health information exchange, which contains some limited amounts of text and laboratory from some but not all other institutions, and from outside institutions” (C10). “[An HIE tool] externally in the state of Indiana has very limited applications because many of the major systems do not participate in [the HIE tool], there’s no information there. I checked [the HIE tool] but it rarely provides me any meaningful information, yep, bi-level especially, and it especially provides no imaging, there’s no imaging there” (C10).
	HIE tools are difficult to navigate	Tools & Technology	3	2	“[HIE tool]’s also not the most user-friendly thing in the world, it’s gotten better than it used to be, but it’s not organized at all, so if you’re looking for a specific piece of information, you’re stuck scrolling through everything to get to it” (C5). "But [HIE tool] was extremely difficult to navigate. So like on [HIE tool], I don’t know that. The document type may just be like clinical documents. I have no clue what it is” (C8)
	Institution-centered mindset regarding information flow	Organization	3	1	“And so that institutional policy to keep the texting within the institution inhibits the ability to communicate with those outside makes us much less efficient... It’s institutional mindset at the level of the senior administrators of the hospitals from practice plans or the advice from their legal consultants.... For me personally as somebody who has a national reputation, it makes it incredibly difficult” (C10).
	Logging into multiple systems to gather information	Task; Tools & Technology	3	1	“It would be nice to have some kind of portal that would include multiple systems so that they didn’t have to have an [healthcare institution 1] portal and a [healthcare institution 2] portal and a [healthcare institution 3] portal, because then they have three different passwords and they can’t remember where they’re logging into anymore. So, again, kind of that standardization where I don’t have to look, oh, [healthcare institution 2] has their own thing and [healthcare institution 3] has their own thing” (R4).
	Constraints around accessing the EHR (including constraints on physical location)	Organization	1	3	“The [healthcare institution]’s tougher because the [healthcare institution], at least traditionally, hasn’t had an external website, so you actually had to be on their premises to log in and look up a patient. And these aren’t necessarily routine. I mean, there are occasional patients who have almost all their records at [healthcare institution], and it’s tremendously hard to take care of a patient you know nothing about” (C8).
	State and federal regulations for patient privacy preventing easy access to records	External Environment	1	3	“I think the reason we can’t see it is because of HIPAA and all these other regulations. I mean, I think if nationally, everybody opened up their EMR, then it really wouldn’t matter where people went, and they could get care wherever was easiest. But so, I mean, I think it’s completely dependent on policy” (R5). “I can’t see patients across state lines, I can’t see people where they are, they have to come to me, the major impediment” (C10).
	Lack of documentation from phone calls between referrer and consultant teams	Task; Tools & Technology	1	2	“And when I call this doctor, this radiation oncologist by the telephone, that’s not documented, there’s nothing recorded. If I forget to say something or he may say, oh, I thought you said this. And there’s no documentation of what was really said... And that you put a note of it in the EMR and you say, I spoke to him and this is what I said. But I think it’s an extra step. There’s something that might get missed. It’s not as robust as a real direct communication and things like that. So certainly outside or externally, there’s a little bit that suffers because I think when we have to do that” (C3).
CL-1: Consultant sends visit notes to the referrer	Lack of a secure, asynchronous tool for communicating with external referrers	Tools & Technology	5	1	“They only use [the asynchronous communication tool] for internal communication. So if somebody in Fort Wayne has a question, I can’t use it to communicate with them because they’re not on our kind of platform. So it makes it easy to communicate. We’re using a personal device, it’s a secure communication, but it only works for internal providers. You can’t use it outside of our systems” (C3). “I think just allowing docs and nurses to talk more freely with each other you know would, would be ideal, but there’s not really a good format for doing that. Then nothing’s been decided upon and so, ultimately, what happens is people communicate in all these you know patched together formats” (R1).
	Contact lists for external referrers are poorly structured	Tools & Technology	5	1	“Then for an external provider, let’s say that they were sent by Dr. John Smith. So I go into my list of John Smiths in the electronic record to generate a letter to them. So how many John Smiths do you think are in? And this has been a big, big problem for me. In some cases, I actually have to go in the list, select a number or select a name and look at the area code on their fax to say that John Smith is in [city], where the patient came from versus it’s [area code], so that’s a John Smith in [other city]. This is a very arduous process which can lead to great frustration and the ultimate frustration is that the letter never gets back to the referring physician” (C11).
	Institution-centered mindset regarding information flow	Organization	4	1	“And so that institutional policy to keep the texting within the institution inhibits the ability to communicate with those outside makes us much less efficient... It’s institutional mindset at the level of the senior administrators of the hospitals from practice plans or the advice from their legal consultants.... For me personally as somebody who has a national reputation, it makes it incredibly difficult” (C10). “But there are some providers who don’t have privileges at the facility. And therefore, none of their patient information is in. So that’s the only institutional policy that strikes me as having an influence on whether we have the information or not” (C9).
	No error message is displayed if notes fail to send	Tools & Technology	2	4	“So for the external, even in that system, if the address is [wrong] or if the fax number is wrong or if something changed, then there may be a lapse. And I may be thinking I sent it and they didn’t get it. So there’s no feedback loop” (C2).
	Lack of a confirmation if notes were received	Tools & Technology	2	4	“I hit the button and then I’m like, “Do they ever get it? I have no idea.” Really no idea. We did a project a few years ago where we tried to call referring docs and say, “Hey, we sent you a note, did you get it?” It wasn’t a very successful project, not because that rate was really low, just because it’s hard to get a hold of people. It’s hard to talk to the person that would actually know if they got the note” (C4).
	Contact information for referrers may not be available	Tools & Technology	2	3	“Today I told a patient... I said I’m gonna change your blood pressure medicine, but I want to call your primary care doctor and talk to him about it to make sure he’s okay... with me doing that. And then, call me back and let me know. Now why should I have to, why should she have to do that? She had no idea what blood pressure medicine probably I’m trying to prescribe, but I don’t have a good way to get your primary care doctor” (C1).
	Contact information for referrers may be inaccurate	Tools & Technology	2	3	“All different providers and keeping that huge list of referring providers up to date and current, and being able to find them when you need to is probably the biggest barrier” (C7). “Just the fact that you don’t have information, you can’t get information to them because the fax number is wrong or the computer has the wrong listing or that person retired a year ago, and they’ve switched providers” (C7).
	Time and staff requirements of communicating and coordinating with external consultants	Tools & Technology; Organization; External Environment	2	3	“So, when we’re getting a consultant involved, there is quite a bit of high-level care coordination, especially in the nursing home setting because we’re trying to avoid a hospitalization” (R2). “And those have to be forwarded to us, and then they get scanned into our records system, and that can be anywhere from a semi smooth process to a completely impossible process that requires multiple other steps, me calling the physician’s office, my referral coordinators calling the referring office, et cetera. So it can be a much less smooth and much less complete process” (C11).
	Lack of enforced policy supporting timely sending of visit notes	Organization	2	2	“I know that we have set some soft targets of getting notes to referrings within a week. So we try to get our documentation done within a week. I know in some of our outlying clinics where we have trainees, they want to get them within a month and it was a very, very soft target” (C3).
	State and federal regulations for patient privacy preventing easy access to records	External Environment	1	3	“Alternatively, in the opposite direction, there are times where we know something very important to the patient. They were allergic to something, or there were procedures that we did the cardiac cath, that sort of thing, and the outside group wouldn’t know that unless somebody handed them a discharge summary, for example. But, that goes against the rules. So, you got to go through med records” (R3).
	Actions might not be taken in the case of an error	Person; Organization	1	1	“I’m ashamed to say I don’t do anything. I don’t know what to do. I don’t know how to find that position and things. If they call and they say, “Hey, we never got it.” My office can print it out and manually fax it and things like that” (C3).
CL-2: System transmits visit notes	Different health systems’ EHRs don’t talk to each other	Tools & Technology; Organization; External Environment	5	2	“With external, most of the time, it’s not going to be available immediately in our EHR because they still don’t talk to each other. So it’ll all be reliant on that faxed information that’s sent over into the chart” (C7). “But as far as I know, at least [with] [EHR 1]. I know it doesn’t talk to other [EHR 1] systems. It certainly doesn’t talk to [EHR 2]. I know [EHR 2] systems can talk to one another, but [EHR 1] doesn’t really talk to other systems” (C3).
	Faxing introduces redundant printing and scanning and is prone to errors	Task; Tools & Technology	5	1	“Well, they never saw a four volume bound record. Particularly if you’re a transplant patient, it’s like, oh God. This patients got five volumes of their chart. So a hundred page fax. I mean, that’s a reality, but why is it a fax? Why doesn’t it just go straight computer to computer?” (C11). “And I feel like we could have a better workflow institutionally and I don’t know if there’s a way to set Epic. There should be a way to set up Epic to do this where when we put in an external order, somehow it faxes it to the right number automatically. It seems sort of silly to rely on humans to print out the referral, have an MA grab it off the printer, fax it on another machine” (R6).
CL-3: Referrer teams processes the visit notes	Too many meaningless notifications	Tools & Technology	4	1	“[T]hey may or may not send me the note, but honestly, it’s kind of lot of glut in my inbox. So, it’s okay if they don’t send it cause I know I’m gonna go back and look for it anyway” (R1). “The bad part of it is that we get notified a lot and on patients that aren’t necessarily ours. For example, for some reason I’m assigned to a lot of patients that are pregnant, but I don’t do OB, but every time one of them has a baby, I get a notification that they were in hospital and had a baby. We get so many inbox messages already. Those are ones I don’t need to get” (R6)
	EHR is difficult to navigate	Tools & Technology	3	2	“If it’s external, again, it’s kind of buried. At Community I usually can see if they actually made the visit and there’s a note. I can see it, but I can’t see it if the patient had a visit, then they no showed and I can usually see the testing that’s ordered, but I can’t really see if something is scheduled and they haven’t had the testing done yet. At least I don’t know how” (R6).
	Consultant’s visit notes are not available to the referrer team on a timely basis; from the referrer’s perspective, consultants are not proactively sending the visit notes	Task; Tools & Technology	2	2	“I think the biggest barriers are probably the private docs that patients are seeing real frequently and you don’t always get the notes or you’ll get the notes three months later” (R6). “For external it’s hit or miss, some people are great about sending a visit note back, and if they send a visit note back, our scanners will grab a hold of that, scan it into the chart and forward it to me to read, depending on who gets it, it may just go into the chart somewhere and I may not ever see it until I see the patient back and go looking for it. Then sometimes I never get a note and we have to try really hard to get them and that’s kind of frustrating” (R4).
	Lack of a notification for important changes to a patient’s medical record	Tools & Technology	2	2	“There have been a few instances where someone went somewhere outside of network to get labs done, and the labs showed a life-threatening result, and I unfortunately didn’t see that result until four or five days later. I think that’s where the biggest problem comes in with that whole process, there’s no alert” (R4).
CL-4: Referrer reviews visit notes	Poorly written visit note	Person; Task			“When a consultant sees somebody once and then assumes that I’m just gonna take over again, but doesn’t really like leave any bit of advice... You know if they could just sort of leave little recommendations about how they do things that would you know would be helpful and I don’t know that consultant’s always think about the long-term in patients and, and how primary care’s gonna manage people once they aren’t seeing them anymore... I’m just saying like some, every now and then I’ll have a consultant sort of see somebody once and then sign off. They’re like okay I saw them and they definitely have this disease and here’s a drug to treat it. I just need a little more information... I usually don’t get notes that are directed at me” (R1).

Table B.2.

The number of barriers associated with each SEIPS factor across different severity levels for all 48 barriers identified. Because more than one SEIPS factor could be associated with each barrier, the algebraic sum on the columns may not add up to the given total. However, the total indicates the true number of barriers rated at each level of severity.

SEIPS Factor	Severity Levels				Total
	Extreme	Major	Moderate	Minor
Person	2	0	1	2	5
Task	2	0	4	7	13
Tools and technology	2	3	13	13	31
Organization	0	2	4	10	16
Internal environment	0	0	0	0	0
External environment	0	3	2	2	7
Total	4	6	16	21

Table B.3.

The number of barriers associated with each SEIPS factor across different likelihood levels for all 48 barriers identified. Because more than one SEIPS factor could be associated with each barrier, the algebraic sum on the columns may not add up to the given total. However, the total indicates the true number of barriers rated at each level of severity.

SEIPS Factor	Likelihood Ratings					Total
	Very Unlikely	Unlikely	Neutral	Likely	Very Likely
Person	3	0	1	1	0	5
Task	3	3	3	2	2	13
Tools & technology	2	10	6	5	4	32
Organization	2	3	2	5	8	16
Internal environment	0	0	0	0	0	0
External environment	2	2	0	1	2	7
Total	12	20	12	14	16