Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Nov 8.
Published in final edited form as: Circ Cardiovasc Qual Outcomes. 2016 Nov 8;9(6):670–678. doi: 10.1161/CIRCOUTCOMES.116.003041

Characterizing Teamwork in Cardiovascular Care Outcomes: A Network Analytics Approach

Matthew B Carson 1, Denise M Scholtens 1, Conor N Frailey 1, Stephanie J Gravenor 2, Emilie S Powell 2, Amy Y Wang 1,3, Gayle Shier Kricke 1, Faraz S Ahmad 1,4, R Kannan Mutharasan 4, Nicholas D Soulakis 1
PMCID: PMC5217475  NIHMSID: NIHMS822664  PMID: 28051772

Abstract

Background

The nature of teamwork in healthcare is complex and interdisciplinary, and provider collaboration based on shared patient encounters is crucial to its success. Characterizing the intensity of working relationships with risk adjusted patient outcomes supplies insight into provider interactions in a hospital environment.

Methods and Results

We extracted four years of patient, provider, and activity data for encounters in an inpatient cardiology unit from Northwestern Medicine’s Enterprise Data Warehouse. We then created a provider-patient network to identify healthcare providers who jointly participated in patient encounters and calculated satisfaction rates for provider-provider pairs. We demonstrated the application of a novel parameter, the Shared Positive Outcome Ratio (SPOR), an objective composite measure that quantifies a given pair’s concentration of positive outcomes over a set of shared patients. We compared an observed collaboration network of 334 providers and 3,453 relationships to 1,000 networks with SPOR scores based on randomized outcomes and found 188 collaborative relationships between pairs of providers that showed significantly higher than expected patient satisfaction ratings. A group of 22 providers performed exceptionally in terms of patient satisfaction. Our results indicate high variability in collaboration scores across the network and highlight our ability to identify relationships with both higher and lower than expected scores across a set of shared patient encounters.

Conclusions

Satisfaction rates appear to vary across different teams of providers. Team collaboration can be quantified using a composite measure of collaboration across provider pairs. Tracking provider pair outcomes over a sufficient set of shared encounters may inform quality improvement strategies, such as optimizing team staffing, identifying characteristics and practices of high-performing teams, developing evidence-based team guidelines, and redesigning inpatient care processes.

Keywords: outcome, statistical analysis, statistical model, big data, network analytics


Interdisciplinary care has become a cornerstone of health care delivery in an increasingly complex healthcare system. Prominent U. S. organizations, including the Agency for Healthcare Research and Quality (AHRQ), Centers for Medicare and Medicaid Services (CMS), and the National Academy of Medicine, prioritize and promote care coordination as a key strategy for improving healthcare quality1. However, defining and measuring the quality and impact of care coordination remains challenging2. Measuring and improving care coordination may be particularly important in cardiovascular disorders, especially heart failure, given the large population burden of disease, frequent comorbidity, and complexity of care.

Numerous quality measures spanning multiple domains have been developed to assess care coordination among interdisciplinary team members2. These measures use collection methods such as surveys, claims data, and electronic health record (EHR) data37. In particular, the EHR offers a rich and routinely collected data source for assessing collaboration. However, assessing collaboration using EHR data is difficult due to the high complexity and volume of data. There are also no standard methods for using patient data to identify relationships among providers.

Social network analysis has been applied broadly in multiple fields for many years and only recently in healthcare6, 8. A network approach can be particularly helpful for measuring care coordination because it provides a scalable framework for considering the interdependency of encounters, the health care providers involved, and the effects these relationships have on each entity in the system. This context is necessary to examine the relationships formed between thousands of providers caring for patients over time and potentially across multiple health care settings.

The widespread adoption of EHRs has fueled interest in “big data” approaches in health care9. Extensive data on individual actions exist within health system data warehouses. Using these data, researchers can construct networks of patients and the providers who care for them during a hospital stay1013. We previously demonstrated that provider networks for heart failure patients can be derived from EHR data and highlighted methods for visualizing collaborations between providers who share common patients14. In this study, we extended our previous work by developing a scoring system that quantifies the quality of collaboration by associating provider networks with specified outcomes.

Here we demonstrate the application of a novel, graph-based method for computing the Shared Positive Outcome Ratio (SPOR), a measure that quantifies the strength of a patient-sharing relationship by calculating the concentration of risk-adjusted positive outcomes for all pairs of providers over a set of shared patient encounters. This study is a first step toward our ongoing effort to characterize and facilitate improved provider collaboration through a flexible platform that measures the strength and dynamics of working relationships between health providers.

Methods

Cohort Description

Northwestern Memorial Hospital (NMH) is a large, urban, academic facility with an annual volume of over 51,000 inpatient admissions in 201615. We collected retrospective EHR data from all patients who were admitted to the NMH cardiology unit from the Emergency Department for observation or inpatient care between January 1, 2012 and February 4, 2016. The cardiology unit has 36 beds (24 inpatient and 12 observation) and cares for approximately 4,600 patients per year, with an average length of stay of 2.6 days. On a typical day the unit is staffed by three attending physicians, three or four cardiology fellows, and four or five mid-level providers (nurse practitioners and physician assistants). Northwestern University’s Institutional Review Board approved this study with a waiver of patients’ informed consent.

Outcomes of Interest

The primary outcome used in this study was the likelihood to recommend (LTR) as reported by the Press Ganey Associates, Inc.© Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) Hospital Survey, which measures patient experience. We selected this metric because it is becoming increasingly important as healthcare organizations move to become more patient-focused16. In addition, it has been shown to be highly correlated with patient perception of teamwork or collaboration among providers as measured by the HCAHPS instrument in both inpatient (NMH Analytics Team, unpublished data, 2016) and outpatient17, 18 environments. Thus we reasoned that teamwork as measured by the SPOR parameter could be associated with patient experience. The response rate for the period coinciding with our data set was approximately 19%. The LTR is measured on a 5-point Likert scale19. We defined a positive outcome as an LTR score of 5/5 (highly likely to recommend or LTR+), the “top box” score as defined by the Centers for Medicare and Medicaid Services (CMS) scoring system, and a negative outcome as a score of 4/5 or below (not highly likely to recommend or LTR−). It is important to note that changing the definitions of positive and negative outcomes may be quite reasonable depending on the application and results may be affected. We used the ESI-level (Emergency Severity Index) (http://www.esitriage.org) for risk-adjustment modeling of the encounters. This tool is designed to capture information on both patient acuity and the number of hospital resources required to treat a patient20.

Data Extraction and Initial Graph Construction

Our data extraction and management pipeline is illustrated in Figure 1. NMH utilizes the Cerner EHR system to collect all inpatient clinical data. This data was transferred via extract, transform, and load (ETL) scripts to the Northwestern Medicine Enterprise Data Warehouse (NM EDW) for electronic health record data21 and housed in operational data stores (ODSs). We used Microsoft SQL Server Management Studio22 and the T-SQL procedural language to export raw data from the NM EDW to comma-separated value (.csv) files. We loaded a cleaned version of this extracted data set into a Neo4j23 graph database, which served as a repository and a query engine for our analysis. We accessed and updated data in this repository using Cypher (Neo4j’s native query language), Python24, and the Py2neo library25. We created and evaluated networks using Python’s NetworkX package26. R27 was used to perform statistical analysis and calculate risk-adjustment factors.

Figure 1.

Figure 1

The data management pipeline used in this study.

For each encounter, we extracted associated properties, a list of all healthcare providers who performed clinical activities, a patient acuity level, and an encounter outcome. Next, we created two types of networks: 1) provider-encounter and 2) provider collaboration. The provider-encounter network was used to identify providers who shared encounters and to characterize collaboration for each pair of providers over their set of shared encounters using the SPOR metric. After collecting associated outcomes and acuity values for each encounter, we used a logistic regression model to risk-adjust outcomes and updated the provider-encounter network to reflect these adjustments. Subsequently, we created a provider collaboration network with providers as nodes and connections between them signifying that two providers share at least x patient encounters. The network was then further modified as described below to incorporate information from the shared positive outcome ratio (SPOR).

Using the Shared Positive Outcome Ratio (SPOR) to Characterize Collaboration

The SPOR is a measure that quantifies the proportion of specific, shared positive outcomes between a pair providers versus shared outcomes with other providers. It measures the strength of an encounter-sharing relationship and helps determine the number of good outcomes two providers achieve when working together versus when they work with any other provider. This method is designed to handle any binary outcome (i.e., positive/negative) and additional variables could be naturally included in the logistic regression model that is used as the basis for risk adjustment. However, choosing meaningful outcomes and associated risk adjustment factors requires proper domain knowledge since the quality of the results depends on the relationship between the outcomes and the data set.

The outcome of interest selected in this demonstration project was patient satisfaction. Prior to calculating the SPOR metric, we weighted the contribution of each encounter outcome (highly likely to recommend Northwestern Medicine to others versus not highly likely to recommend) using patient acuity as an adjustment factor and a corresponding set of probabilities of the encounter outcomes given the acuity level. Logistic regression was used to confirm associations of acuity with the outcomes and to estimate probabilities of the outcomes given the data. Adjusting the outcomes in this manner generously rewards unexpectedly good outcomes, heavily penalizes unexpectedly bad outcomes, and gives smaller rewards and penalties for expected outcomes (see Supplemental Material, Equation 5). The purpose of the risk adjustment is to consider the implicit variability in the attributes of encounters shared between providers.

The SPOR value for a pair of providers j and j’ can be defined as follows:

SPORj,j'=AjAj,ri(yi)AjAj,ri(yi)AjAj,E(rix)AjAj,E(rix) (1)

where Aj and Aj’ are the sets of patient encounters involving providers j and j’, respectively, ri is the risk-adjusted outcome, x is a set of baseline covariates, yi is the encounter outcome ([0,1]), and E(ri x) is the expected outcome (see Supplemental Material, Equation 4). The denominator, which is similar to the Jaccard index28, measures the prevalence of encounter sharing between two providers without considering outcome. The numerator reveals the ratio of positive to negative encounter outcomes that two providers share relative to the positive to negative outcome ratio for all encounters involving either provider. In essence, the SPOR is a summary of observed versus expected risk-adjusted outcomes for encounters shared by two providers. An illustrated example is shown in Figure 2.

Figure 2.

Figure 2

The Shared Positive Outcome Ratio, or SPOR, weights relationships according to the relative success of a provider pair. The SPOR answers this question: ‘How many more positives outcomes do these providers attain when they collaborate versus when they collaborate with other providers?’ In this example, 30 patient encounters are shown. Some patients report a positive outcome (highly likely to recommend Northwestern Medicine to others, red), while others report a negative outcome (not highly likely to recommend, black). Provider 1 (P1) interacts with 20 of these patients and Provider 2 (P2) interacts with an overlapping 20 patients. Providers 1 and 2 share 10 patient encounters with 9/10 reporting a positive outcome. If the rate of satisfaction were the same inside the overlap as it is outside of the overlap, the SPOR would be 1, which is the expected value. In this example, however, both providers have greater success when working together (i.e., inside the overlap) and the SPOR value is greater than 1. This metric was calculated for each pair of providers in the network.

Incorporating the SPOR Metric into the Network Analysis

Finally, we calculated the SPOR value for each pairwise relationship and added this as the edge weight to the relationships in the provider collaboration network. We created five separate collaboration networks, each with a different threshold for the number of encounters required to constitute a collaborative relationship. The rationale behind testing these various thresholds was to find a balance that would filter “emerging” relationships that were based on few encounters and thus strongly affected by each new shared encounter outcome, while allowing potentially interesting relationships to remain.

To test for significantly high-scoring collaborations in the real provider network, we created a copy of the provider-encounter network and randomly assigned risk-adjusted outcomes to each encounter. This process was repeated 1000 times to create a set of “random” collaboration networks. The SPOR value of each relationship in the real network was compared to that of each corresponding relationship in each random network. The significance of a given SPOR score in the real network was determined by the frequency that the value exceeded its counterpart in the corresponding random networks. The resulting p-value was used to define extreme high (p ≤ 0.05) and low (p ≥ 0.95) SPOR values. Subsequently, we categorized providers who participated in the largest number of these extreme-valued collaborations into high- and low-SPOR groups. Providers were placed in one of these groups if they had 1) more than 10 collaborations involving more than 6 encounters, and 2) more than 10% of their total collaborative interactions in the network with a p ≤ 0.05 (“high SPOR”) or p ≥ 0.95 (“low SPOR”).

Final Data Set Description

The final cleaned data set included 1,104 encounters, each with an associated likelihood to recommend (LTR) score. There were 777 encounters for which the patient indicated that they were highly likely to recommend (LTR+) and 327 encounters resulting in the patient being not highly to recommend (LTR−). Encounters were identified as either Observation (58%) or Inpatient (42%). We identified 1,474 providers, each holding one of five general positions (Physician (Hospitalist, Specialist, and Referring), Physician Assistant, Advanced Practice Clinician, Resident/Fellow, or Pilot Nurse). Our data set included 2,190 unique activity types, which we grouped into 17 categories. Each activity was either a “note” (includes forms) or an ”order”. Each provider performed at least one activity during each encounter with which they were associated. The activity type and the number of actions performed were not considered when calculating SPOR values. The total number of individual provider actions was 162,644. The Emergency Department was the admission source of all encounters.

Results

Data Set Statistics

The full provider-encounter network included 2,578 nodes (1,474 providers and 1,104 encounters) and 16,443 directed edges. Each edge pointed from a provider node to an encounter node, indicating that the provider performed one or more actions during the encounter. The encounter-level descriptive statistics for our data set are shown in Table 1. Though highly variable, the average cardiology unit patient had a total length of stay of 62 hours (including time in the Emergency Department) and was assigned acuity level 2 (”Emergent”). The average encounter involved the work of 53 providers, each of whom performed six actions.

Table 1.

Encounter-level statistics for our data set.

Total LoS (hrs.) Activity Count Action Count Provider Count
Minimum 4 40 56 15
1st Quart 25 98 162 31
Median 34 115 216.5 40
Mean 61.9 131.6 317.4 53.4
3rd Quart 68 148 357 60
Max 1751 563 5368 610
St. Dev. 85.8 54.4 315.2 41.6

Descriptive statistics showing length of stay (LoS) in hours, activity count T(the number of times an activity type occurred), action count (the number of activity instances or provider actions), and the number of providers who performed at least one activity during the encounter.

Collaboration Networks

We analyzed significantly high and low SPOR values for five provider collaboration networks with different shared encounter thresholds. We observed that the distribution of SPOR values narrowed as more shared encounters were required to define a collaboration (Figure 3). In general, there was lower variance among SPOR values for networks based on higher thresholds (Table 2). The volatility of SPOR values in the low-threshold networks motivated setting a minimum requirement for shared encounters. Ideally, a chosen threshold would create the relative stability of an approximately normal distribution while not removing potentially interesting relationships from the network. Based on our threshold analysis and discussions with team clinicians, we performed all further analysis on the collaboration network with ≥ 6 shared encounters between a pair of providers defining a collaborative relationship. This threshold analysis should be performed for each data set analyzed using our framework. The choice of threshold should be study-dependent and SPOR results may be affected.

Figure 3.

Figure 3

SPOR distribution densities. The effect of five shared patient threshold values on the density of the SPOR value distributions in the collaboration network is shown. The SPOR value is in log2 form. The distribution narrows as the threshold is increased, revealing the trade off between the relative stability of an approximately normal distribution and the inclusion of potentially interesting relationships based on fewer shared encounters.

Table 2.

Provider collaboration network statistics.

Collaboration
Threshold
≥ 2
encounters
≥ 4
encounters
≥ 6
encounters
≥ 8
encounters
≥ 10
encounters
# providers 1,005 558 334 209 134
# collaborations 28,707 7,814 3,453 1,936 1,235
SPORs with p ≤ 0.05 1,202
(4.2%)
377 (4.8%) 188 (5.4%) 101 (5.2%) 61 (4.9%)
SPORs with p ≥ 0.95 1,346
(4.7%)
383 (4.9%) 167 (4.8%) 92 (4.8%) 54 (4.3%)
SPOR Values
95th % 1.34 1.30 1.25 1.20 1.18
90th % 1.31 1.26 1.18 1.16 1.14
Mean 1.02 1.01 1.00 1.00 1.00
10th % 0.72 0.78 0.82 0.84 0.86
5th % 0.60 0.70 0.75 0.78 0.81
SD 0.26 0.18 0.15 0.13 0.11

Though the size of the networks decreased sharply as the threshold increased, the percentage of significant SPORs remained relatively consistent. The decrease in extreme SPOR values and standard deviation indicated a tighter SPOR distribution for higher threshold networks (see also Figure 3).

We identified 139 providers who participated in one or more high-scoring collaborations and 125 providers involved in one or more low-scoring collaborations. Sixty-three providers were involved in both high-scoring and low-scoring collaborations. Applying the high- and low-scoring group definitions detailed in the Methods section, twenty-two providers fit the criteria for the high SPOR group (Table 3) and twenty-two providers qualified for the low SPOR group (Table 4). These two sets did not overlap when a minimum of 10 collaborations involving ≥ 6 encounters was required, although without this restriction the groups did share members. The average number of extreme valued-collaborations was similar for the high and low groups (3.55 and 3.95, respectively), however, the average number of total collaborations for a provider was lower for the high SPOR group (19.73) that the low SPOR group (24.95). An average of 18% of total collaborations based on six or more shared patients were significantly higher than expected (p ≤ 0.05) for members of the high SPOR group. For the low SPOR group, the average proportion of collaborations based on six or more shared patients that were significantly lower than expected (p ≥ 0.95) was 16%. Notably, providers in the high-scoring group were involved in an average of 34.55 encounters total, while those in the low-scoring group were involved in an average of 41.91. High SPOR group members had a higher percentage of their total encounters result in positive outcomes (78%) vs. the low SPOR group (65%).

Table 3.

Providers involved directly in patient care who participated in a significant number of high SPOR collaborations.

High SPORs
Provider Info Collaboration Statistics Associated Encounter Statistics
DEID Position # collabs ≥ 6
encounters
# SPORs p
≤ 0.05
% + Outcomes Total
Encounters
% Encounters w/
+ Outcome
9002361 Nurse 68 10 15% 57 81 70%
9001554 Attending Physician 32 4 13% 31 42 74%
9001141 Attending Physician 30 6 20% 30 39 77%
9001327 Advanced Practice
Clinician
27 6 22% 38 46 83%
9000101 ED Attending
Physician
23 3 13% 37 48 77%
9004501 Attending Physician 22 6 27% 16 24 67%
9000732 Attending Physician 21 4 19% 18 20 90%
9001813 Resident/Fellow 21 2 10% 32 40 80%
9000763 Attending Physician 20 7 35% 21 26 81%
9000014 Attending Physician 17 2 12% 27 35 77%
9001770 Resident/Fellow 17 2 12% 18 24 75%
9002409 Advanced Practice
Clinician
16 4 25% 12 16 75%
9002533 Resident/Fellow 15 2 13% 22 29 76%
9003269 ED Attending
Physician
15 2 13% 28 37 76%
9003687 Resident/Fellow 15 2 13% 29 37 78%
9002133 Resident/Fellow 13 4 31% 24 27 89%
9002454 Resident/Fellow 11 2 18% 26 31 84%
9002460 ED Attending
Physician
11 2 18% 23 31 74%
9002014 Resident/Fellow 10 3 30% 25 30 83%
9000042 ED Attending
Physician
10 2 20% 28 38 74%
9002062 Attending Physician 10 2 20% 26 30 87%
9001560 Resident/Fellow 10 1 10% 22 29 76%
Averages 19.73 3.55 18% 26.82 34.55 78%

Twenty-two providers had 1) at least 10 collaborations involving 6 or more encounters, and 2) at least 10% of these collaborative relationships in the top 5% of SPOR scores (p ≤ 0.05). The number of associated encounters with positive outcomes and the total number of associated encounters for each provider are also shown. The list is sorted from most to least by the number of collaborations involving six or more encounters.

Table 4.

Providers involved directly in patient care who participated in a significant number of low-SPOR collaborations

Low SPORs
Provider Info Collaboration Statistics Associated Encounter Statistics
DEID Position # collabs ≥ 6
encounters
# SPORs p
≥ 0.95
% + Outcomes Total
Encounters
% Encounters
w/ + Outcome
9002799 Nurse 54 14 26% 38 65 58%
9001854 Resident/Fellow 51 5 10% 40 58 69%
9000883 Attending Physician 47 5 11% 67 94 71%
9002116 Nurse 42 4 10% 33 46 72%
9001994 Hospitalist 41 5 12% 44 69 64%
9001195 Radiologist 39 4 10% 44 69 64%
9002272 Referring Physician 31 3 10% 24 36 67%
9001375 Radiologist 30 3 10% 37 55 67%
9004500 Attending Physician 23 6 26% 23 34 68%
9000679 Attending Physician 23 4 17% 24 39 62%
9003079 Resident/Fellow 21 3 14% 22 35 63%
9000223 Nurse 21 2 10% 20 33 61%
9002588 Attending Physician 18 3 17% 22 31 71%
9000921 ED Attending
Physician
18 2 11% 30 47 64%
9002774 Nurse 17 2 12% 10 18 56%
9002046 Resident/Fellow 12 3 25% 15 30 50%
9002502 Resident/Fellow 11 2 18% 22 32 69%
9002531 Resident/Fellow 10 6 60% 11 22 50%
9001386 Attending Physician 10 4 40% 21 29 72%
9001790 Resident/Fellow 10 3 30% 18 28 64%
9002452 Resident/Fellow 10 3 30% 13 25 52%
9000468 Nurse 10 1 10% 21 27 78%
Averages 24.95 3.95 16% 27.23 41.91 65%

Twenty-two providers had 1) at least 10 collaborations involving 6 or more encounters, and 2) at least 10% of these collaborative relationships in the bottom 5% of SPOR scores (p ≥ 0.95). The number of associated encounters with positive outcomes and the total number of associated encounters for each provider are also shown. The list is sorted from most to least by the number of collaborations involving six or more encounters.

A subset of the provider collaboration network consisting of 22 providers and 30 relationships is shown in Figure 4. This small sample of the larger network highlights the interconnected nature of provider relationships, but also shows that successful collaboration in term of patient satisfaction is highly variable. Providers involved in high-scoring, significant collaborations (highlighted relationship) are often involved in other collaborations with an average or low-valued SPORs. The underlying causes of this variability will be explored in future studies.

Figure 4.

Figure 4

A subset of the provider collaboration network consisting of 22 providers and 30 collaborations. Relationships are labeled with the SPOR score for the collaboration. The properties of the highlighted relationship are displayed on the bottom left and include SPOR rank (the rank order of the SPOR coefficient), p-value (pval), SPOR coefficient (coef), and the total number of shared encounters on which the collaboration is based (num_collabs).

Discussion

Through analysis of a cardiology unit data set using our SPOR method, we have demonstrated that working relationships between provider pairs in an inpatient cardiology unit are unequal when considered in the context of patient satisfaction. Despite the fact that many providers participate in both high- and low-scoring relationships, increased collaboration frequency between members of a provider pair may improve resulting patient outcomes. However, top collaborators in the cardiology unit had fewer total established collaborative relationships on average than those in the low scoring group. In addition, these top collaborators were involved in fewer total encounters but had a much higher percentage of positive outcomes for those encounters versus the low-scoring group, suggesting that contributions from strong individual providers may improve outcomes for a collaborating pair.

Characterization and evaluation of a collaboration network reveals previously unknown strengths, weaknesses, and patterns of interaction. Our method can be used to measure the overall performance of a clinical service in terms of a chosen outcome as well as to identify an individual’s potential as an effective collaborator. A low SPOR score does not indicate that a provider has inferior clinical skills; rather, it may reveal previously unrecognized structural or organizational barriers. Clinical workflows and protocols can be overlaid on top of the collaboration network to identify potential problem areas, which we demonstrated in a recent study29. These data reviewed periodically by hospital and provider leadership may lend insight into teamwork barriers and allow for construction of interventions to improve collaboration. This could be formatted in the context of quarterly quality review on the unit, or by discussion among unit staff sharing best practices as well as addressing barriers. Furthermore, the flexibility of our framework allows it to be used to study other important outcomes like 30-day readmissions, mortality, patient-reported outcomes such as quality of life, and operational metrics such as length of stay and cost. Other risk adjustment models could also be introduced30.

This study was intended as a demonstration project and as such has a number of limitations. First, while the EHR can be a valuable source for patient encounter data, it does not capture the entirety of the communication spectrum among providers within a clinical setting. As a result, our current models are missing important information about working relationships that could affect patient outcomes. However, if available this auxiliary data could be incorporated to create a more informed collaboration model. Second, our method focuses on pairs of providers rather than a more comprehensive analysis of the social network surrounding the patient. We chose this approach because a provider pair represents the building block for the entirety of the care team. Understanding how a particular pair performs may elucidate strengths and weaknesses within a team and identify potential areas for intervention. In addition, due to the complexity, heterogeneity, and re-assortment of team members for heart failure in our data set, we could not consistently identify more than two member teams with enough patients and outcomes. To better characterize team recombination, future analyses with larger data sets will attempt to develop models for understanding collaboration among groups of multiple providers and the relationship to patient outcomes, as well as an individual provider’s contribution to these collaborative relationships. Third, as this was a retrospective and descriptive study, we were not able to validate our model with the data available. To address this issue, future work will include sources of qualitative data such as focus groups of providers, provider surveys, and interviews to determine factors that could potentially affect collaborative relationships. We plan to build a SPOR model concurrently with these qualitative studies for validation purposes.

Conclusions

We have shown that on a global level, collaboration between providers is highly variable in terms of patient satisfaction. We have demonstrated a novel approach called SPOR, a metric that quantifies the ratio of specific positive outcomes shared between two providers, to identify extreme high- and low-scoring relationships over a set of shared patient encounters. Objective measures of collaborative relationships provide a foundation for evaluating methods for improving care coordination and patient outcomes.

Supplementary Material

Supplemental Material

What is known

  • Teamwork comprises a key component of high quality, coordinated care, but few methods to measure teamwork exist.

  • The widespread adoption of EHRs has created the opportunity to use social network analysis to examine the relationship between teams and patient-centered outcomes.

What the study adds

  • The Shared Positive Outcome Ratio demonstrates the ability to link high performing and low performing teams with relevant, risk-adjusted patient outcomes such as patient satisfaction and likelihood to recommend.

  • The SPOR enables evaluation of complex and dynamic teams, understanding that teams and team-members change constantly and some teams work together more often than others in the real clinical environment.

  • Future work should expand the SPOR method to evaluate larger provider teams as well as individual provider contributions to patient outcomes in these teams.

Acknowledgments

Funding Sources: Research reported in this publication was supported, in part, by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number UL1TR001422 and the National Library of Medicine Grant Number K01LM011973-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Conflict of Interest Disclosures: None.

References

  • 1.Centers for Medicare & Medicaid Services. Readmissions Reduction Program. [Accessed August, 2015]; http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html.
  • 2.Care Coordination Measures Atlas Update. Agency for Healthcare Research and Quality (AHRQ) [Accessed August, 2014];2014 Jun; http://www.ahrq.gov/professionals/prevention-chronic-care/improve/coordination/atlas2014/index.html.
  • 3.Benham-Hutchins MM, Effken JA. Multi-professional patterns and methods of communication during patient handoffs. Int J Med Inform. 2010;79:252–267. doi: 10.1016/j.ijmedinf.2009.12.005. [DOI] [PubMed] [Google Scholar]
  • 4.Weenink JW, van Lieshout J, Jung HP, Wensing M. Patient Care Teams in treatment of diabetes and chronic heart failure in primary care: an observational networks study. Implement Sci. 2011;6:66. doi: 10.1186/1748-5908-6-66. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Nageswaran S, Ip EH, Golden SL, O'Shea TM, Easterling D. Inter-agency collaboration in the care of children with complex chronic conditions. Academic pediatrics. 2012;12:189–197. doi: 10.1016/j.acap.2012.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Chambers D, Wilson P, Thompson C, Harden M. Social network analysis in healthcare settings: a systematic scoping review. PloS one. 2012;7:e41911. doi: 10.1371/journal.pone.0041911. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Ong MS, Olson KL, Cami A, Liu C, Tian F, Selvam N, Mandl KD. Provider Patient-Sharing Networks and Multiple-Provider Prescribing of Benzodiazepines. Journal of general internal medicine. 2015 doi: 10.1007/s11606-015-3470-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Christakis NA, Fowler JH. Connected : the surprising power of our social networks and how they shape our lives. 1st. New York: Little, Brown and Co.; 2009. [Google Scholar]
  • 9.Pah AR, Rasmussen-Torvik LJ, Goel S, Greenland P, Kho AN. Big Data: What is it and what does it mean for cardiovascular research and prevention policy. Current Cardiovascular Risk Reports. 2015:1–9. [Google Scholar]
  • 10.McDonald KM, Sundaram V, Bravata DM, Lewis R, Lin N, Kraft SA, McKinnon M, Paguntalan H, Owens DK. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies (Vol 7: Care Coordination) Rockville (MD): 2007. [PubMed] [Google Scholar]
  • 11.Gray JE, Davis DA, Pursley DM, Smallcomb JE, Geva A, Chawla NV. Network analysis of team structure in the neonatal intensive care unit. Pediatrics. 2010;125:e1460–e1467. doi: 10.1542/peds.2009-2621. [DOI] [PubMed] [Google Scholar]
  • 12.Hripcsak G, Vawdrey DK, Fred MR, Bostwick SB. Use of electronic clinical documentation: time spent and team interactions. J Am Med Inform Assoc. 2011;18:112–117. doi: 10.1136/jamia.2010.008441. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Vawdrey DK, Wilcox LG, Collins S, Feiner S, Mamykina O, Stein DM, Bakken S, Fred MR, Stetson PD. Awareness of the Care Team in Electronic Health Records. Appl Clin Inform. 2011;2:395–405. doi: 10.4338/ACI-2011-05-RA-0034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Soulakis ND, Carson MB, Lee YJ, Schneider DH, Skeehan CT, Scholtens DM. Visualizing collaborative electronic health record usage for hospitalized patients with heart failure. J Am Med Inform Assoc. 2015;22:299–311. doi: 10.1093/jamia/ocu017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.U.S. News Best Hospitals 2015–16. US News and World Report. 2016 [Google Scholar]
  • 16.Coffman J, Yale P. Would you recommend this hospital to a friend? 2007 [Google Scholar]
  • 17.Singh SC, Sheth RD, Burrows JF, Rosen P. Factors Influencing Patient Experience in Pediatric Neurology. Pediatr Neurol. 2016;60:37–41. doi: 10.1016/j.pediatrneurol.2016.04.002. [DOI] [PubMed] [Google Scholar]
  • 18.Bible JE, Shau DN, Kay HF, Cheng JS, Aaronson OS, Devin CJ. Are Low Patient Satisfaction Scores always Due to the Provider? Determinants of Patient Satisfaction Scores During Spine Clinic Visits. Spine (Phila Pa 1976) 2016 Jan 15; doi: 10.1097/BRS.0000000000001453. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]
  • 19.Likert R. A technique for the measurement of attitudes. New York: 1932. [Google Scholar]
  • 20.Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med. 2000;7:236–242. doi: 10.1111/j.1553-2712.2000.tb01066.x. [DOI] [PubMed] [Google Scholar]
  • 21.Starren JB, Winter AQ, Lloyd-Jones DM. Enabling a Learning Health System through a Unified Enterprise Data Warehouse: The Experience of the Northwestern University Clinical and Translational Sciences (NUCATS) Institute. Clin Transl Sci. 2015;8:269–271. doi: 10.1111/cts.12294. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Microsoft SQL Server Management Studio [computer program] 2012 [Google Scholar]
  • 23.The Neo4j Manual v2.2.5. 2015 http://www.neotechnology.com [computer program] [Google Scholar]
  • 24.Python Software Foundation. Python Language Reference, version 2.7. http://www.python.org [computer program] [Google Scholar]
  • 25.Py2neo version 2.0.8. 2015 http://py2neo.org/2.0/ [computer program] [Google Scholar]
  • 26.Hagberg AA, Schult, Daniel A, Swart, Pieter J. Exploring network structure, dynamics, and function using NetworkX. Paper presented at:Proceedings of the 7th Python in Science Conference (SciPy2008); Aug 2008; Pasadena, CA USA. [Google Scholar]
  • 27.Vienna, Austria: R Foundation for Statistical Computing [computer program]; 2012. R: A Language and Environment for Statistical Computing. [Google Scholar]
  • 28.Jaccard P. Nouvelles recherches sur la distribution florale. Bull Soc Vaudoise Sci Nat. 1908;44:223–270. [Google Scholar]
  • 29.Kricke GS, Carson MB, Lee Y, Benacka C, Mutharasan RK, Ahmad FS, Kansal P, Yancy CW, Anderson AS, Soulakis ND. Leveraging electronic health record documentation for Failure Mode and Effects Analysis team identification. J Am Med Inform Assoc. 2016;0:1–6. doi: 10.1093/jamia/ocw083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Haas LR, Takahashi PY, Shah ND, Stroebel RJ, Bernard ME, Finnie DM, Naessens JM. Risk-stratification methods for identifying patients for care coordination. Am J Manag Care. 2013;19:725–732. [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Material

RESOURCES