Does a Community-Engaged Health Informatics Platform Facilitate Resource Connectivity? An Evaluation Framework

Abstract

Community-engaged health informatics (CEHI) integrates informatics with community-based participatory public health. Addressing social determinants and population health requires mobilization of health-related resources in communities. We present a framework for evaluating the process and outcomes of a CEHI platform designed to improve connectivity among community health resources.

The GetHealthyHeights.org CEHI platform was implemented in an urban low-income community. It was designed to facilitate connectivity among health-related community-based organizations (CBOs). To evaluate the process towards and the achievement of connectivity, a conceptual framework, methodology, and operational measures were defined.

A system-level approach, such as social network analysis, is required to capture the community as one dynamic unit. The evaluation framework specifies network connectivity metrics based on a social network survey. A network survey of CBOs (n=35) at baseline demonstrates utility of social network data for characterizing connectivity among community resources. The evaluation framework models how informatics and community resources improve population health.

Introduction

Evidence indicates that zip code rivals genetic code as a determinant of overall health, urging more interventions on the “zip code level”¹. Accordingly, there is increasing demand for an ecological systems paradigm for community interventions, with focus on: community capacity; problem identification through community engagement; empowerment of the community; and the permeating role of culture². As the importance of social determinants of health is increasingly recognized, health systems must seek collaborations with community-based partners that are equipped to meet diverse social service needs of patients ³. From the healthcare financing perspective, populationbased accountable care models provide additional incentives for health systems to address social determinants through community-based services⁴.

Community-engaged health informatics (CEHI) is an emerging field with many opportunities for novel discoveries, as it combines concepts and methods from biomedical informatics⁵, community-based public health approaches⁶, as well as other fields such as community informatics⁷. CEHI takes an ecological systems approach to community health². It extends the notion of Learning Health System to the community-level, and investigates the ecology of community health information in a “cyber-social ecosystem”⁸.

We applied community-based participatory research (CBPR)⁶ to design a CEHI platform GetHealthyHeitghts.org (GHH) for the medically underserved, urban, largely Latino community of Washington Heights-Inwood (WAHI)⁹. GHH features an innovative participatory architecture with functionality for facilitating community stakeholder interactions. In addition to community residents, and a range of other community stakeholders, GHH specifically serves health-oriented CBOs, as they are viewed as essential assets for building community health interventions¹⁰. As an intervention, GHH was designed to facilitate connectivity among stakeholders in the community.

The design of GHH builds on the lessons learned from GetHealthyHarlem.org^{11, 12}. While the Harlem portal was designed primarily for community member participation, the GHH team decided to primarily focus on communitybased organizations and other groups and agencies that care about health of WAHI. GHH is intended to be a CEHI platform that allows exploration of novel functionalities that promote community engagement and community health. The mission of GHH was collectively defined by the GHH Steering Committee as “an online community that engages people and organizations in Washington Heights-Inwood to discover, connect, and share resources to get healthy”.

Selected functions include a community calendar, a local service directory, posting of multiple types of content (e.g., articles, videos, and links), the ability to comment and rate content, integration of social media for content sharing, use of Google Translate (especially for Spanish translation of content), creation of pages for local organizations and the ability to form groups that other users can join. Community organizations have an essential role in creating and disseminating content through GHH. Figure 1 shows an illustrative screen shot of GHH. An initial version of GHH was launched to the public in April 2015. As of February 2016 (23 months of public use), the website had 43,990 page views, 16,860 use sessions (average 733 per month), 39 partner organizations, and 286 registered users¹³.

Figure 1.

Screenshot of GetHealthyHeigths.org

GHH seeks to include and engage all health-related resources in the community. A comprehensive resource mapping was conducted and is described elsewhere¹⁴. Resources encompass public, non-profit, private, and informal sectors, and range from large organizations (e.g., universities, hospitals) to small informal entities (e.g., bulletin board, walking group). The GHH team is currently in the process of defining a taxonomy to better classify types of health-related community resources, organizations, and services. CBOs are an important sub-type of resource and play a central role in GHH. The evaluation approach described in this paper specifically focuses on detecting change over time in network connectivity among CBOs, and attributing the change to GHH.

CBOs play an important role in networks of community health resources and services but they have poor capacity for processing, using, and exchanging information¹⁵. Care coordination is a well-documented problem within the health care system generally¹⁶, suggesting similar challenges with coordination of community-based services and resources. Emerging literature in the area of knowledge translation is beginning to highlight the potential of CBOs in translating evidence into the community^{15, 17}. But many barriers hinder evidence use and evidence-based practices of CBOs. Research has consistently identified that CBOs struggle with: access to evidence; time to process evidence; skills to review, summarize, and synthesize evidence; research terminology; and local applicability and acceptability of evidence^18-20. Strategies are needed to address these barriers in order to realize the potential of CBOs as important agents of improving population health. The GHH CEHI platform targets these challenges through a set of online functionalities specifically designed to facilitate connectivity and information access among CBOs in the WAHI community. Enhanced connectivity of a community health resources network may provide positive synergistic effects²¹, and has a large potential pay-off in terms of health-related social capital²², and resulting benefits to community wellness²³.

Network connectivity means that local community-based organizations (CBOs) are sharing information, collaborating, and cross-referring clients^{24, 25}. Paarlberg and Varda argue that network exchanges are the most essential driver of capacity in the non-profit sector²¹. Connectivity is an important outcome because it is expected to make delivery of programs and services more efficient, and to facilitate health-related social capital²², contributing to improved community wellness and reduction in health disparities²³. In the context of health informatics, there is a vision of seamless exchange of health data across organizations, and CBOs are increasingly considered part of the health data exchange network. For this future vision of sharing data among CBOs to become reality, trust among organizations becomes a critically important intermediate connectivity construct. Broad-based data sharing will require trust as a quality of organizational linkages.

A system-level approach is required to study connectivity on the community level. Social network analysis provides such approach. Some prior research has used social network analysis to demonstrate outcomes of community-level interventions. For example, Ramanadhan and colleagues ²⁴ used network analysis to show that a cancer prevention coalition building intervention increased the connectivity among members of an intersectoral agency network in Massachusetts. The PARTNERtool has been used in several studies to measure social networks of community partnerships²⁶. It provides an established instrument for operationalizing the kinds of linkages and exchanges that constitute network connectivity.

The overall rationale of the GHH CEHI platform as an intervention and its impact on health draws from several theoretical influences, including literature on community engagement, social networks, and social capital²⁷. Based on literature, our conceptual model postulates that network connectivity contributes to social capital²², and social capital contributes to community health²³.

The GHH CEHI platform is purposefully designed to facilitate health resource network connectivity. The architecture of participation in GHH⁹ mediates connections and interactions among community stakeholders. The evaluation framework we present here seeks to specify some of the processes by which functionality in GHH, and participation in GHH, are expected to lead to more resource connectivity.

Prior research has not investigated the impact of a comprehensive CEHI intervention such as GHH. Studies are starting to explore ways of applying informatics methods in the context of community engagement^28-32. Compared to other applications of health informatics in the community engagement context, GHH is unique in its focus on CBOs as the primary community partners. One reason why GHH partners closely with CBOs is the desire to enhance sustainability of GHH by building it in partnership with stable local community assets³³. Lack of sustained use is a known challenge for information technology (IT) interventions that rely on voluntary motivation of users³⁴, and is potentially addressed through a community-level approach that engages CBOs to champion the IT system.

This paper presents a conceptual model and describes how it is operationalized. Our view of operationalization comes from the social science research perspective³⁵. We describe the conceptual underpinnings and the design of the evaluation framework. We also present baseline data for the network survey to substantiate how key outcome constructs in the framework were operationalized and measured.

Methods

Community Setting of Evaluation Study

The Washington Heights-Inwood (WAHI) neighborhood is located in upper Manhattan within New York City (NYC), north of Harlem and directly south and west of the Bronx. It is a densely populated urban area of 2.8 square miles and four zip codes with approximately 200,000 residents³⁶. A large proportion of WAHI residents are Hispanic (71%), and 93% belong to a racial/ethnic minority group. African Americans represent 7% of the population³⁷. Almost half (48%) of residents are foreign-born, mainly from Latin America, with 2/3 from the Dominican Republic. Many ethnic groups are represented as foreign-born residents come from a total of 55 countries³⁸. A large proportion (39%) of residents have limited English proficiency³⁷. The median household income was $37,460 in 2013, and 27% lived below the federal poverty level³⁸. A large percentage (30%) of adults did not graduate from high school. The most common sources of employment are services and sales industries³⁹. Housing quality is rated among the lowest in NYC³⁷. Health concerns in the community are significant compared to NYC as a whole. Among WAHI residents, 27% rate their health as fair or poor, compared to 22% citywide, and 16% reported that they went without needed medical care. Obesity rate is 22%, and 22% of residents report they had no physical activity in the past month. Only 36% had influenza vaccinations³⁷.

Approach to Defining Evaluation Framework

The overall approach includes three major components: (1) conceptual modeling, (2) selection of evaluation design and methodology, and (3) definition of operationalized measures.

The plan to evaluate process and outcomes of GHH evolved during the design stage of the project and was based on conceptual modeling of relevant constructs. Important key constructs included: “engagement” to capture active participation of users with GHH, “network connectivity” to capture a system level outcome on the level of the community as a whole, and “health-related social capital” as a construct that bridges network connectivity to community health outcomes. Detailed conceptual analysis was conducted to further break down components of each of the key constructs. Several iterations of logic models and conceptual diagrams were developed to capture the conceptual framework. Community stakeholders were engaged and provided input into the models, primarily through the GHH Steering Committee.

An evaluation design and methodology were developed based on the conceptual modeling and considerations of feasible data collection options. While recognizing the limitations of a pre-post design, an outcome evaluation design was selected to capture change in network connectivity over time in the target community. Consistent with GHH focus on CBOs, the team decided to primarily focus the evaluation on network connectivity among CBOs. Baseline data collection was conducted during early implementation of GHH, with a plan to collect similar data at later time points.

The PARTNERtool²⁶ was selected as an established measure of network interactions among community agencies. It was adaptable for the purposes of the GHH evaluation. The PARTNERtool provided a foundation for defining operationalized measures of constructs that constitute “network connectivity”, as further described in the Results section below.

The process evaluation was designed to document and measure how organizations and individuals engage in GHH. Process evaluation data sources included Google Analytics for GHH, system back-end data from GHH use logs, and records of participation in GHH governance and other GHH-related activities. In the outcome evaluation design, the process measures also serve as independent variables, with the plan of measuring associations between GHH engagement and network connectivity outcomes for individual CBOs.

Baseline Network Survey Methodology

The primary inclusion criteria for the network survey was: non-profit organization that provides health-related services to the WAHI community and has a physical office location within the four targeted zip codes. The research team went through a systematic process of identifying the list of CBOs. This process involved decisions about what kinds of organizations to include as “health related”. The initial list of organizations was based on the researchers’ prior experience of working in the community, complimented with results from a systematic community resource mapping. The resource mapping was based on five data sources on community resources, including most recent files from the Internal Review Service listing all organizations with 501(c)(3) non-profit status within the four zip codes. We have described the resource mapping process elsewhere¹⁴.

The team identified 25 qualifying non-profit organizations, including 8 multi-service CBOs, 6 senior centers, 2 nursing homes, 2 health centers/health center networks, 2 behavioral health providers, 3 child/youth service providers, 1 freestanding food pantry, and 1 HIV/AIDS service provider. The team also decided to invite three local city-government organizations to participate in the survey: local Community Board (part of city government), and 2 recreation centers (operated by city). In addition, the team decided it was important to represent the following seven entities in the survey to capture a more comprehensive picture of the community’s network: city government (libraries, health department, police department, etc.); Columbia University; Yeshiva University; K-12 public, private, and parochial schools; New York Presbyterian Hospital community programs; New York State Psychiatric Institute (located within the community); and faith-based organizations. The 25 non-profits and the three city government entities constitute the pool of 28 organizations invited to complete the survey. The 28 organizations and the seven additional entities constitute the set of 35 entities listed on the survey. Each survey respondent was asked about their connections to all other 34 entities.

We used an adapted version of the validated PARTNERtool²⁶. The PARTNERtool was designed to collect network interaction data from public health collaborations. It includes a section where the respondent is asked to indicate relationships and interactions with all listed agencies. Organizational leaders were recruited to respond to the surveys. The instructions stated that the Executive Director (or equivalent, or someone he or she designates) should fill out the survey as a representative of the organizational perspective. The survey administration was managed through the PARTER tool website.

The survey protocol was approved by the Columbia University Medical Center Institutional Review Board. The survey data were collected in summer and fall of 2016. It should be noted that GHH was already open to the public at that time, and many of the organizations in the survey already began initial use of GHH before they were recruited to participate in the survey. Thus, the baseline survey was collected during early stage implementation of GHH.

Results

Evaluation Framework

Figure 2 depicts the conceptual logic of the causal chain of outcomes we theorize for the evaluation framework. The evaluation focuses on measuring engagement in GHH as the primary independent variable and connectivity in the community resource network as the primary outcome variable. The other outcome constructs are specified to create a full picture of how and why we think GHH contributes to community wellness. In particular, the logic model defines health-related social capital as the construct that links resource network connectivity to health outcomes. Even though we do not measure them in the current evaluation, we postulate several components of social capital that the GHH intervention can improve. For example, we argue that improved connectivity will lead to efficiency in the way the community as a whole is able to deliver health programs and services.

Figure 2.

Logic model of outcome constructs

Further conceptual analysis was conducted to elaborate on the two constructs the evaluation is designed to measure, namely, engagement in GHH and connectivity. This step is grounded in a more concrete level, looking at the actual functionality built into GHH and the actual activites of connecting among CBOs. The two bottom boxes of Figure 3 represent these two domains. The model suggests that the GHH system functionality (and participation in GHH governance) facilitate specific ways in which CBOs connect among one another. The top two boxes of Figure 3 represent how the evaluation measures the constructs in the bottom boxes. The top section of the figure captures the essential elements of the evaluation plan. Measures of the processes (and independent variable) of GHH use and participation are described on the top left, and specific network connectivity measures are described in the top right. The study design examines change in network metrics over time, but in addition, as indicated in the figure, we plan to measure associations between GHH use/participation and connectivity indicators for individual organizations.

Figure 3.

Key constructs and evaluation measures

* Constructs measured in current evaluation

It should be noted that the engagement of CBOs occurs within and outside of the actual online platform. In fact, in- person meetings related to governance of GHH appear to be a powerful mechanism for engaging organizations around GHH. Other engagement processes, such as sharing content or finding resources, reside more directly on the GHH website.

Baseline Network Results

The baseline network survey was sent to 28 organizations; with a 61% response rate. The survey listed 35 total organizations; 28 that received surveys and seven that were listed to represent selected groups or entities (e.g., faithbased organizations) that were not recruited to respond to the survey.

Organizations that responded reported that they collectively had 273 connections with other entities. The average number of connections per organization was 13 (out of a possible 34). The top seven most connected had an average of 25 connections per partner, leaving the remaining 28 organizations averaging 10 connections per partner. Table 2 shows selected network metrics for the entire network, along with the definitions of the metrics.

Figures 4 and 5 show network diagrams depicting types and frequencies of baseline connections among the entities. As shown in Figure 4, about one-third (32%) indicated that they only had awareness of one another, while 47% indicated cooperative activity connections, 14% indicated coordinated activity connections, and 7% indicated integrated activity connections with one another.

Figure 4.

Types of connections among network entities in baseline network survey

Figure 5.

Frequency of connections among entities in baseline network survey

Discussion

Our evaluation framework offers a conceptual and methodological map for measuring the impact of a community- level informatics intervention on network connectivity among community organizations. We trace conceptual and operationalized definitions from engagement with functionality of the information system (GHH) to different dimensions of interactions and exchanges that constitute network connectivity. We also discuss the intervention change model that postulates that network connectivity translates to health-related social capital²², which, in turn, is expected to contribute to positive population health outcomes²³.

The baseline results of the social network survey of CBOs, using the PARTNERtool²⁶, demonstrate that the survey methodology produces a rich set of network metrics for describing the state of the CBO network at baseline. Among the many available metrics are density of connections, and different types and frequencies of connections. “Trust” is an important example of a specific quality of a connection that provides a conceptually meaningful outcome in the context of our evaluation. With “Trust” at 64%, density at 39%, the community appears to have many positive connections, but with ample room for increase from baseline. Examination of the baseline visualizations shows that many of the possible binary connections are not in place at baseline, and suggests that any new edge that is added to the network in the future can be examined as evidence of change, and potentially characterized resulting from GHH facilitation. The social network methodology allows analyses of the entire network as a whole system, as well as drilling down to the level of individual organizations. Having both levels is extremely useful for an evaluation framework in a transactional context such as CBOs connecting in a community and engaging with a system designed to facilitate such connections.

Engagement is a central construct within CEHI. The design of GHH with an architecture of participation is an operationalized instance of engagement processes⁹. In the evaluation framework, use of GHH functionality becomes not only a process evaluation metric, but also means of measuring engagement as an independent variable. An association between use of GHH and improvement over time in network connectivity measures serves as evidence that GHH has its intended impact.

The GHH evaluation highlights the role of CBOs as special kind of health-related resource. Collectively, CBOs can establish a backbone of network exchanges that translates into community-level benefits that are more than the sum of its parts²¹. CBO networks are a natural mechanism for addressing social determinants of health. In the future, medical care systems should pursue closer connections with CBO networks and other community resources³. This will be a starting point for a Learning Health System⁸ that includes community services and resources. The future vision of such Learning Health System includes sharing and flow of data across medical care and community entities. The construct of “trust”, which we examine in our evaluation as a component of inter-organizational connectivity, will be a critical pre-requisite for data sharing and exchange to ever become reality.

Limitations

We only present a preliminary stage of work that needs to be supported by further data collection and analysis. However, we believe it is valuable to present a focused conceptual analysis that constructs a coherent and organized model of the phenomenon under evaluation and an operationalized way of measuring it. Our initial analysis shows that much more can be done, even within the realm of operationalizing the measures. For example, internal correlations within the baseline network survey dataset can be used to further examine validity of sub-constructs of connectivity.The baseline dataset can also yield numerous additional network metrics both for the entire network and for its subunits. For example, we plan to examine measures of strong ties (bonding social capital) and weak ties (bridging social capital) within the network⁴⁰. We can also further specify the statistical analyses approaches to be used to measure change in network metrics over time.

A pre-post evaluation design implemented in only one community has obvious limitations, but is a valid starting point for further work. It was feasible to collect baseline data only after implementation of GHH had already begun. At present time we observe community organizations increasingly engaging around GHH. Once we collect systematic data over time we will be able to further describe mechanisms by with GHH functionality facilitates the engagement.

Operationalizing valid constructs and study designs in the context of a complex information technology poses special challenges and limitations⁴¹. It is a limitation of validity if only one system is built to “instatiate” the constructs. It should be recognized that the specific and idiosyncratic ways in which this system gets designed may account for the evaluation results.

Conclusion

We argue that mobilizing community-based resources through informatics is an important strategy that should be leveraged to improve population health. GHH is an instance of such strategy, linking engagement in a CEHI technology to network connectivity among entities in a geographic community.

Table 1:

Selected network metrics for whole network

Network Metric	Baseline Survey Score	Definition of Metric
Density	39%	Percentage of ties present in the network in relation to the total number of possible ties in the entire network.
Degree Centralization	56%	The lower the centralization score, the more similar the members are in terms of their number of connections to others (e.g. more decentralized).
Trust	64%	The percentage of how much members trust one another. A 100% occurs when all members trust others at the highest level.