Measuring engagement in a collaborative learning health system: The case of ImproveCareNow

Abstract

Background

Collaborative learning health systems have demonstrated improved outcomes for a range of different chronic conditions. Patient and healthcare provider engagement in these systems is thought to be associated with improved outcomes. We have adapted an observational framework to measure, and track over time, engagement in ImproveCareNow, a collaborative learning health system for children with inflammatory bowel disease.

Introduction

We developed a categorical classification scheme for engagement in ImproveCareNow. Each tier is defined in terms of observable individual behaviors. When an individual completes one or more qualifying behavior, s/he is classified as engaged at that tier. Individuals are entered into a database, which is accessible to care centers throughout the ImproveCareNow network. Database records include fields for individual name, behavior type, time, place, and level of engagement.

Results

The resulting system is employed at 79 ImproveCareNow care centers in the United States. The system recognizes four levels of engagement. Behaviors are recorded in a managed vocabulary and recorded in an online database. The database is queried weekly for individual engagement behaviors, which are tracked longitudinally. Center‐ and network‐level statistics are generated and disseminated to stakeholders.

Conclusion

It is possible to monitor longitudinal engagement in a collaborative learning health system, thereby charting progress toward engagement goals and enabling quantitative evaluation of interventions aimed at increasing engagement.

1. INTRODUCTION

The documented improvement in healthcare outcomes in collaborative learning health systems (CLHSs) illustrate the benefit to patients and healthcare providers of such networks compared to traditional approaches to clinical care. ¹ , ² , ³ Recent research has examined predictors of improved outcomes in such systems, including patient activation, ⁴ , ⁵ clinical quality improvement activities, ⁶ , ⁷ , ⁸ and new treatment protocols. ⁹ , ¹⁰ , ¹¹ Importantly, however, unknowns remain, including how individual participation and engagement in a CLHS may affect activation and improved outcomes. ¹²

In a CLHS, engagement refers to the degree to which people (patients and families, clinicians, researchers, and others) are involved in creating and sharing information, knowledge, and knowhow to improve health and healthcare. ¹² , ¹³ , ¹⁴ , ¹⁵ For example, in one CLHS (ImproveCareNow, described below), a group of highly engaged young people with inflammatory bowel disease (IBD) worked together to create a digital product describing experiences of patients who had undergone an ostomy. ¹⁶ The rationale for doing so included that they had “experienced a lack of psychosocial education about our surgeries. This negatively impacted our ability to adjust postoperatively to having an ostomy.” Sharing such resources with the community (eg, https://www.improvecarenow.org/a_guide_to_gutsy_living) enables patients for whom this therapy is indicated to learn about the experience in the voice of other patients, potentially easing postoperative psychosocial adjustment.

This is but one example of how CLHSs enable patients, clinicians, researchers, and other stakeholders to collaborate at scale in order to improve outcomes and generate new knowledge. Seid et al. ¹⁵ have recently described CLHSs as actor‐oriented architectures in which collaboration is an important cultural norm. In that view, collaboration contributes to the diffusion of ideas for improving outcomes, the sharing of better tools and processes, and the self‐organization of people pursuing aligned goals among others. In order to elucidate and better understand CLHS mechanisms of action, it is important for CLHSs to have measures of engagement in collaborative processes. An approach for longitudinally collecting and monitoring reliable indicators of engagement is therefore necessary.

In addition to research questions related to engagement, there are operational reasons for collecting engagement data. Consistent with common quality improvement process, CLHSs need the ability to track growth in engagement over time, identify areas for improvement, evaluate the impact of interventions aimed at increasing engagement, and shape the narrative of their work. With such a capability, CLHSs can evaluate the return on investment for engagement activities and make the case for more care centers to participate. In this report, we describe a system for classifying, recording, and tracking engagement in a specific CLHS, the ImproveCareNow (ICN) network. ¹⁷

2. METHODS

The approach entailed defining an engagement classification and coding scheme based on observable behaviors indicating involvement in ICN, implementing that in a database, and computing descriptive statistics from the database to provide engagement situational awareness.

2.1. Population

ImproveCareNow (https://www.improvecarenow.org/) is a CLHS focused on pediatric inflammatory bowel disease (IBD; Crohn's disease and ulcerative colitis), founded in 2007. Care centers in ICN range from small private practices to children's hospitals associated with academic medical centers. As of December 2018, ICN consisted of 109 care centers, comprising approximately 30 000 patients (the majority [61%] under 18 years of age), plus parents, primarily in the United States. There are many ways for parents and patients to be involved with ICN. Examples include consenting to have clinical data entered in the ICN patient registry ¹⁸ ; following ICN on social media; accessing and utilizing resources such as toolkits and guides (eg, ostomy, body image, and travel toolkits; accommodations for IBD and transition to adult care guides available at https://www.improvecarenow.org/toolkits_and_guides) ¹⁶ ; using physician visit planning tools ¹⁹ ; attending ICN community conferences in person or online; contributing to or developing new guides, tools, and toolkits; and participating in patient or parent advisory panels and as leaders in the network, among many others. Data presented in this report focused on parents and patients associated with 79 care centers that had at least one patient in their practice listed in the engagement database (see below) between the dates January 8, 2017 and May 31, 2019.

2.2. Engagement classification

Given the above definition of engagement (ie, the extent to

which an individual in a CLHS takes part in the production of information, knowledge, and know‐how ¹² ), a tiered, categorical engagement classification scheme was developed. The scheme identifies an individual's engagement‐related behaviors in ICN, such as accessing and using ICN resources (eg, online tools and information) and involvement in ICN activities (eg, outreach, conferences, committees). Each tier represents progressively higher levels of engagement in terms of time and effort required to undertake the corresponding actions. Definitions were developed and used to assign individuals to specific engagement tiers when they perform specific behaviors, represented in the database in a set of standardized codes.

In a way similar to that used in community organizing projects, ²⁰ , ²¹ four levels of increasing engagement were identified by ICN project staff as being of interest. These were (a) awareness, which we defined as parents or patients having knowledge that ICN exists and provides value, such that they allow ICN to connect with them directly (1‐way communication for example via email or Twitter); (b) participation, defined as parents or patients using existing resources and tools (eg, attending ICN and local care center events; attending an engagement campaign training boot camp; participating in a project or group); (c) contribution, in which parents or patients work to improve existing resources and tools (eg, care center level quality improvement [QI] activities; serving as care center team leads or on committees/workgroups; writing an ICN blog post; presenting at a Community Conference); and (d) ownership, in which parents or patients lead initiatives to improve or create resources and tools (eg, leading a project or group).

2.3. Coding engagement actions

Based on these definitions, each engagement level was defined in terms of codes corresponding to engagement actions available to ICN patients and parents of patients (Tables 1, 2, 3, 4). Codes were structured in the format verb:object. As shown in Table 1, individuals who express interest in or seek information on ICN resources or activities, as expressed by the verbs to receive, submit, follow and like, were classified as aware. The objects of awareness verbs were online resources (email and social media) and paper follow‐up cards received at clinic visits. Table 2 summarizes codes for participation, which correspond to a larger diversity of verbs (eg, to register, attend, tweet, comment, answer, join) and objects (eg, blogs, webinars, working groups, special meeting activities). Table 3 elaborates codes for contribution, and Table 4 describes codes for owner.

TABLE 1.

Codes denoting “awareness” in ICN

Action	Code	Updates	Frequency
Sign up for CIRCLE community	receives:email_circle	Automatic	Continual
Follow LOOP blog	receives:email_loop	Automatic	Continual
Research interest	receives:email_research	Automatic	Continual
Receives local communications	receives:email_local	Automatic	Continual
Contact us	submits:contact_us	Automatic	Continual
Follow on Twitter	follows_icn:twitter	Automatic	Weekly
Interact with Facebook page	likes:facebook_post	Manual	Weekly
Network participant	receives:email_network	Manual	Biweekly
Submits follow‐up card	submits:follow_up_card	Manual	Monthly

TABLE 2.

Codes denoting “participation” in ICN

Action	Code	Updates	Frequency
Register for exchange	registered:exchange	Manual	Bimonthly
Attendance at community conference	attends:community_conference	Manual	Semiannual
Attendance on ICN webinar	attends:webinar	Manual	Bimonthly
Tweet at ICN	tweet_at_icn:twitter	Automatic	Continual
Post to Facebook page	comments:facebook_page	Automatic	Continual
Member of PAC	participates:pac	Manual	Bimonthly
Member of PWG	participates:pwg	Manual	Bimonthly
Patient experience	participates:patient_experience	Manual	Ad hoc
Follows through on a campaign ask (eg, t‐shirt campaign)	participates:engagement_action	Manual	Ad hoc
Attends a local event	attends:local_center_event	Manual	Ad hoc
Answers survey	answered_survey	Automatic	Ad hoc
Participates LOOP	participates:loop	Manual	Monthly
Joins engagement bootcamp taskforce	joins:bootcamp_taskforce	Manual	Ad hoc
Attends bootcamp training	attends:bootcamp	Manual	Ad hoc
Reviews bootcamp training	reviews:bootcamp	Manual	Ad hoc
Joins a group (research, innovation project, etc)	joins:project_group	Manual	Ad hoc
Attends 1:1	attends:one‐on‐one	Manual	Ad hoc
Downloads an IBD resource (toolkit, webinar, etc)	downloads:ibd_resource	Manual	Continually
Reads a loop post (tracked through email clicks)	reads:loop_post:website	Manual	Weekly

TABLE 3.

Codes denoting “contribution” in ICN

Action	Code	Updates	Frequency
LOOP contributor	contributes:loop	Manual	Ad hoc
eNewsletter Contribution (local, pwg, pac)	authors:enews_email_blast	Manual	Ad hoc
Submit myICN story	submits:story_card	Manual	Ad hoc
Contributes to a broad range of PWG activities	contributes:pwg_activities	Manual	Ad hoc
Contributes to a broad range of PAC activities	contributes:pac_activities	Manual	Ad hoc
PWG Workgroup member (communications, outreach, etc)	contributes:pwg_taskforce	Manual	Bimonthly
PAC Workgroup member (communications, outreach, etc)	contributes:pac_taskforce	Manual	Bimonthly
Plan community conference, webinar, engagement fair, etc	contributes:event	Manual	Ad hoc
Donate to ICN	donates:improvecarenow	Automatic	Ad hoc
Submit a proposal (research, innovation fund, etc)	submits:proposal	Automatic	Ad hoc
Edits bootcamp training	edits:bootcamp	Manual	Ad hoc
Presents bootcamp training	presents:bootcamp	Manual	Ad hoc
Holds 1:1 conversation	holds: one‐on‐one	Manual	Ad hoc
Joins engagement innovation community	joins:engagement_innovation_community	Manual	Annually
Provides feedback (engagement)	provides_feedback:engagement	Manual	Ad hoc
Provides feedback (research)	provides_feedback:research	Manual	Ad hoc
Provides feedback (QI)	provides_feedback:qi	Manual	Ad hoc
Provides feedback (other)	provides_feedback:other	Manual	Ad hoc

TABLE 4.

Codes denoting “ownership” in ICN

Action	Code	Updates	Frequency
Event organizer (fundraiser, IBD day, house party)	organize:event	Manual	Ad hoc
PWG leaders	leads:pwg_activities	Manual	Bimonthly
PAC leaders	leads:pac_activities	Manual	Bimonthly
Innovation community leader	leads:engagement_innovation_community	Manual	Ad hoc
Joins ICN leadership group	icn:leadership_group	Manual	Ad hoc
Bootcamp taskforce leader	leads:bootcamp_taskforce	Manual	Ad hoc
Designs bootcamp	designs:bootcamp	Manual	Ad hoc
Leads a project or a group (research team, innovation project)	leads:project_group	Manual	Ad hoc
Leads engagement innovation community	leads:engagement_innovation_community	Manual	Ad hoc

2.4. Application of codes

Codes are assigned to individuals based on their actions. For example, individuals signing up to receive ImproveCareNow's eNewsletter (the CIRCLE eNewsletter) receive two separate tags (signs_up:homepage_sidebar, which details how they signed up, and receives:email_circle, which details they will receive an eNewsletter). When an individual receives an email and downloads an IBD resource, they will then receive a new tag, such as downloads:ibd_resource. Individuals who attend one or more events (eg, monthly webinars) are tagged for each event with the date of the event appended to the code (eg, attends:webinar_network:virtual:03‐04‐29). To ensure inter‐rater reliability, all codes are reviewed for accuracy and validity by a different person trained in the methodology and coding disagreements are resolved by consensus. New tags and their assignment to engagement tiers are considered and agreed upon by ICN researchers on a periodic basis (eg, quarterly). For instance, if a new activity is discovered, a tag following the established conventions will be proposed, along with the appropriate tier. These are reviewed by team members before being adopted.

2.5. Code storage

Engagement codes are entered manually by project personnel or automatically (in the case of social media and website interactions). ICN utilizes a commercial online database (https://nationbuilder.com/), to store person‐based, dynamic profiles (eg, contact info, actions taken on the ICN website, interactions with project team) and social matching (eg, links public social media profiles to email addresses in the ICN database). The database includes filtering capabilities and allows data export in different formats, enabling manipulation and visualizations of the data outside the database user interface. Data generated from social media and website interactions are logged continuously.

2.6. Visualizing and disseminating results

Counts of the number of individuals at each level of engagement at individual care centers and across the entire ICN network are computed from database queries. These counts are assembled into center‐level and network‐wide longitudinal plots of the number of individuals at each engagement tier. Plots and selected descriptive statistics are disseminated monthly to stakeholders (eg, ICN executive directors, principal investigators, project managers, quality improvement coaches) by email.

2.7. Analytic approach

The database is queried to retrieve and count the number of records containing behavior codes meeting each engagement tier definition. These numbers are tracked over time to visualize and statistically characterize trends and assess center‐level interventions. Summaries are distributed to stakeholders via email.

2.8. Ethics statement

This work was undertaken to improve ongoing engagement activities in ImproveCareNow, and not to produce generalizable knowledge applicable to other CLHSs. As such, it constituted operational improvement activities that are exempt from ethics review. The primary purpose of this report is to share lessons learned from implementation. Data depicted in this report are governed by the ICN CIRCLE community data privacy policy, available at https://www.improvecarenow.org/privacy_policy.

3. RESULTS

Application of this system is illustrated below by three illustrative use cases.

Illustrative Use Case #1: Engagement Behavior of Individuals. Figure 1 shows the progressive engagement behavior of a theoretical person in ICN. This individual became aware by signing up for ICN's CIRCLE eNewsletter on the ICN website and became a participant when they attended a Patient Advisory Council (PAC) monthly webinar. Their transition to contributor came when they published a post to ICN's blog (https://www.improvecarenow.org/loop). Finally, they became an owner when they took over as the lead of a PAC taskforce.

FIGURE 1.

Sequence of engagement behavior of a theoretical person in ICN, and the corresponding codes

Illustrative Use Case #2: Intervention Assessment. Beginning in April 2018, ICN began testing methods for patients and families to become aware in the clinical environment. Two interventions were assessed to subscribe patients and parents to the ICN eNewsletter: (a) approaching patients and parents during clinic appointments with paper subscription cards to fill out manually and (b) an analogous approach utilizing electronic tablets. In the former case, written information was solicited on the cards and transcribed by volunteer staff into the eNewsletter web site to trigger subscriptions, whereas in the latter approach patients or parents entered their information into the eNewsletter site directly. Figure 2A,B shows the results for two centers that utilized tablets. The number of patients aware at each center is plotted in black as a function of time. Increases were observed at both centers (the intervention began in September, 2018 at Center A and in February, 2019 at Center B). Figure 2C,D shows the results for two centers that used paper cards. Here, we see similarly strong increases in awareness at Centers C and D. In red we depict the efficiency of the interventions (ie, the fraction of people who were asked who actually signed up). In the case of the tablet interventions, the efficiency is generally >70%, whereas for cards it is generally <40%.

FIGURE 2.

Assessment of an interventions to increase engagement at four different care centers. Black: sign up numbers. Red: percent asked who signed up. Centers where affiliated with academic medical centers and had different sizes (estimated patient populations: A = 113, B = 518, C = 980, D = 774)

Illustrative Use Case #3: Network‐wide engagement over time. In Figure 3, we illustrate the growth of the different engagement tiers for the ICN network since August of 2017. Note several features are evident on the plot, including those due to interventions designed to increase patient/parent awareness (described above in Use Case #2). The discontinuity visible in the participation curve in January to February of 2018 is due to new tracking of one‐on‐one meetings between engagement staff and patient or parent volunteers.

FIGURE 3.

The growth of engagement tiers for the ICN network since August of 2017

4. DISCUSSION

We have described an approach to measure, record, and longitudinally track patient and parent engagement in the ImproveCareNow CLHS. The approach utilizes manual and automated observation of individual engagement‐relevant behaviors, which are recorded in a coding schema and mapped into engagement tiers. Higher tiers represent increasing awareness, use, and eventually creation and sharing, of information, knowledge, and knowhow to improve health and healthcare. The resulting system enables longitudinal monitoring of patient and parent engagement in ICN. Illustrative use cases show how data are used to assess interventions (such as using cards vs tablets to enroll patients and parents to increase awareness) and to monitor how network‐level engagement data change through time. We are not aware of similar approaches in other CLHSs.

Data collected in this system thus enables learning related to engagement throughout ICN. By following individuals longitudinally, engagement‐relevant behaviors are monitored, providing data to inform hypotheses about what activities result in the most engagement and potential interventions to increase engagement (eg, by providing needed information and services ²² or making it easier to access information and services). Data from the system is also useful for evaluating center‐level experiments testing such interventions. This system likewise enables use of center‐level QI methodologies to increase the reliability and effectiveness of engagement initiatives, and supports the setting of center‐level engagement targets (eg, quarterly targets defined by extrapolation of engagement trends).

When aggregated across centers, such data can be pooled to provide network‐level engagement statistics. Our data suggest that, in ICN, there is a decreasing number of individuals at increasing levels of engagement: the number of individuals classified as aware outnumber those classified as participants; who outnumber those contributing; who outnumber those owning ICN activities. This suggests face validity of the prototype, as it is generally consistent with a well‐described power law distribution in which a few very active users account for the majority of contributions to several online peer‐production platforms. ²³

In building this system, we adapted an approach used in political community organizing, which, like ICN, often relies on volunteers becoming progressively engaged in campaign activities. ¹⁵ , ²⁴ In addition to the capabilities and uses highlighted above, the approach may be especially powerful when data are combined with data from other network resources. For example, if combined with information from the ICN patient registry to compute denominators, it may be possible to estimate measures such as engagement prevalence and engagement incidence in ICN, thereby enabling quantitative engagement monitoring and intervention evaluation. As another example, the availability of reproducible, longitudinal engagement data may make it possible to investigate potential relations between engagement and clinical outcomes. The degree to which engagement, as defined in this report, is associated with patient activation (demonstrated skills, knowledge, and motivation to effectively manage one's health and participate in health care decisions ²⁵ ) and improved patient clinical outcomes is currently unknown, though engagement at levels of participation and higher might align with activation and similar processes associated with improved outcomes. If such an association exists, engagement promotion may prove to be another tool in the armamentarium available to improve outcomes.

ICN began in 2007 and has evolved since its founding. Over time, the need for measures describing ICN function and improvement opportunities were recognized and work undertaken to address such needs. At this time, it is not possible to accurately or meaningfully estimate the resources spent on devising the approach described in this report. Nonetheless, our purpose in describing the approach is so that others may undertake analogous work to address their needs, given their unique circumstances and resources available. Indeed, as new CLHSs come into being, and as existing networks continue to grow, an organized, reproducible approach is needed for monitoring engagement. A system such as has been described here is especially important as centers grow, center personnel leave the system taking their memories and insights with them, and as additional centers join the network. Strengths of our approach include that, because the tiers of engagement are well defined in terms of observable behaviors, codes should be reproducible through time and when different people code observations. By using an online database, data can be entered by different people at different locations, making it possible to scale the approach to more and more care centers. The approach provides detailed data that can be used for many purposes, including investigating predictors of engagement.

The approach has limitations, including incomplete adoption of the database by all ICN care centers, potential under‐reporting of engagement activities and observations, and potential lags in manual reporting and data entry. An important limitation is that the approach cannot estimate engagement if individuals are not doing activities that are observable and thus able to be coded. This is likely more of an issue at the awareness tier than at higher tiers. Nonetheless, the approach described in this report provides a pathway toward engagement measurement and tracking in a CLHS, a pathway that can be developed further and enhanced in the future. It is our hope that other CLHSs will be able to adapt and advance the approach described here.

CONFLICT OF INTEREST

Michael Seid and Peter A. Margolis declare a financial interest in intellectual property licensed by CCHMC to Hive Networks, a company developing information technology to support collaborative learning health systems. Authors David M. Hartley, Christian Keck, and Mary Havens declare that they have no conflicts of interest.

Hartley DM, Keck C, Havens M, Margolis PA, Seid M. Measuring engagement in a collaborative learning health system: The case of ImproveCareNow. Learn Health Sys. 2021;5:e10225. 10.1002/lrh2.10225