The Cystic Fibrosis Learning Network: A mixed methods evaluation of program goals, attributes, and impact

Abstract

Introduction

The Cystic Fibrosis (CF) Foundation sponsored the design, pilot testing, and implementation of the CF Learning Network (CFLN) to explore how the Foundation's Care Center Network (CCN) could become a learning health system. Six years after the design, the Foundation commissioned a formative mixed methods evaluation of the CFLN to assess: CFLN participants' understanding of program goals, attributes, and perceptions of current and future impact.

Methods

We performed semi‐structured interviews with CFLN participants to identify perceived goals, attributes, and impact of the network. Following thematic analyses, we developed and distributed a survey to CFLN members and a matched sample of CCN programs to understand whether the themes were unique to the CFLN.

Results

Interviews with 24 CFLN participants were conducted. Interviewees identified the primary CFLN goal as improving outcomes for people living with CF, with secondary goals of providing training in quality improvement (QI), creating a learning community, engaging all stakeholders in improvement, and spreading best practices to the CCN. Project management, use of data, common QI methods, and the learning community were seen as critical to success. Survey responses were collected from 103 CFLN members and 25 CCN members. The data revealed that CFLN respondents were more likely than CCN respondents to connect with other CF programs, routinely use data for QI, and engage patient and family partners in QI.

Conclusions

Our study suggests that the CFLN provides value beyond that achieved by the CCN. Key questions remain about whether spread of the CFLN could improve outcomes for more people living with CF.

1. INTRODUCTION

Learning health networks are multi‐center collaborations that support system learning and are designed to drive continuous quality improvement (QI). ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ Although networks vary in form, size, shape and scope, they commonly: foster collaboration between healthcare professionals, patients and families, and researchers; connect people across geographic regions and clinical disciplines to share knowledge, resources, and best practices; drive goal‐directed actions toward a shared vision and mission; and provide systems for data sharing and knowledge dissemination. Participation in a learning health network can improve health outcomes. ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵

The Cystic Fibrosis (CF) Care Center Network (CCN), accredited by the US CF Foundation (CFF), includes 286 programs that deliver care to nearly 40 000 patients. ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ Patient‐level process and outcome measures are captured during all clinical encounters and entered into the CFF Patient Registry (CFFPR). ²⁰ Near real‐time patient and program‐level reports are accessible to care teams. ²¹ Patient‐level reports may be reviewed by care teams prior to patient visits and shared with patients during visits. Program‐level data can provide population monitoring of process and outcome measures over time.

Starting in 2014, CFF sponsored the design, pilot testing, and subsequent implementation of the CF Learning Network (CFLN). ²² The goal was to explore how the current model of care across the CCN ¹⁶ , ¹⁸ could become a robust learning health system. ²³ The CFLN is comprised of 36 programs from among the 286 in the CCN. The CFLN is a Collaborative Learning Health System, ²⁴ also called a Learning Health Network. Learning Health Networks use an actor‐oriented network organizational architecture ²⁵ to enable all stakeholders to engage in collaborating at scale to improve health and health care. The CFLN implemented a set of interventions based on the Network Maturity Grid ²⁶ to facilitate the involvement of stakeholders; structures, processes, and protocols that facilitate collaboration; and a commons where participants can create and share resources. ²² Participating programs agree on a shared purpose, develop local SMART aims, and use The Model for Improvement (MFI) ²⁷ , ²⁸ to structure learning and tests of change.

While many programs in the CCN apply QI methods to address gaps in process and outcome measures, ¹⁷ these efforts are localized to the individual program ²⁹ which contrasts with the coordinated learning approach supported by the CFLN. For example, CCN programs participating in the 12‐month Learning and Leadership Collaborative ²⁹ set program‐specific improvement goals and tested localized interventions related to coordination of interprofessional care during clinic visits. The CFLN, which supports orchestrated learning and multi‐program QI, tested care coordination interventions across multiple sites until all programs collectively reached 80% reliability in process improvement. ³⁰ By intentionally organizing learning structures and data‐driven testing across multiple programs to the point of process reliability in the CFLN, spread and scale is possible across the diverse settings of the CCN.

Prior evaluations of the CFLN have focused on network maturity and impact on chronic care processes and indicators of a collaborative infrastructure. ²² , ³⁰ However, little is known about participants' perceived value of the network, the impact of the network on becoming a CF learning health system, or the complexity of implementing the network in different settings.

2. QUESTIONS OF INTEREST

Given the importance of structures, processes, and participant roles to facilitate collaboration, we sought to: (1) understand CFLN participants' grasp of program goals and key attributes. To appreciate the ways these changes are experienced by participants, we sought to: (2) assess the perceived impact of the CFLN, including relative advantage compared to the CCN. To inform future implementation and scaling of the CFLN, we sought to: (3) assess whether participant perceptions vary by program and respondent characteristics.

3. METHODS

3.1. Phase 1: Semi‐structured interviews to elicit program theory

We conducted semi‐structured interviews of health professionals and patient and family partners (PFPs) engaged in the CFLN. We used purposive sampling to identify 25 key stakeholders with different skill sets and longevity in the CFLN.

The interview guide (Supplemental Appendix (Appendix S1)) was informed by the TIDieR framework ³¹ and included questions about perceived goals of the CFLN, essential components and how they were expected to support goal achievement, potential impact, and opportunities for improvement.

Forty‐five minute interviews were conducted between October 25, 2021 and December 9, 2021 by two authors (A.V.C., J.K.). Interviews were recorded and transcribed. Transcripts were anonymized to remove identifying details. Detailed field notes were created during the interview and processed immediately after each interview to highlight themes related to the TIDieR framework. To ensure accuracy and limit bias, field notes were shared with each interviewee and corrections were made as indicated.

Initial interview codes and themes were developed using an inductive approach and reviewed based on the constant comparative method. ³² Codes were independently applied to 100% of field notes by A.V.C. and J.K., and applied to relevant portions of transcripts that illustrate these themes. Differences in coding were reviewed and final coding decisions were made via consensus. Qualitative analyses were conducted using Atlas.ti version 9.1.3.

3.2. Phase 2: Electronic survey to explore learning network impact

The second phase sought to understand the extent to which CFLN participants endorsed themes identified in phase 1 and to identify potential differences in perceptions between CFLN participants and a matched sample of individuals from 36 CCN programs not participating in the CFLN. Programs were matched on type (adult, pediatric) and size (small: 0‐70 patients; medium: 71‐140 patients; and large: 141+ patients).

Thematic analysis of phase 1 interviews informed survey content (Supplemental Appendix (Appendix S1)). Topics for both CCN and CFLN respondents included: involvement in QI, use of data to support improvement, and contributions of PFPs to QI. Topics for only CFLN respondents included: inclusivity, networking, communication, efficiency, resources and research, impact, and future directions of the CFLN.

CFF leaders sent invitations on behalf of the study team. Invitations were sent to: (a) the leadership team at each CFLN program, including physician leaders (n = 39), QI leaders (QILs) (n = 53) and PFPs (n = 85), and (b) leaders from matched CCN programs, including physicians named as CF Program Directors (n = 37) and nurses or other multidisciplinary team members named as CF Program Coordinators (n = 41). To enhance response rates, three reminders were sent. Respondents completed the survey via Qualtrics (www.qualtrics.com) between February 10, 2022 and March 10, 2022. CCN leaders were asked to share the survey invitation with patient/family partners or advisors.

Respondents were asked to identify their program, which was used to derive program characteristics from CFF databases (eg, size, region, program type, and date of joining the CFLN). When program name was not identified, location of the survey respondent was used to match respondents to programs, where possible.

Data were summarized with descriptive statistics. Surveys with complete responses to core questions (ie, asked of CCN and CFLN respondents) were included in comparative analyses. Variables rated on Likert scales were reported as the proportion reporting often/always or strongly agree. Differences in proportions between CCN and CFLN respondents were identified via chi‐square tests and Fisher Exact Tests. We used a P‐value threshold of <.05 to identify significant differences between groups, using two‐sided significance tests. Quantitative analyses were conducted using SPSS version 28.

Secondary analyses were used to identify variation by program size (small, medium, large), program type (adult, pediatric), and respondent type (physician leaders/CF Program Directors, QILs/CF Program Coordinators, and PFPs/advisors). Three CFLN respondents identified as both PFPs and QILs and were classified as QILs to create non‐overlapping respondent sub‐groups.

3.3. Human subjects approval

The study was approved by the Dartmouth College Committee for the Protection of Human Subjects (Study: 32372). An information sheet was used to describe the study and research procedures. A script was used to describe the research and obtain verbal consent for the interview. By completing the survey, respondents consented to participate.

4. RESULTS

4.1. Participants

We interviewed 24 CFLN members (96% response rate), including CFLN faculty/sponsors (n = 5) and staff (n = 3), physician leaders (n = 6), QILs (n = 4), and PFPs (n = 6). One physician leader did not respond to the invitation. Participants' experience with the CFLN varied, with some having participated since the design phase in 2014 and others joining between 2016 and 2020 (cohort 1: n = 7, cohort 2: n = 4, cohort 3: n = 5). Team members were distributed between adult (n = 8) and pediatric (n = 7) programs; and midwest (n = 5), northeast (n = 5), southeast (n = 2), and west (n = 3) regions of the United States. The median interview length was 46 minutes (range: 41 to 54 minutes).

Surveys were completed by 103 CFLN members (30 physicians, 26 QILs, 43 PFPs, 4 other members of the QI team) and 25 CCN members (15 CF Program Directors, 10 CF Program Coordinators). The overall response rate was 50%, with higher rates among CFLN members (58%) than CCN members (32%). Response rates in both groups were highest for physicians (CFLN: 76.9%; CCN: 40.5%), followed by PFPs (CFLN: 55.1%), and QILs/program coordinators (CFLN: 53.1%, CCN: 24.4%). Although 25 of the 36 CCN programs invited to participate reported an active Patient and Family Advisory Council or Board in 2021, no CCN patient/family partners or advisors participated. Demographic characteristics between CFLN and CCN were similar for program type, size, and geographic region. CFLN respondents had more experience participating in CFF‐sponsored QI programs (excluding the CFLN) in the last 10 years than CCN respondents (Supplemental Appendix: Table 1).

4.2. Program theory

4.2.1. CFLN global aim

There was strong consensus among interview respondents on the primary goal of the CFLN: to improve clinical and quality of life outcomes for people with CF. There was also consensus on secondary goals to achieve this aim, including: training in QI methodology, developing a learning community, engaging all stakeholders (including PFPs) in changing care practices, and spreading learnings to the CCN. Goals were seen as stable over the course of the CFLN; however, participants frequently noted that goals had become increasingly clear and focused over time.

4.2.2. Perceived critical features of CFLN success

Several themes emerged from the interviews as critical to the CFLN's success. Illustrative quotations are included in Table 1.

TABLE 1.

Quotations to illustrate the most critical components of the CFLN

Theme	Illustrative Quote
Project management and operational offerings	“When they have programs that run 10 to 12 weeks, we have deadlines, we have meetings, we have coaches. It seems everyone is more engaged and everyone is accountable to do something for the project. And I think we work better when we have a timeline like that.”—PFP, #14 “The community conferences are fabulous. First, they give training. We get to learn from each other to varying degrees. It's also a focused time, opposed to an hour a week or an hour every week or a half hour, whatever it is. The team can really spend some serious time together, which just is not manageable in any other context.”—Physician leader, #17
Learning community	“[The CFLN] creates venues for rapid collaboration. So it's not only collaboration in general, it's rapid collaboration where you can put together multiple teams in a very short period time and move projects really quickly. […] it's a lot easier to understand if the teams are on the same page, if they are working on the same projects, if someone is innovating in a different field and the regular meetings kind of allow that information to pass through.”—Physician leader, #2 “They really provided a space and a venue for teams to connect and have community around their innovations with telehealth, and share ideas as far as, this is working in my center, your center is really struggling with this, maybe you can steal some of our ideas, and we all went back and forth.”—QIL, #8
Use of data	“The support of the CFLN in their collection of data and reporting of data using CF Smart Reports to give us access to the data. […] We can see it, we can use it, we can come up with our own projects based on that data outside of the Learning Network if we wanted to.”—QIL, #23 “It has to have data and it has to have a way for people to look at and share data.”—CFLN Leader/Sponsor, #7
QI training and tools	“It's increased our confidence and our ability to actually use concrete tools, rather than just kind of doing things in a less structured manner.”—QIL, #13 “You cannot do the work without learning the tools. So, I think it's essential that the CFLN can teach us the skills as we are working on the project, and coach us, and boost us up a little bit, graduate us into a higher level of care.”—QIL, #21
Triad leadership structure	“Having that triad leadership of patient and family partner, a QI leader, and a physician leader has really distributed the work to participate. But also given them the opportunity to grow in their own skills and their own knowledge of improvement work.”—CFLN leader/sponsor, #3 “The Network Leadership Team is very essential and having that headed up by clinicians and patient family partners and people who are on the ground doing this work […] Being able to all come together in that capacity is essential to moving the network forward.”—QIL, #8

Project management and operational support

Interview respondents reported that project management and operational support enables distributed CFLN teams to advance improvement objectives. They described the operations team (including a program manager, five project coordinators, three QI specialists, and two data analysts) as well‐organized, supportive, and knowledgeable. They stated that this team holds programs accountable for meeting reporting deadlines and goals and sets the pace and discipline of the network via action period calls, monthly Network Leadership Team meetings, and calls for physicians, QILs, and PFPs.

Learning community

Interview respondents reported that the CFLN learning community provides opportunities to collaborate, share, learn, and celebrate with peers around a shared purpose, common goals, and specific projects. They stated that sharing data, resources, and best practices supports dialogue across programs to determine how to improve care. They identified several learning opportunities, including communities of practice (eg, timely data entry), 180‐day challenges, Innovation Labs (eg, telehealth), regular webinars, and Community Conferences.

Use of data

Interview respondents reported that reviewing, sharing, and using data supports rapid‐cycle improvement at local and network levels. They stated that timely data entry into the CFFPR allows members to identify changes in health outcomes linked to improvement initiatives. They also reported receiving training on how to analyze and use data over time to evaluate impact of QI projects.

QI training and tools

Interview respondents reported that QI training and tools enable teams to quickly identify and address gaps in process and outcome measures and rapidly pivot to address evolving challenges. They stated that initial and ongoing training in the MFI and use of tools, such as key driver diagrams, supports improvement projects.

Leadership structure

Interview respondents reported that the triad leadership structure  ²² (physician leaders, QILs, and PFPs) supports governance and guides decision‐making and prioritization of local and network level activities. They stated that this integrated leadership structure models that seen with coproduction. ³³ , ³⁴ , ³⁵

4.3. Perceived current and potential impact of the CFLN

Several themes emerged from interviews on CFLN impact and are illustrated in Table 2. Interview respondents indicated that the CFLN fostered a learning community focused on QI and a culture of coproduction; it was seen as improving process and outcome measures. Respondents saw potential impact for improved use of data and spread of best practices to the CCN. These themes were explored in survey responses from CCN and CFLN respondents.

TABLE 2.

Perceived current or potential impact of the CFLN

Theme	Illustrative quote
Learning community focused on QI	“The centers that are part of the CFLN create areas of focus on pertinent patient outcomes that really revolve more around the delivery of care. And whether that's telehealth, quality of life initiative, recognizing exacerbations earlier. Those were all things that people might have been doing at one center, but within the Learning Network, it allowed multiple centers to come together. And when you have multiple centers working together, it allows you to be able to see the differences and you have the power of the population behind you at that point.”—Physician leader, #16 “Having the local experience helps the network, having the network experience helps the local team.”—Physician leader, #19
Culture of coproduction	“Our crowning jewel is co‐production at the team level. We've been able to really help our patient and family partners not just be advisors, but to be designers, to be leaders.”—CFLN staff, #4 “Coproduction is probably what's been the biggest change for us. […] the concept of coproduction has been very small scale at our clinic. But it's such a central feature to the Learning Network. That value has really started infusing throughout the clinic for us.”—QIL, #13
Improving process and outcome measures	“Improving patient's care, improving their quality of life. And for me personally, that's been very gratifying because I see those outcomes. It's not like I'm reading about them or hoping for them to happen. I see the actual outcomes because after the project is over and we are implementing them, it's sustained.”—PFP, #14 “A successful CFLN would lead to improved health‐related quality of life, enhanced relationships between patients and care teams, enhanced relationships among care team staff, and improved health equity within our community.” QIL, #8
Spread of best practices	“Having the number of centers we have in the CFLN is wonderful, but I feel like the CFLN is not successful until we have made changes to care across all centers… I think there needs to be some dissemination throughout the entire network, not just network, but community for CF patients.”—QIL, #23 “What we learn and what we adapt should not be for the 33 teams of the CF learning network, we need to really spread that to the rest of the care center network. And, that could be adding more teams, but to be honest it has to then be in some way we spread these tools and techniques in a national commons, in a national resource.”—Physician leader #19

4.3.1. Learning community focused on QI

Interview respondents saw the CFLN as supporting a learning community of people focused on QI and strengthening local QI practices. As shown in Table 3, survey responses between CCN and CFLN clinical team members were largely similar with respect to length of time involved with QI, use of QI frameworks, proportion of interdisciplinary team receiving QI training, team members involved in QI, and sources used to learn about QI happening elsewhere. CFLN clinical team members were more likely than those in the CCN to report “often” or “always” learning about QI work happening elsewhere through team meetings with a QI expert (P = .023) or making connections with people from other CF programs working on similar QI projects (P < .001).

TABLE 3.

Quality improvement and coproduction behaviors between respondents participating in the CFLN and respondents from the larger CCN

	CCN care team respondents (n = 25)	CFLN care team respondents (n = 60)	P‐value
Time involved in quality Improvement; n (%)			NS
0 to 2 years	2 (8.0)	6 (10.0)
3 to 5 years	9 (36.0)	16 (26.7)
6 or more years	14 (56.0)	38 (63.3)
QI frameworks used; n (%) often/always
Model for Improvement	10 (66.7)	42 (81.1)	NS
Clinical Microsystems	12 (60.0)	34 (64.2)	NS
Lean	7 (38.9)	6 (13.3)	0.038
Six Sigma	2 (11.8)	4 (9.1)	NS
Proportion of multidisciplinary team receiving QI training; n (%)			NS
1 to 25%	5 (20.0)	4 (7.1)
26 to 50%	6 (24.0)	15 (25.0)
51‐75%	5 (20.0)	20 (33.3)
76‐100%	7 (28.0)	17 (28.3)
Do not know	2 (8.0)	4 (6.7)
Team involved in program's QI work; n (%)
Physicians	23 (92.0)	99 (96.1)	NS
Dietitians	24 (96.0)	95 (92.2)	NS
Social workers	24 (96.0)	89 (86.4)	NS
Patient and family partners	17 (68.0)	97 (94.2)	<0.001
Respiratory therapists	21 (84.0)	83 (80.6)	NS
Nurses	21 (84.0)	91 (88.3)	NS
Advanced practice providers	14 (56.0)	67 (65.0)	NS
Pharmacists	14 (56.0)	59 (57.3)	NS
Behavioral health specialists	13 (52.0)	51 (49.3)	NS
Trainees in health fields (e.g., resident/fellow)	12 (48.0)	36 (35.0)	NS
Administrative staff	7 (28.0)	40 (38.8)	NS
Quality improvement advisors	7 (28.0)	27 (26.2)	NS
Data analysts	3 (12.0)	26 (25.2)	NS
Not sure who has been involved	3 (2.9)	0 (0.0)	NS
Sources used to learn about QI work happening at other CF programs; n (%) often/always
National conferences	20 (80.0)	50 (83.3)	NS
During team meetings with a QI expert (local QI resource, external coach)	6 (24.0)	30 (50.8)	0.023
Peer‐reviewed publications	12 (48.0)	22 (37.3)	NS
Communication with peers at other institutions	6 (24.0)	26 (43.3)	NS
CFF e‐mail Listserv	11 (4.40)	22 (37.3)	NS
CFF newsletters	7 (28.0)	19 (31.7)	NS
Discipline‐specific/regional conferences	8 (32.0)	18 (30.5)	NS
Improvement readiness; n (%) positive culture a	17 (68.0)	51 (85.0)	NS
Resources to support QI, n (%) often/always
I can find the resources and tools I need to conduct QI projects	16 (64.0)	48 (80.0)	NS
Training in QI methods is available to me and my team members when needed	13 (56.5)	42 (70.0)	NS
I have made connections with people from other CF programs who are working on similar QI projects	7 (28.0)	41 (68.3)	<0.001

^a Positive “Improvement Readiness” culture is defined by an average score of 4 or higher across five questions rated on a 1‐5 Likert scale: The learning environment in my work setting: (1) Utilizes input / suggestions from the people who work here; (2) Integrates lessons learned from other work settings; (3) Effectively fixes defects to improve the quality of what we do; (4) Allows us to gain important insight into what we do well; (5) Is protected by our local management.

Within the CFLN, PFPs reported less experience with QI than clinical team members. They were less likely than clinical team members to “often” or “always” learn about QI work happening elsewhere through conferences, publications, or CFF resources; find tools/resources to conduct QI projects; or report making connections with people from other programs (Supplemental Appendix: Table 2).

4.3.2. Use of data

CFLN interview respondents identified data‐driven improvement as a critical feature of a successful learning network but identified opportunities to improve the use of data. Survey responses identified significant differences between CCN and CFLN care team respondents with respect to processes that support timely data entry, availability of timely data, and routine sharing of data with all members of the improvement team (Table 4). Within the CFLN, pediatric program respondents were more likely than those from adult programs to “often” or “always” share performance data with institutional leaders (68%, n = 23 vs 37%, n = 7, P = .044). There were no other differences in access to or use of data by program type, program size, or respondent type.

TABLE 4.

Perceptions of data use by care teams in the CCN and CFLN

	CCN care team respondents a (n = 25)	CFLN care team respondents a (n = 61)	P‐value
Data access, n (%) often/always
Process to support timely data entry	17 (68.0)	58 (96.7)	<0.001
Data I have is timely	17 (68.0)	58 (95.1)	<0.001
Easily access the data I need to conduct quality improvement projects	17 (68.0)	52 (85.2)	NS
Data display and interpretation, n (%) often/always
Can create run/SPC charts to graph key outcomes of our site performance	13 (52.0)	36 (61.0)	NS
Provided with run/SPC charts to graph key outcomes of our site performance	6 (26.1)	29 (49.2)	NS
Data is presented to me in a way that is easy to interpret	14 (58.3)	48 (78.7)	NS
Data into action, n (%) often/always
Data is presented to me in a way that informs our next steps in QI	13 (56.5)	47 (77.0)	NS
Data is routinely shared with all members of our improvement team	16 (66.7)	58 (95.1)	<0.001
Data is shared with institutional leaders	10 (45.5)	31 (54.4)	NS
I know how our local CF program performs on key measures compared to other programs within the CF community	16 (66.7)	48 (80.0)	NS

^a Analyses are limited to care team members, including CF program directors or physician leaders and CF program coordinators and QILs.

4.3.3. Culture of coproduction

The CFLN was seen by interview respondents as creating a cultural shift toward coproduction, with PFPs actively engaged in improving healthcare service delivery at the local and network level. Some respondents noted a transformation in the level of involvement of PFPs while others indicated a long history of deep engagement with PFPs. Interview respondents often noted that PFPs were engaged from the start of CFLN projects and informed intervention designs (including goals, measures, and key drivers); some PFPs were leading QI projects.

Survey responses (Table 5) confirmed these themes. CFLN clinical team members were twice as likely as those from the CCN to report that PFPs “often” or “always” actively participated in their team's QI efforts (89%, n = 54 vs 44%, n = 11). Among programs that included PFPs in QI, those in the CFLN were significantly more likely than those in the CCN to report that PFPs “often” or “always” were onboarded to QI aims, received training in QI methods, participated in QI team meetings, and received reimbursement for participating in QI. Nearly three‐quarters (74%, n = 34) of PFPs in the CFLN reported actively participating in QI.

TABLE 5.

Perceptions of coproduction between the CCN and CFLN

	CCN care team respondents (n = 25)	CFLN care team respondents (n = 60)	CFLN PFP respondents (n = 43)	P‐value*
Patient involvement in QI; n (%) often/always
PwCF and/or family members receive onboarding to understand QI project aims	6 (33.3)	38 (70.4)	29 (76.3)	.005 a NS b
PwCF and/or family members receive training in QI methods	3 (16.7)	29 (53.7)	29 (74.4)	.006 a .042 b
PwCF and/or family members participate in regular program QI team meetings	6 (30.0)	46 (80.7)	35 (87.5)	<.001 a NS b
PwCF and/or family members involved in QI receive some payment or reward for participating	0 (0.0)	45 (81.8)	27 (73.0)	<.001 a NS b

^*Missing data are excluded from analyses.

^a P‐value for comparisons between CCN care team respondents and CFLN care team respondents.

^b P‐value for comparisons between CFLN care team respondents and CFLN PFP respondents.

4.3.4. Improving process and outcome measures

The CFLN was perceived as improving process and outcome measures at the patient and program levels. Interview respondents perceived that the CFLN contributed to improved health outcomes (eg, lung function, quality of life) and care processes (eg, shared agenda setting for clinic visits). They indicated the CFLN improved individual and team professional growth and provided opportunities for science and research. Interview respondents reported that team functioning was enhanced by a greater understanding of how teams do their work; providing structure to team activities; and increasing team cohesiveness and efficiency.

Survey responses demonstrated that two‐thirds of CFLN members “strongly agreed” that the CFLN improved care processes locally (65%) and across the network (66%), and half of CFLN members “strongly agreed” that the CFLN improved health outcomes locally (50%) and across the network (52%). More than half of CFLN survey respondents “strongly agreed” that the CFLN was personally meaningful (68%), supported their ability to partner with a broader range of team members (60%), enhanced productivity of local CF programs (58%), and increased leadership skills (55%). Less than half “strongly agreed” that the CFLN supported their professional development goals (49%) or contributed to joy in work (44%).

4.3.5. Spread of best practices

Most CFLN interview respondents indicated the CFLN should spread best practices to the CCN. Survey responses (Figure 1) showed that three‐fifths (62%) of CFLN members “strongly agreed” that best practices were currently identified and shared within the CFLN. A smaller proportion “strongly agreed” that output from ideas tested within the CFLN was efficiently shared across CFLN programs (55%) or CCN programs (48%).

FIGURE 1.

Perceptions of CFLN members on network impact

4.4. Participant perceptions across contextually diverse settings

Interview respondents indicated that the ability to successfully engage in and benefit from the CFLN was associated with a variety of institutional characteristics. Interview respondents stated that successful CFLN teams had supportive local environments (eg, institutional support, resourcing, leadership); were interested and motivated to participate in QI activities; had stable care teams with low staff turnover; and had resources and protected time to engage in the CFLN. CFLN physician leaders often identified financial incentives from CFF as important for enabling them to engage in the CFLN. Respondents from some smaller programs reported that lower staffing levels and patient volume negatively impacted their ability to perform and interpret data from rapid cycle, small‐scale tests of change. In contrast, respondents from some larger programs identified the need for more targeted collaboration with other large centers, noting that their large size made it difficult to apply improvement processes that worked well in smaller centers.

Survey responses from CFLN members provided an opportunity to explore the effect of program characteristics on perceptions of CFLN operations and impact. Notably, perceptions of the CFLN often differed by size of program, with small programs reporting more positive views than either medium or large programs with respect to networking, efficiency, and impact (Supplemental Appendix: Table 3). There were no differences in perceptions of the CFLN between adult and pediatric programs.

4.5. Improvement opportunities

CFLN interview and survey respondents agreed that the CFLN was well‐positioned to support future QI efforts. Two‐thirds of survey respondents “strongly agreed” with feeling hopeful about the future of the CFLN (66%) and half “strongly agreed” that the CFLN was well‐positioned to accelerate improvements. However, few reported being highly informed about the future direction and vision of the CFLN over the next year (35%) or next 5 years (13%), and only half (52%) “strongly agreed” that they could provide input into the type of work being done by the CFLN (Figure 2). Small programs were more hopeful about the future than medium or large programs. There were no other differences in perceptions of the future by program size or type.

FIGURE 2.

Perceptions of CFLN members on network operations and future directions

CFLN interview respondents identified several opportunities for improving the CFLN. Operational changes included improved efficiency of meetings, of communication, and of data collection requests; stronger data analytic resources; and availability of an electronic platform to exchange information, ideas, and resources between programs. Less than half of survey respondents “strongly agreed” that CFLN meetings were efficient (46%) or that the pace (38%) or amount of time dedicated to the CFLN (32%) was sustainable. Only half “strongly agreed” that data collection tasks were relevant to the work of their program (52%). Notably, small programs were significantly more likely to “strongly agree” with each of these statements, compared to medium or large programs (P < .05) (Supplemental Appendix: Table 3). There were no differences in perceptions by program type.

Interview respondents also identified additional improvements, including structural changes to support self‐organization of CFLN QI teams, individualized support for local QI teams, a sharper focus on a small number of health outcomes, greater ability to conduct research across the network, and a refined leadership structure to streamline decision‐making.

5. DISCUSSION

CFLN stakeholders demonstrate a cohesive collective understanding of the CFLN's goals, attributes, and perceived impact across a variety of contexts. Several factors were seen as critical to the CFLN's ability to successfully improve outcomes for people living with CF. These included developing a community of learners with common purpose (supported by project management and collaborative learning opportunities), providing QI training and resources, using data to drive improvement, and engaging diverse clinical team members and PFPs in local and network‐level leadership of the CFLN. These findings are aligned with intended growth of the CFLN within the domains of the network maturity grid, specifically in engagement and community building, QI, data and analytics, and systems of leadership. ²⁶

Respondents perceived the CFLN as having an impact on four areas: the creation of a learning community focused on QI, a culture of coproduction, improvements in process and outcome measures, and potential for spreading CFLN knowledge within the CFLN, to the CCN, and to local QI efforts. Survey findings demonstrate that CFLN members have more positive responses than CCN respondents in networking, using data to drive improvement, and involving PFPs in QI activities.

Survey respondents from small CF programs were more likely to identify positive perceptions of networking, efficiency, and impact than those from medium or large programs. Variation across programs of different sizes may be associated with multiple factors. Data collection demands, local infrastructure and support, and CFF grant funding may influence the ability to successfully engage in, and benefit from, the CFLN. In contrast to small programs, less than a third of medium and large programs indicated that the pace or time dedicated to the CFLN was sustainable.

Our study identified novel findings that merit further exploration. Most CFLN survey respondents reported that the CFLN helped them become a better leader. Our evaluation was not structured to assess if improved leadership was linked to exposure to other effective leaders, role modeling, distributed leadership, or if exposure to the QI curriculum led to improved self‐efficacy and leadership skills. We are curious about how exposure to the CFLN might promote effective leadership. While CFLN members often reported that the pace of work and time dedicated to QI was unsustainable, most reported that CFLN participation enhanced local CF program productivity. Busy clinical teams likely benefit from interventions that “fit” with and enhance their already heavy clinical workflow. ³⁶ This suggests that careful attention to the design and deployment of learning network interventions may be critical to success, scale, spread and sustainability. Moreover, while most participants reported that the CFLN was personally meaningful, impact on joy in work was lower. We collected our data about 24 months into the COVID pandemic which may have confounded these findings.

5.1. Strengths and limitations

Our study had notable strengths and important limitations. First, purposive sampling for interviews allowed for identification of multiple domains of interest, which were then addressed by a more representative matched sample of CCN and CFLN respondents. Second, survey response rates were higher among CFLN members than CCN members, and among physicians than QILs/coordinators. Relative to CCN members' more variable exposure to structured QI training, ²⁹ the CFLN's ongoing QI learning environment may have increased response rates to a QI‐focused survey. Moreover, the survey request originating from CFF may have generated a higher response rate among physician leaders, who serve as the primary contact for many requests from CFF (eg, completing accreditation reports, program grants, and budgets). Third, the PFP recruitment methodology varied between CCN and CFLN programs. We were unable to track invitation numbers or response rates for PFPs within CCN programs. Our lack of PFP recruitment from CCN programs limits our ability to identify their perspectives on QI activities. Nonetheless, the structure, culture, and expectations of the CFLN is that patients and families have agency and are equal participants, which represents heightened engagement relative to traditional patient/family advisory panels. Fourth, this study focuses on subjective perceptions of the CFLN and its impact and responses may be impacted by social desirability biases and heightened awareness of QI themes promoted in the CFLN. Finally, our evaluation does not include objective measures of impact. Changes associated with specific CFLN initiatives are reported elsewhere (eg, telehealth). ²² , ³⁰

5.2. Future directions

Our findings identify several opportunities to strengthen the CFLN. First, CFLN participants desire greater and more timely access to data and enhanced capacity to analyze data. Other learning networks have developed real‐time, self‐service data access and demonstrated that members who access data more frequently have improved outcomes. ³⁷ Second, an effective ecosystem for sharing QI resources can support the efficiency and effectiveness of QI work and collaboration across programs, including opportunities for refresher training or resources in QI. ³⁸ , ³⁹ An ecosystem integrated with other CF systems could support spread of knowledge to the CCN. ³⁷ Third, team networking and collaboration are highly valued and can be enhanced by stronger connections among programs of similar size or facing similar challenges. Fourth, the CFLN should consider testing the most effective models for rapid dissemination of knowledge. ⁴⁰ Fifth, the CFLN should consider encouraging and supporting self‐organization of QI initiatives within the CFLN. ⁴¹ Sixth, individualized, on‐call “coaching” of centers by QI experts would be appreciated by some CF programs. Finally, the CFLN is primed to move beyond a focus on “process” outcomes toward impacting health and equity metrics. Other networks have evaluated their impact in novel ways not yet explored by the CFLN. ¹¹ , ¹³ , ⁴² , ⁴³ , ⁴⁴ The CFLN may consider: deploying robust research methods (eg, a priori comparator groups with different tiers of interventions, stepped wedge designs, ⁴⁵ and regression discontinuity designs ⁴⁶ ) to evaluate CFLN innovations; using network analysis to identify and leverage critical nodes and connectors within the CFLN; identifying novel contributions that influence real‐world practice and research; and measuring value of results using economic analyses.

Drawing from these results, CFF is committed to continuing to enhance an ecosystem that hosts data and resources and enables communication across the CCN. Work is underway to enhance the CFFPR and program activities, such as the CFLN, are transitioning resources and communication to an internal CFF platform as a step toward creating a robust ecosystem. Moreover, organized learning structures of the CFLN are being leveraged to advance methods to test and spread new innovations. In 2023, a third of programs in the CFLN will be involved in applying QI science to achieve reliable processes for using home spirometry and home cultures and addressing social determinants of health. Other programs in the CFLN will test the spread of newly identified processes. As the CFLN becomes more sophisticated with multi‐site design, introducing advanced implementation methods will be possible.

5.3. Conclusion

In conclusion, our evaluation found that the CFLN is highly regarded by its members and supports valuable networking opportunities, use of data to drive improvement, and engagement of PFPs. Several key questions remain about the “ideal” model of a learning network, optimum size, and whether spread to the CCN can improve outcomes and care experience for more people living with CF.

CONFLICT OF INTEREST

Authors Bruce C. Marshall and Kathryn A. Sabadosa are employees of the Cystic Fibrosis Foundation. The authors state they have no conflicts of interest.

ETHICS STATEMENT

The study was approved by the Dartmouth College Committee for the Protection of Human Subjects (Study: 32372, approved November 5, 2021).

Supporting information

Appendix S1. Supporting Information

Van Citters AD, Buus‐Frank ME, King JR, et al. The Cystic Fibrosis Learning Network: A mixed methods evaluation of program goals, attributes, and impact. Learn Health Sys. 2023;7(3):e10356. doi: 10.1002/lrh2.10356