Anticipating adaptation: tracking the impact of planned and unplanned adaptations during the implementation of a complex population-based genomic screening program

Abstract

In 2021, the Medical University of South Carolina (MUSC) launched In Our DNA SC. This large-scale initiative will screen 100,000 individuals in South Carolina for three preventable hereditary conditions that impact approximately two million people in the USA but often go undetected. In anticipation of inevitable changes to the delivery of this complex initiative, we developed an approach to track and assess the impact of evaluate adaptations made during the pilot phase of program implementation. We used a modified version of the Framework for Reporting Adaptations and Modification-Enhanced (FRAME) and Adaptations to code adaptations made during the 3-month pilot phase of In Our DNA SC. Adaptations were documented in real-time using a REDCap database. We used segmented linear regression models to independently test three hypotheses about the impact of adaptations on program reach (rate of enrollment in the program, rate of messages viewed) and implementation (rate of samples collected) 7 days pre- and post-adaptation. Effectiveness was assessed using qualitative observations. Ten adaptations occurred during the pilot phase of program implementation. Most adaptations (60%) were designed to increase the number and type of patient contacted (reach). Adaptations were primarily made based on knowledge and experience (40%) or from quality improvement data (30%). Of the three adaptations designed to increase reach, shortening the recruitment message potential patients received significantly increased the average rate of invitations viewed by 7.3% (p = 0.0106). There was no effect of adaptations on implementation (number of DNA samples collected). Qualitative findings support improvement in effectiveness of the intervention after shortening the consent form and short-term positive impact on uptake of the intervention as measured by team member’s participation. Our approach to tracking adaptations of In Our DNA SC allowed our team to quantify the utility of modifications, make decisions about pursuing the adaptation, and understand consequences of the change. Streamlining tools for tracking and responding to adaptations can help monitor the incremental impact of interventions to support continued learning and problem solving for complex interventions being delivered in health systems based on real-time data.

Our study characterizes the types of adaptations made and shows how changes made to recruitment and outreach strategies impact key reach and implementation outcomes for the genomic screening program.

Implications.

Researchers: Accounting for adaptations in a pragmatic way can support our understanding of the utility of these modifications and inform ongoing, rapid enhancements to best support reach, effectiveness, adoption, implement, and maintenance.

Practitioners: We provide practical tools for practitioners to track adaptations throughout a study.

Policymakers: This approach could be especially useful for programs being implemented in learning health systems, where data-driven approaches are expected.

INTRODUCTION

A growing body of evidence indicates that interventions are regularly modified or adapted during implementation and delivery [1–3]. Adaptations to evidence-based interventions include changes made to the program itself, the implementation strategy, or context of delivery [4–9]. Consideration of these changes throughout the program’s implementation improves interpretation of program findings, identification of which aspects of the intervention were effective in specific settings to ultimately enhance real-time refinements to the intervention, and provide guidance for future delivery [4].

Implementation science can support understanding of the purpose, timing, and nature of adaptations and their impact on key program outcomes. Stirman-Wiltsey developed the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) that identified 7 levels at which modifications can occur (e.g., individual, population, community), 12 types of modifications (e.g., tailoring, shortening/condensing), and 8 adaptation initiators (e.g., individual, team, researchers). Systematic tracking of adaptations has provided evidence about implications of adaptations on intervention outcomes. For example, cultural tailoring of interventions has been shown to improve reach and representativeness of participants compared to non-adapted programs [1–3]. Conversely, spontaneous adaptations that occur during intervention uptake in new settings and adaptations made later during the implementation process may be less effective than planned adaptations occurring at early phases of implementation [5, 8]. Pragmatic methods for documenting and analyzing adaptations from multiple methods (qualitative and quantitative) from numerous sources (stakeholders, program meetings) are needed to support the understanding of the impact of adaptations on key program outcomes.

Large scale, complex interventions delivered in health systems are especially likely to require modifications or adaptations as they are being implemented [10]. Specifically, a growing number of population-based genetic screening programs are being implemented in health systems throughout the USA to facilitate the identification of individuals at high-risk of hereditary conditions [11–13]. In population-based screening programs, healthy adults provide saliva or blood samples to screen for pathogenic variants in genes associated with Hereditary Breast and Ovarian Cancer, Lynch Syndrome, and Familial Hypercholesterolemia [11–13]. The implementation of these screening programs is a complex intervention, which requires interacting components, intervention targets at multiple levels, and influence through numerous pathways [14–16]. Describing adaptations and the associations between adaptations and implementation outcomes can help advance the impact of population-based screening programs and the field of implementation science.

The Medical Univresity of South Carolina (MUSC) launched a large-scale population-based genomic screening program called In Our DNA SC in November 2021, with the pilot phase of the program taking place in 10 clinical settings between November 8, 2021 and March 8, 2022 [17]. We used a modified version of FRAME to track and code adaptations made during the pilot phase of In Our DNA SC and RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) to consider key outcomes [5, 6, 18, 19]. The purpose of this study is to (i) use the FRAME to describe the multiple methods used for adaptation tracking and analysis, and (ii) describe the impact of programmatic adaptations made on implementation outcomes for the In Our DNA SC population-based screening program. We will also discuss methodological challenges and recommendations for tracking, analyzing, and modifying programs based on real-time feedback.

METHODS

Setting and description of project

The goal of In Our DNA SC is to provide genetic screening to 100,000 South Carolinians in addition to obtaining whole exome DNA sequence for future research and clinical applications. The program screens for Centers for Disease Control and Prevention’s Tier 1 conditions of Hereditary Breast and Ovarian Cancer, Lynch Syndrome, and Familial Hercholesterolemia [17, 20]. Eligibility to participate during the pilot phase of this program included: being over the age of 18, ability to speak English, primary residency outside of New York State, and having a clinical visit at one of the 10 participating pilot clinic sites within the next 7 days. Individuals received a message through the MyChart patient portal three days before their visit. Individuals viewed the MyChart message and then could consent to participate in the study. Once consented, a standing order was automatically generated for sample collection during upcoming routine appointments. During the clinical visit, a trained staff provided the saliva sample collection kit and returned it to the Helix laboratory for processing. Results were returned to participants through the patient portal within 8–12 weeks after collection. Those who tested positive for one of the Tier 1 conditions received a phone call to disclose results prior to releasing the results to the patient record and were offered free genetic counseling with MUSC genetic counselors.

Development of coding system and hypothesis generation

Prior to launching the In Our DNA SC program, we developed a system to track adaptations and changes made over time. Specifically, we used a modified version of the FRAME adaptation framework [5, 21]. Types of adaptations were established based on the sub-categories of: whether adaptations were planned or unplanned, who was responsible for the change, the goal of the adaptation, level of delivery, nature of the change, basis by which the change was made, and short-term impact of the change on the program outcomes.

We also used RE-AIM to quantify whether the adaptations impacted outcomes related to reach, effectiveness, and implementation. Reach was assessed through the rate of recruitment messages viewed and rate of those who enrolled. Implementation is defined as how well the program is delivered. This was assessed through the rate of DNA samples collected among those enrolled in the study. At the conclusion of the pilot phase of the program and prior to data analysis, we generated three hypotheses about the impact of adaptations on reach and implementation outcomes based on the literature. Shortening recruitment messages and consent, as well as increasing contact with potential participants (e.g., phone calls) has been shown to improve reach and engagement of participants [22–24]. Techniques to support implementation in clinical settings, including infrastructure/technical changes have been shown to enhance clinical innovations in practice [25]. Effectiveness was assessed qualitative and focused on experience and participant satisfaction with the program.

Overview of adaptations and outcomes tracking process

We documented adaptations in real-time using a REDCap database and completed proactive coding of the adaptation using categories from the FRAME adaptation framework and RE-AIM. We used a multifaceted approach to track adaptations, including weekly leadership team meetings, work group meetings (e.g., marketing, technology and data integration, and research enablement), as well as through email and personal communication. At the conclusion of the pilot phase, we completed reactive coding of adaptations. This step involved two independent coders (first author CGA and a trained research coordinator) independently coding each logged adaptation using the FRAME framework. The reviewers met to review changes and reach consensus through discussion prior to finalizing the adaptation dataset. Quantitative outcomes of reach (number of individuals who open recruitment message and number of individuals who enroll) and implementation (number of samples collected) were captured through weekly data pulls from the electronic health record system.

Data analysis

Frequency and percentages were calculated for demographic characteristics (gender, race, ethnicity, age) of participants who were invited to participate, enrolled, and provided samples for In Our DNA SC during the pilot phase (November 8, 2021 to March 7, 2022). We also captured the frequency and percent of FRAME constructs, including whether the adaptation was planned, who was responsible for the change, the goal of the adaptation, level of delivery, nature of the change, on what basis the change was made, and short-term impact. We tested three hypotheses generated by the study team. To assess Hypothesis 1: Shortened patient portal recruitment messages (occurred on 1/6/22) and shortening the consent form (occurred on 2/2/22) will increase the rate of recruitment messages viewed (Reach as outcome), we constructed a segmented linear regression model in three segments using a general linear model framework. To assess Hypothesis 2: Calling individuals who express interest via patient portal message (occurred on 12/202/22), shortening the Patient portal recruitment message (occurred on 1/6/22), and shortening the consent form (occurred on 2/22/22) will increase the rate of enrollment (Reach as outcome), we constructed a segmented linear regression model in four segments. We assessed Hypothesis 3: Clinical enhancements (occurred on 12/1/21) will increase the rate of samples collected using segmented regression in two segments (Implementation as outcome). Each hypothesis was considered independently, and the 7-day pre-adaptation and 7-day post-adaptation model-based averages of each outcome were compared using Wald t-tests based on the regression estimates. SAS v9.4 (SAS Institute, Cary, NC) was used for statistical modeling. Qualitative data from interviews with 20 program participants were analyzed using content analysis to assess the impact of adaptations on effectiveness of the program. A list of codes was developed by the study team based on the interview guide and two members of the study team (CGA and SG) independently coded the interviews. Disagreement was resolved through discussion between investigators and modifying code definitions. We focus on reporting themes related to effectiveness, with full qualitative analysis available elsewhere [20].

RESULTS

Summary of in our DNA SC pilot phase participants

Recruitment for the pilot phase of the In Our DNA SC took place among 10 sites via MUSC’s electronic health record system (patient portal) between November 8, 2021 and March 7, 2022 (inclusive). Sites included two OBGYN specialty sites and five family medicine/primary care practices, with 80% of the sites located in Charleston county. During this period, 23,269 messages were sent to patients (71.12% female, 68.21% White, 96.48 non-Hispanic, 17.67% between 30 and 39 years old) and 1,641 (7.1%) enrolled (74.65% female, 85.56% White, 96.53% non-Hispanic, 20.84% between 30 and 39 years old), and 736 (44.85%) samples were collected (72.42% female, 87.91% White, 97.69% non-Hispanic, 20.65% between 30 and 39 years old).

Description of adaptations made during in our DNA SC pilot phase

A complete summary of adaptations made over the pilot phase is included in Table 1 and classification of adaptations is included in Table 2. A total of 10 adaptations were made during the pilot phase. Four adaptations were “planned” as part of the pilot phase of the program. These included: changes in work group meeting structure, addition of research staff, shortening recruitment messaging, and shortening the consent form. A variety of individuals were responsible for adaptations, including researchers (30%), entire or most of the team (20%), specific work groups (20%), and other individuals (e.g., study coordinators or required by regulatory team, 20%). The primary goals of the adaptations were to increase the number and type of patient’s contacted (Reach) (60%) to help make it possible to involve more teams, team members, or staff.

Table 1.

Detailed description of adaptations

Adaptation number	Date	Description of adaptation
N/A	November 8, 2021	Program launched in 10 pilot clinical sites with first participant enrolled.
1	November 8, 2021	MyChart messages unable to be sent automatically because of existing limitations in Epic. Research coordinator manually sending MyChart messages.
2	November 19, 2021	Change in work group meeting structure to accommodate shifting needs of program.
3	December 1, 2021	Technical solutions and updates to ensure label printer configuration, packing list set-up for clinics working across multiple departments, and packing list for Helix shipments
4	December 2, 2021	Modified inclusion criteria for identification of participations from 7 days out from clinical visit to 7 days plus “last minute” scheduling (2–3 days).
5	December 10, 2021	Second research coordinator onboarded.
6	December 20, 2021	Began calling individuals who express interest via MyChart message.
7	January 6, 2022	Shortened MyChart recruitment message sent to potential participants.
8	February 9, 2022	Modified follow-up SmartText message in MyChart for those who express interest to clarify process for study enrollment.
9	February 14, 2022	Change to end of consent form to add a check box to the signature statement to attest to use of signature.
10	February 22, 2022	Shortened version of consent form implemented.
N/A	March 7, 2022	Pilot phase ended. Employee and individuals outside of 10 pilot sites eligible to enroll in program.

Table 2.

Classification of adaptations made (n = 10)

Were adaptations planned?	N	%
Planned	4	40
Who was responsible for this change
Entire or most of the team	2	20
Provider	1	10
Administrator	1	10
Researcher	3	30
Developer	1	10
Stakeholder	0	0
Specific Work Group (Cross Functional and Evaluation and Implementation Science)	2	20
Other (Study Coordinator, Required by Regulatory Team)	2	20
What was the goal?
To increase the number or type of patients contacted (Reach)	6	6
To enhance the impact or success of the intervention for all or important subgroups (Effectiveness)	1	1
To make it possible to involve more teams, team members, or staff (Adoption)	3	30
To make the intervention delivered more consistently to better fit our practice, patient flow, or HER for practical reasons (Implementation)	2	20
To save money or other resources (Implementation)	0	0
To institute or sustain the intervention (Maintenance)	2	20
To respond to external pressure or policy	2	20
Other (To respond to needs of work group; To respond to IRB requirements)	2	20
At what LEVEL of delivery (for whom/what is the modification made?)
Individual patient level	7	70
Group	1	1
Individual practitioner level	0	0
Clinic/unit level	2	20
Hospital level	0	0
Network level	0	0
Systems level	1	10
Specific work group	0	0
Other (Implementation Teams)	3	30
What is the nature of change
Tailoring to individuals	1	10
Adding a component	3	30
Removing a component	1	10
Condensing a component	2	20
Extending a component	0	0
Substituting for a component	0	0
Changing the order of components	0	0
Integrating with other programs	0	0
Repeating a component	0	0
Loosening the structure or protocol	1	10
Otherwise changing the intervention	0	0
How or on what basis were these changes made?
Based on our vision or values	2	20
Based on a framework	0	0
Based on our knowledge or experience	4	40
Based on QI data, summary information or results	3	30
Based on pragmatic/practical considerations	2	20
Based on financial incentives	0	0
Based on feedback or suggestions	4	40
Others (To remain compliant (2); planned addition of staff)	3	30
Short-term impact
No major changes	2	20
Positive	5	50
Negative	0	0
Unknown	3	30

Changes were primarily made for the individual/patient (70%), with some adaptations (30%) designed to support the success of implementation teams (30%). The nature of the changes were diverse, including adding a component (30%) (e.g., updating packing list and orders, calling individuals who express interest, and modification to the consent form), condensing a component (20%) (e.g., shortening patient portal recruitment message, shortening the consent form), tailoring to individuals (10%) (e.g., modifying follow-up SmartText message in Patient portal for those who express interest), removing a component (10%) (e.g., removing/editing the consent form to shorten), and loosening the structure of the protocol (10%) (e.g., modifying inclusion and exclusion criteria). Changes were primarily made based on knowledge and experience (40%) or from quality improvement data, summary information, or results (30%).

Impact of adaptations on RE-AIM outcomes

Reach

Three adaptations were designed to increase reach of the program to participants, including shortening the patient portal recruitment message, shortening the consent form, and calling individuals who express interest via patient portal message (Table 3). We hypothesized that shortening patient portal recruitment message (occurred on January 6, 2022) and shortening the consent form (occurred on February 22, 2022) would increase the rate of messages viewed. We found that shortening the message significantly increased the absolute average rate of messages viewed 7 days pre-adaptation (32.7%) compared to 7 days post-adaptation (42.0%) (p = 0.0106); however, there was no difference in rate of messages viewed based on the change to the consent form (p = 0.34). Our second hypothesis related to reach was that calling individuals who express interest via patient portal (occurred on December 20, 2021), shortening the patient portal recruitment message (occurred on January 6, 2022), and shortening the consent form (occurred on February 22, 2022) would increase the enrollment rate. We found no significant differences in enrollment rate based on these adaptations.

Table 3.

Changes to reach and implementation

	Estimate	Standard error	p Value
Hypothesis 1: Shortened patient portal recruitment messages and shortening the consent form will increase the rate of recruitment messages viewed (Reach)
Change pre-and post-shortened MyChart message	+0.09	0.03	0.01
Change pre- and post-shortened consent	+0.03	0.03	0.34
Hypothesis 2: Calling individuals who express interest via patient portal message, shortening the MyChart recruitment message, and shortening the consent form will increase the rate of enrollment (Reach)
Change pre- and post-calling individuals	+0.01	0.02	0.54
Change pre-and post-shortened MyChart message	−0.02	0.03	0.58
Change pre- and post-shortened consent	−0.03	0.02	0.14
Hypothesis 3: Clinical enhancements will increase the rate of DNA samples collected (implementation)
Change pre- and post-clinical enhancements	+0.06	0.16	0.69
*Changes are based on differences between the model-based estimates 7 days prior to and 7 days after the adaptation initiation.

Effectiveness

Only one adaptation was designed specifically to enhance effectiveness of the intervention, which we defined as how well the genomic screening program is disseminated to sociodemographic subgroups. We modified the consent form based on qualitative feedback from participants and the In Our DNA SC community advisory board (occurred on February 22, 2022). This included shortening the consent, simplifying, and condensing language within the consent form. Prior to making modifications, interviewees suggested concern about length of the form and participant understanding, “It takes 20 minutes to go through. I just think you’re gonna lose a lot of people because people don’t have the attention span. It’s intimidating and just you’re going to lose people. But if you say, here’s the gist of it and throughout it have hyperlinks to click on it, then it’s more digestible. Or not as intimidating.” Qualitative finding suggests that this modification enhanced the impact or success of the intervention for participants. For example, “I think it was several pages long the whole thing, but I think it was fairly straight forward.” This was considered a planned modification, which was decided upon by the full team based on the vision and values of the program, data about uptake, and feedback or suggestions.

Implementation

We also assessed the impact of enhancements of clinical sites capabilities that occurred on December 1, 2021 on implementation outcome of sample collection rate in the pilot locations. We found no significant difference in the rate of samples collected pre-and post-adaptation (p = 0.6920) (Table 3).

DISCUSSION

The FRAME provided a structure to track adaptations through multiple methods over the course of the pilot phase of In Our DNA SC. We identified 10 unique adaptations and assessed their role on reach, effectiveness, and implementation outcomes for the program. This approach offered reflective understanding of adaptations and enhances understanding both of how to improve population-based screening as well as consideration for tracking, analyzing, and modifying programs based on real-time feedback.

Most adaptations were considered reactive or unplanned. Generally, guidance for implementation recommends avoiding reactive adaptations; however, given the nature of the program and concept of population-based genomic screening, we found that changes were difficult to anticipate. Further, changes were driven by an identified need, with data being rapidly collected, analyzed, and used to optimize the program [7–9]. Despite changes, we only found that one adaptation significantly increased Reach (as measured by rate of recruitment messages viewed) but did not ultimately increase enrollment. Although patient portals are often used for outreach and recruitment, the nature of this study (collecting DNA samples), may require more robust stakeholder engagement and high-touch outreach efforts, especially among rural and minority populations [26–29]. Further, our qualitative observations of the impact of adaptations bolstered understanding of the need for various changes. Although modifications to the consent form did not enhance the primary outcome of interest (reach), being responsive to qualitative evaluation feedback about the consent form enhanced overall user experience and satisfaction (effectiveness).

Prior literature has indicated a high burden when tracking adaptations, reducing the effectiveness of tracking and usefulness of data to inform decision-making [5]. To reduce burden and support utility of tracking efforts, we developed a REDCap database designed using FRAME to capture adaptations in real-time. This approach allowed us to quickly track adaptations from various sources (e.g., meetings, emails, formal interviews) and reduced the burden of data collection on the research team and implementers. Use of various sources and mixed methods is supported by the implementation science literature [30]. We initially coded all aspects of the adaptation at the time of recording and then retrospectively reviewed the codes and updated with two coders. While adaptations were tracked in real-time, it was difficult to systematically use the information to inform further modifications to implementation.

Using an iterative approach could help plan adaptations, inform program modifications, and monitor incremental adaptations [18]. For example, a recent pharmacogenomic initiative implemented at the Department of Veterans Affairs included a Plan-Do-Study-Act cycle to evaluate the implementation resources and processes, which were then mapped to the Consolidated Framework for Implementation Research to identify implementation barriers and facilitators, which were addressed using the Expert Recommendation Implementing Change Strategies [31]. Other opportunities to rapidly assess programs is through an iterative RE-AIM process or action planning process could allow for mapping implementation strategies (i.e., methods or techniques used to enhance the adoption, implementation, or sustainment of a program) [32]. This approach could be especially useful for programs being implemented in learning health systems, where data-driven approaches are expected. Alternatively, the Dynamic Sustainability framework could offer a structure to move from adaptations tracking and toward adaptations-informed action planning and ongoing quality improvement to continue learning, adapting interventions to ensure fit in varying contexts and populations [32].

This study is not without limitations. Our data included adaptations made only during the program roll-out and we did not capture pre-implementation or sustainment adaptations. Relatedly, we captured effectiveness, qualitatively based on the research team’s coding of impact. Identifying the quantitative measures for these outcomes could help bolster understanding of the impact of adaptations on each outcome. Our outcome of implementation focused solely on rate of sample collection and did not consider implementation process indicators such as fidelity or dose delivered. We excluded maintenance from this analysis, given the short timeframe. Further, we considered each adaptation independently and assessed impact based on changes in outcomes 7-days pre- and 7-days post-adaptation. Future efforts could consider the combined effect of adaptations, as well as longer term impact of each adaptation.

CONCLUSIONS

This study provides a practical examination of adaptations that occurred during the implementation of a large-scale, complex intervention being implemented in a learning health system. We focused specifically on the population-wide genomic screening program, In Our DNA SC; however, methodological aspects of the adaptation tracking used for this program can be expanded to implementation or program in both clinical and non-clinical settings (e.g., community, public health). Accounting for adaptations in a pragmatic way can support our understanding of the utility of these modifications and inform ongoing rapid enhancements to best support initiative’s reach, effectiveness, adoption, implement, and maintenance.

COMPLIANCE WITH ETHICAL STANDARDS

Conflict of interest: The authors declare they have no competing interests.

Informed consent: We received approval from MUSC IRB.

Consent for publication: All authors have consented for publication.

Transparency: (i) This study was not formally registered, (ii) the analysis plan was nor formally pre-registered, (iii) de-identified data from this study are not available in a public archive. De-identified data from this study will be made available (as allowable according to the institutional IRB standard) by emailing the corresponding author, (iv) analytic code used to conduct the analyses presented in this study are not available in a public archive. They are available by emailing to corresponding author; (v) materials used to conduct the study are not publicly available. They are available by emailng the correspondning author.

AUTHORS CONTRIBUTIONS

Conception and design (CGA, KJH, KS, and PR), Acquisition of data (CGA, DPJ, AJ, and SG), Analysis and Interpretation (CGA, PN, KJH, and SG), Drafting the article (CGA, PJN, and KJH), Revising for Intellectual Content (DPJ, PJN, KJH, AJ, SG, KS, PSR, CM, KW, KC, MF, LM, and LL). All authors have read and approved the final manuscript.