Developing a Common Framework for Evaluating the Implementation of Genomic Medicine Interventions in Clinical Care: The IGNITE Network’s Common Measures Working Group

Abstract

Purpose

Implementation research provides a structure for evaluating the clinical integration of genomic medicine interventions. This paper describes the Implementing GeNomics In PracTicE (IGNITE) Network’s efforts to promote: 1) a broader understanding of genomic medicine implementation research; and 2) the sharing of knowledge generated in the network.

Methods

To facilitate this goal the IGNITE Network Common Measures Working Group (CMG) members adopted the Consolidated Framework for Implementation Research (CFIR) to guide their approach to: identifying constructs and measures relevant to evaluating genomic medicine as a whole, standardizing data collection across projects, and combining data in a centralized resource for cross network analyses.

Results

CMG identified ten high-priority CFIR constructs as important for genomic medicine. Of those, eight didn’t have standardized measurement instruments. Therefore, we developed four survey tools to address this gap. In addition, we identified seven high-priority constructs related to patients, families, and communities that did not map to CFIR constructs. Both sets of constructs were combined to create a draft genomic medicine implementation model.

Conclusion

We developed processes to identify constructs deemed valuable for genomic medicine implementation and codified them in a model. These resources are freely available to facilitate knowledge generation and sharing across the field.

INTRODUCTION

In 2013, the National Human Genome Research Institute established the IGNITE (Implementing GeNomics In PracTicE) Network to support development, implementation, and dissemination of methods that incorporate genomic medicine information into clinical care.¹ The Network includes six projects, affiliate members, coordinating center, and Working Groups to facilitate cross-network collaboration.¹ Through the work of network members, IGNITE is poised to provide a substantial impact on genomic medicine in the ‘real-world.’ “Its stated goals are to expand and link existing genomic medicine implementation efforts; develop new collaborative projects and methods for genomic medicine implementation in diverse settings and populations; contribute to the evidence base of outcomes following the use of genomic information for clinical care; and define, share and disseminate best-practices of genomic medicine implementation, diffusion and sustainability in diverse settings.”

While all six IGNITE projects are aligned through their work in genomic medicine, each differs in their intervention (i.e., pharmacogenomics, disease risk assessment, or disease diagnosis), and patient and provider populations. For specifics on each project’s study, refer to the IGNITE website (ignite-genomics.org). IGNITE’s diversity is, in large part, a direct reflection of the current state of affairs in genomic medicine: an explosion of research results that have not yet sufficiently infiltrated clinical practice to reach patients or providers, particularly those with the fewest resources and greatest challenges to achieving better health. IGNITE members are committed to understanding and addressing barriers to dissemination of actionable genomics findings, and to understanding population impact, particularly among underrepresented populations. IGNITE is thus comprised of a substantial network of diverse clinicians, practice settings, patients, and investigators in geographically widespread areas with variable levels of exposure to genomics. As such, IGNITE affords a rich opportunity to explore, test, institutionalize, and disseminate programs to translate genomic medicine into routine practice. Together, IGNITE investigators, representing a wide array of genomic, clinical, stakeholder engagement, and technological expertise, are discovering best-practices for conducting research, building systems to make genomic information more accessible and actionable, and rigorously measuring their barriers, facilitators, and impact.²

The wealth of evidence being generated by each IGNITE project is beginning to lay the groundwork for understanding relevant issues around barriers to implementation, effectiveness, and stakeholder value. However, given the diversity of projects, common themes could be difficult to discern. To help facilitate identification of foundational evidence that could broadly guide implementation of genomic medicine, the network developed the Common Measures Working Group (CMG) (https://ignite-genomics.org/network/working-groups-interest-groups/) at its inaugural meeting in June 2013. The mission of the working group is “to gather data, evaluate, and disseminate methods of genomic medicine implementation research across diverse projects conducted by IGNITE members.” To achieve this mission, the working group developed a plan to identify constructs and measures relevant to evaluating genomic medicine interventions as a whole, standardize data collection across projects, combine data in a centralized resource for cross network analyses, and develop a testable genomic medicine implementation research model based on the findings of IGNITE research. The database, containing data derived from the body of implementation research and standardized across contexts, is the type of queriable resource that could guide implementations in a learning healthcare system as described by Chambers, et al.³ In this way, the group felt it could balance the diversity, enhance external validity, and increase statistical power over what could be generated by each project alone. The objective of this paper is to describe the process the CMG is employing to meet its mission and the products of the work to date.

METHODS AND RESULTS

The CMG, comprised of 23 members (representing every project plus interested IGNITE affiliates) met twice monthly via Web-Ex. To achieve its goals, the CMG established a process for identifying common measures, defined steps to evaluate those measures, and reviewed existing literature for similar projects² (Figure 1).

Figure 1.

Common Measures Working Group Process Plan for Developing Common Measures

After reviewing the implementation research literature, the CMG selected the Consolidated Framework for Implementation Research (CFIR)⁴ (http://cfirguide.org/) to guide our approach, since the framework allowed us to: 1) think broadly about common measures, ensuring that it addressed the full spectrum of implementation characteristics; and 2) share knowledge generated by IGNITE with other genomic medicine projects and networks. CFIR is a framework developed to clearly described terminology and definitions for a comprehensive collection of constructs, drawn from published models (http://cfirguide.org/).⁵ As such, it pulls together all existing implementation models into a comprehensive ontology of overarching themes (domains and constructs) that are postulated to be important in implementation. Unlike implementation models, it does not assume there is any specific relationship between individual constructs or constructs and outcomes. It simply lists them all in an organized (consolidated) format- thus its name “Consolidated Framework”. It comprises 39 constructs organized across five domains (Figure 2): 1) Innovation Characteristics; 2) Outer Setting; 3) Inner Setting; 4) Characteristics of Individuals; and 5) Process, which are hypothesized to interact in dynamic and complex ways. Thus the CFIR is an ideal mechanism for establishing a common structure to facilitate knowledge generation and sharing across the diverse settings, populations, and innovations that make up the IGNITE network projects.^6,7

Figure 2. The Consolidated Framework for Implementation Research.

Bolded constructs are those identified as high priority by IGNITE.

The methods and results below are organized according to the CMG’s goals outlined in the introduction: identify constructs and measures relevant to genomic medicine as a whole, standardize data collection across projects, combine data in a centralized resource for cross network analyses, and develop a testable genomic medicine implementation research model based on the findings of IGNITE research.

Identify Constructs and Measures Relevant To Genomic Medicine

The CMG followed three steps to identify and develop common measures that could facilitate genomic implementation research: (1) review CFIR constructs and prioritize them in order of perceived value for genomic medicine; (2) compile a list of all measures collected by each project and map them to CFIR constructs (if able); and (3) identify constructs relevant to genomic medicine that are not represented in CFIR. In this way, we would be able to assess two different but equally important aspects of the existing projects’ protocols: where projects were assessing similar constructs, and where constructs that were deemed valuable to the network were not being measured.

Step 1 (Methods)

each project team independently rated (in order of perceived value for genomic medicine) the 39 CFIR constructs across all five domains using a 3-point scale: 3=very important, 2=somewhat important, 1=not important. Next, ratings for each of the 39 constructs were summed and ranked in order of value (highest summed score to lowest). The ten highest ranked constructs, plus any constructs not in the top ten that had been rated as “very important” by at least one project, were identified as high-priority. These constructs were presented to the full working group and discussed to ensure consensus on the list of high priority CFIR constructs.

Step 1 (Results)

Initial rankings identified 16 CFIR constructs or sub-constructs supported by all projects trialability, adaptability, relative advantage, and cost (domain: INTERVENTION CHARACTERISTICS); patient needs and resources (domain: OUTER SETTING); organization’s structural characteristics, networks and communications, culture, implementation climate (relative priority), and readiness for implementation (leadership engagement, available resources, access to knowledge and information) (domain: INNER SETTING); knowledge and beliefs about the intervention, and self-efficacy (domain: INDIVIDUALS’ CHARACTERISTICS); and planning, engaging, executing, and reflecting & evaluating (domain: PROCESS) as potential high-priority constructs for the network. Just as importantly, no projects supported intervention source, design quality & packaging, peer pressure, organizational incentives/rewards, other personal attributes, or external change agents as high- priority constructs. Table 1 shows the final results of the ranking procedure, identifying the ten highest ranking constructs, which were then used to frame data collection for the network.

Table 1.

Shows those constructs identified as high priority and associated measures

CFIR Domain	High Priority CFIR Construct	Identified Measure
Outer setting	No high priority constructs	Not assessed
Inner Setting	Implementation climate Readiness for implementation	Organizational Readiness for Implementation Change survey12 none
Individuals’ Characteristics	Knowledge and beliefs about the intervention	Qualitative interviews
	Self-efficacy	none
Intervention Characteristics	Relative Advantage	none
	Cost	Electronic Medical Record Data on utilization
Process	Engaging	none
	Executing	none
	Reflecting and evaluating	Qualitative interviews
Non-CFIR Domain	Non-CFIR Construct	Measure
Patients’ Characteristics	Demographics	NIH standard definition for demographics
	Self-reported Health	Single item from SF-12
	Healthcare Activation	PAM11
	Medication Adherence	Voils Medication Adherence survey9
	Social Determinants	Health Literacy Question10
	Family and Community	Various depending upon aspect measuring
	Information Sharing	Intention to share

Step 2 (Methods)

Each project listed all of the domains (and corresponding measures) that they planned to assess or were already assessing (depending upon how far along they were in initiating their study) within the patient, provider, and health system levels. For example, domains in the patient level included: demographics, quality of life, laboratory data, decisional processes, social influences, health literacy, attitudes, beliefs, knowledge, readiness to change, behavior, satisfaction, intention, genetic results, family history, and health outcomes. This classification aided in identifying projects that were measuring the same domains but with different instruments, facilitating discussions amongst projects as to whether a) projects not currently measuring a domain where others were, would want to include that domain in their assessments, and b) whether projects assessing the same domain with different measures would be willing to use a common instrument. Next, all measures were mapped to their corresponding CFIR construct. Mapping was initiated first by each project independently, and then by one of the CFIR developers (LJD). During this process, if a measure was consistently mapped to the same construct, it was automatically assigned to the construct; if there was disagreement, consensus was reached through discussions facilitated by Dr. Damschroder. Since, as is often the case, multiple measures were mapped to the same construct, measures within constructs were prioritized using the same process as that for the ranking constructs (step 1).

Step 2 (Results)

Table 1 lists the highest priority CFIR domains associated with each of the high priority constructs. Note that most of the measures initially planned for inclusion in study protocols did not assess high priority CFIR constructs and most of the high priority constructs did not have published measurement instruments.

Step 3

In many cases, measures in step 2 could not be mapped to a CFIR construct. Since existing evidence suggests that identifying and describing key contextual factors, even those that fall outside of established CFIR construct, is valuable because they can affect patient outcomes through their influence on implementation, we included these domains and associated measures as high priority constructs when they were considered critical for understanding genomic medicine. This determination was made in the same way that CFIR constructs were determined to be high priority (step 1). To do this, when measures could not be mapped to CFIR constructs, they were “mapped” to their underlying domain and the domain added as a construct to the list of high priority genomic medicine constructs (from step 1).

Step 3 (Results)

After comparing CFIR constructs to the projects’ compiled list of domains and measures, it became clear that CFIR lacks a well-defined representation of patient-related domains. The CFIR domain “Individual’s Characteristics” reflects individuals within the organization and while patients are part of the health care organization and are essential to assessing intervention effectiveness, they are a less influential component of implementation success in healthcare settings than administrators and physicians. The working group therefore identified patient domains highly relevant to genomic medicine implementation and included these in Table 1 as non-CFIR high priority constructs.

The ultimate result was a matrix that consisted of CFIR and non-CFIR constructs prioritized for their value to the network with a list of associated measures (also ranked). The matrix provided a simple method for identifying which high-priority constructs the network was already measuring and which were not. When measures were listed under a high-priority construct, the highest ranking measure was adopted as the preferred measure. For high-priority constructs with no corresponding measures, a literature search was undertaken to identify if existing instruments could be adapted to genomic medicine, and, if none was identified, then measures were developed. These processes are described in the next section.

Standardize Data Collection Across Projects

To maximize the potential value of cross-network analyses, the working group needed to standardize data collection across projects wherever possible. This process occurred as an extension of step 2 in the “Identify constructs and measures relevant to genomic medicine” process (above). Two synergistic approaches were employed to facilitate standardized data collection: The first was to promote adoption of a single measure for each domain being assessed by more than one project, and the second was to develop measures for high-priority CFIR and non-CFIR constructs if no published standardized measure existed.

Promote adoption of a single measure for each domain (Methods)

As described earlier, all projects listed the domains they were measuring across the spectrum of stakeholders: patients, providers, and health systems. During this process it became clear that, in several cases, two or more projects were planning on assessing a domain (such as patient quality of life) but were using different measures to do so. To promote adoption of a single measure for as many measures as possible, regardless of whether they fell into a high priority construct/domain, the working group encouraged discussion amongst the projects by specifically calling attention to those domains and enabling project leadership to discuss them on a case by case basis.

Promote adoption of a single measure for each domain (Results)

During this process, common measures for several different domains were adopted by projects measuring them. Specifically, the network adopted the following measures: a single item from the SF-12⁸ for self-related health, Voils’ survey for patient medication adherence⁹, a single item for patient health literacy¹⁰, the Patient Activation Measure¹¹, and the Organizational Readiness for Implementation Change¹². In addition, two projects initially planning on only assessing patient measures through the electronic medical record modified their protocol to incorporate patient/provider surveys using common measures identified here and below to enhance the feasibility of cross network analyses.

Develop measures for high priority CFIR and non-CFIR constructs (Methods)

As described in the “Identify Constructs and Measures Relevant To Genomic Medicine” process, Table 1 represents high priority CFIR and non-CFIR constructs important for genomic medicine. Importantly this process identified constructs as valuable that no project had previously considered measuring. To facilitate measurement, the CMG conducted literature reviews. When measures were identified, they were described and coded for standardized data collection across projects; when measures could not be identified, the working group developed measures with the help of a psychometrician.

Develop measures for high priority CFIR and non-CFIR constructs (Results)

Four surveys were standardized and coded for use across the projects planning to survey patients and/or providers; all are available for download from the IGNITE SPARK toolbox (https://ignite-genomics.org/spark-toolbox/). The first survey agreed on and coded patient demographic and education measurement using the NIH-preferred measures. The second is a post-intervention patient survey that incorporates measures for patient attitudes towards a specific genomic intervention, intentions for sharing intervention results, and preferences for return of results. The third codifies provider’s demographics and practice/setting characteristics, including age, gender, race, ethnicity, practice setting, profession, specialty, and years in practice. The fourth is a provider survey designed as a pre-implementation survey that could also be employed post-intervention as well. This survey, initially 14 items which were then refined to 10, was developed by the CMG to address high-priority CFIR constructs (Table 1) including implementation climate, readiness for implementation, knowledge about the intervention, self-efficacy, and relative advantage.

Lastly, to promote the cross-network analysis, we categorized study interventions (i.e., as pharmacogenomic, disease risk assessment, or diagnostic) and identified several cross-study outcomes that could be assessed despite the disparate interventions. These included uptake of the genomic intervention by providers, frequency provider’s act on intervention recommendations, success of intervention on each study’s primary patient endpoint, pre-post change in patient’s diet and/or exercise habits, frequency that intervention results are actionable, and impact on mortality, disease incidence, hospitalizations, or ER visits.

Combine Data In A Centralized Resource For Cross Network Analyses

Methods

Common data were stored in a relational database (details provided in online supplement). To demonstrate the potential value of such a database, we performed a preliminary analysis using R software on patient-level data deposited in the resource database prior to August 12, 2016. Patient demographic features and responses to the patient survey were summarized within and across the IGNITE projects using summary statistics. Differences across projects were evaluated using chi-squared tests for project/intervention, sex, race, ethnicity, and education, and ANOVA F-test for age.

Results

Of the six projects, one was not collecting individual patient data and therefore was not able contribute to the patient data analysis. Of the remaining five projects, one was collecting data on paper and was not able to provide an electronic version by the time of manuscript completion, and another opted not to implement the demographic or patient survey. Therefore, the patient analysis represents data contributed from three projects.

Across these three projects, there were 2430 patients (Table 2). All demographic features were reported by two projects, while one project reported all demographic features except for education level. Overall, the demographic features had high response rates: age and sex were reported for nearly all patients (99.5% for both age and sex), while ethnicity, race, and education response rates were slightly lower (77.0%, 90.0%, and 89.9% respectively). We note that nearly all demographic features, except sex, differ by project (age/race/education p-values<1.0×10⁻⁵, ethnicity p-value=0.002). Patient responses to the pre-implementation common measures questions were less complete than the demographics responses. Four of the seven common measures questions had responses submitted by at least one project, and no single question had responses from all three projects. Three questions had responses from two different projects, and one question had response from only one project. The variability in number of projects reporting any given question resulted in patient response rates ranging from 9.87% to 73.4% of the 2340 patients with demographic information.

Table 2.

Patient demographics and Patient baseline survey analysis from common measures database

	Projects Contributing Data (N)	Patients Completing survey% (N)	Response Summary Level – percent/Mean (sd)	Demographic features by which responses vary Feature (p-value)*
Demographic Feature
Age	3	99.5% (2419)	56.84 (14.0)	Project (< 0.00001)
Sex	3	99.6% (2420)	Male – 32.3% Female – 67.8%	none
Ethnicity	3	77.0% (1871)	Non-Hispanic - 97.1% Hispanic – 2.8% Prefer not to Answer- 0.1%	Project (0.002)
Race	3	90.7% (2203)	American Indian/Native Alaskan – 0.4% Asian – 1.0% Black/African American – 14.4% Native Hawaiian/Pacific Islander – 0.2% White – 81.8% More than one Race – 2.1% Prefer not to Answer – 0.1%	Project (< 0.00001)
Education Level	2	89.8% (2183)	High school (12 years) or less – 13.7% Some college – 19.0% College graduate – 27.5% Postgraduate – 39.8%	Project (< 0.00001)
Patient pre-implementation survey items
“Is it a good idea to ___[e.g. get genetic testing] to find out whether ___[e.g. at risk for getting a disease]”	2	24.1% (586)	Strongly disagree - 0.34% Disagree – 0.34% Neither agree nor disagree – 7.34% Agree – 52.73% Strongly Agree – 39.25%	Project (< 0.00001); age (0.000017)
“Do you plan to share [test] results with any one?”	2	24.6% (597)	No – 19.3% Yes – Family – 69.7% Yes – Friends - 19.3% Yes – Health Professional – 33.3% Yes – Other – 1.0% Unsure – 6.0%	Project (< 0.00001 – 0.00145); age (0.00010 – 0.045); sex (0.0028); race (0.00051 −0.025)**
“How confident are you filling out medical forms by yourself?”	2	73.4% (1783)	Extremely – 78.74% Quite a bit – 13.29% Somewhat – 5.33% A little bit– 1.74% Not at all – 0.90%	Project (< 0.00001); age (0.002); ethnicity (0.03); race (< 0.00001); education level (< 0.00001)
“How confident are you that you could get health-related advice or information if you needed it?”	1	240 (9.9%)	Completely confident – 57.1% Very confident – 23.8% Somewhat confident – 16.2% A little confident – 1.7% Not confident at all – 1.2%	none

Develop a testable genomic medicine implementation research model based on the findings of IGNITE research

Methods

By following the steps outlined above, the working group gathered an abundance of data that, once organized, were able to inform a structural understanding of genomic medicine research implementation. We therefore leveraged these data to develop an IGNITE Genomic Medicine Implementation research conceptual model derived from published high-level models and informed by findings across the six IGNITE projects. The goal of the model is to serve as a resource for researchers, such that new evidence can inform and refine the model over time. The initial model was developed based upon high-priority constructs, both within and outside of CFIR. Strategies used by the team to implement genomic medicine interventions were conceptualized as mechanisms of action aimed at effective implementation but that would be moderated (positively or negatively) by high-priority constructs at the organization, physician, and patient levels. More effective implementations of genomic medicine interventions were hypothesized to lead to better clinical or process outcomes.

Results

Based on high-priority constructs identified by the project teams, moderators that are hypothesized to influence implementation effectiveness were defined across four domains: 1) intervention characteristics; 2) characteristics of the organization’s inner setting; 3) characteristics of clinicians involved with delivering the genomic medicine intervention; and 4) characteristics of the process of implementation. Therefore, the model flow suggests measuring those domains prior to implementation so they can inform the selection of the optimal implementation strategy. Figure 3 displays how the domains relate to each other and the implementation strategy; but specific constructs are not listed in the figure as they may differ in other implementation projects (e.g. assessing experience with return of results rather than intention to share results). For IGNITE they are reported in Table 1.

Figure 3.

Genomic Medicine Implementation Model Developed as part of the IGNITE Network

DISCUSSION

The IGNITE Network’s Common Measures Working Group brought together six disparate projects to synthesize a model for understanding the process of genomic medicine implementation into clinical practice and to identify/develop standardized measures that may help to inform a broader understanding of genomic medicine implementation research. The key steps taken by the group to advance understanding beyond each individual project are formalized in this paper: 1) to outline a pathway for other networks who are interested in using a similar methodology; and 2) to provide tools for individual institutions/projects interested in genomic medicine implementation research. The model (Figure 3), developed with input from providers with diverse clinical settings, environments, and populations, is a simplified linear depiction of our complex nonlinear efforts that can is “testable” in that it is meant to be refined as results from ongoing work inform our understanding of this fast moving field. For example, if other implementation projects identify different or additional high priority constructs those should be added to list of options for that domain or if researchers find that in order to be successful all genomic implementation processes should include specific characteristics that are un-related to the constructs then the model should be updated to represent those findings.

One key finding of our work is that while the CFIR provides an excellent foundation upon which to conceptualize implementation and as a framework provides representation of constructs from across the entire field of implementation science, it does not fully capture all domains pertinent to genomic medicine. To address this, we expanded beyond the CFIR to incorporate domains relevant to patients, families, and local communities as CFIR (and thus the implementation models upon which it derived its list of constructs) is more specifically focused on factors relevant to the health system rather than local community values. In genomic medicine, the patient and the local culture around the patient are extremely important, with a large body of work devoted to understanding patient perceptions, anxiety, and personal utility. While these are not unique to genomic medicine, they have special emphasis when interventions address topics related to genomics- particularly for patients who are generally healthy and not pursuing a diagnostic odyssey.

Another key finding, that most projects were not initially measuring any of the CFIR constructs they had identified as high-priority, is an essential insight into genomic medicine’s current perceptions of implementation and forays into the field of implementation science. This work may help to guide other implementation projects to consider a broader conceptual view of implementation and which constructs could be valuable for them to address when selecting measures and outcomes. It is important to address these issues early prior to projects developing protocols to improve the ability to synergize data collection and methods. Five projects changed their protocols to incorporate relevant measures: two to incorporate patient surveys and three to adopt high priority measures (the “common measure”) for constructs that they were initially planning to measure, but with different measurement instruments. This shows the value of the measures developed; however, we were still limited in the data we were able to capture given that most projects had already begun when the CMG completed its evaluations.

By thoroughly addressing these questions early in the network formation, we were able to enhance the scientific rigor of the network as a whole and facilitate cross study analyses that can address a wide variety of questions around implementation and effectiveness such as: (1) provider demographics, attributes and attitudes that predict uptake of genomic medicine; (2) the extent to which patients plan to and actually share their genetic information with relatives, friends and others; (3) patient attitudes toward genetic testing and its potential benefits and risks as perceived by those undergoing a variety of interventions; (4) influence of the genomic intervention and type of genomic intervention on health, well-being and self-care/preventive behaviors by patients and provider practices. It is important to note that, while the diversity of the projects is desirable and leads to greater generalizability of cross-network findings, it is also a challenge for combining data. Therefore, a number of questions cannot be answered by analyzing data from across the network; many can only be analyzed within each project. Yet, whenever possible, combining data across the network will provide unique insight into the field as a whole. We have defined cross-network measures as those that are being collected from three or more projects. To demonstrate the potential for these analyses, we reported a preliminary analysis of patient demographics and responses to the CMG developed patient survey (Table 2). These results are not meant to be taken as definitive “findings,” but, rather, as an example of what these data might look like.

The results of the Common Measures Working group’s efforts, which include the implementation model, surveys, and listing of high priority constructs, are available in the IGNITE Network’s “Supporting Practice through Application, Resources and Knowledge” (SPARK) toolbox (https://ignite-genomics.org/spark-toolbox/). SPARK provides free access to the products of our work, which may be useful to clinicians, patients, and educators, as well as researchers. It contains resources to educate and inform providers and patients as well as recommended measures to assess implementation effectiveness. The implementation research model is unique in that it is specifically tailored to genomics, a rapidly growing but still relatively underutilized tool within healthcare.