Protocol to Evaluate an Enterprise-Wide Initiative to Increase Access to Lung Cancer Screening in the Veterans Health Administration

Abstract

Introduction

The Veterans Affairs Partnership to increase Access to Lung Screening (VA-PALS) is an enterprise-wide initiative to implement lung cancer screening programs at VA medical centers (VAMCs). VA-PALS will be using implementation strategies that include program navigators to coordinate screening activities, trainings for navigators and radiologists, an open-source software management system, tools to standardize low-dose computed tomography (LDCT) image quality, and access to a support network. VAMCs can utilize strategies according to their local needs. In this protocol, we describe the planned program evaluation for the initial 10 VAMCs participating in VA-PALS.

Materials and Methods

The implementation of programs will be evaluated using the Consolidated Framework for Implementation Research to ensure broad contextual guidance. Program evaluation measures have been developed using the Reach, Effectiveness, Adoption, Implementation and Maintenance framework. Adaptations of screening processes will be assessed using the Framework for Reporting Adaptations and Modifications to Evidence Based Interventions. Measures collected will reflect the inner settings, estimate and describe the population reached, adoption by providers, implementation of the programs, report clinical outcomes and maintenance of programs. Analyses will include descriptive statistics and regression to evaluate predictors and assess implementation over time.

Discussion

This theory-based protocol will evaluate the implementation of lung cancer screening programs across the Veterans Health Administration (VHA) using scientific frameworks. The findings will inform plans to expand the VA-PALS initiative beyond the original sites and can guide implementation of lung cancer screening programs more broadly.

1. Introduction

In 2020, lung cancer is expected to cause 135,720 deaths in the United States, which is more than colorectal, breast and prostate cancer deaths combined [1]. Most patients (74%) are diagnosed with lung cancer at advanced stages when there is less chance for curative treatment, which contributes to its high mortality[1]. Stimulated by the results of the Early Lung Cancer Action Project (ELCAP) [2–4], two large U.S. (the National Lung Screening Trial [NLST]) and European (the Dutch-Belgian Lung Cancer Screening Trial [NELSON]) multi-center, randomized-controlled trials demonstrated that early detection with low-dose computed tomography (LDCT) decreased lung cancer-specific mortality in high-risk individuals [5, 6]. Following the results of these trials, many professional organizations developed guideline recommendations [7–11] and the United States Preventive Services Task Force (USPSTF) issued a grade B recommendation for annual lung cancer screening with LDCT in the high-risk population [12]. The USPSTF defines high-risk individuals as being between the age of 55 to 80, current or former smokers, who have quit within the past 15 years, and have a minimum of a 30 pack-year history of cigarette smoking [12]. Under the Affordable Care Act, LDCT screening is considered a preventive care service with coverage by most private insurance plans [12]. In 2015, the Centers for Medicare & Medicaid Services (CMS) added coverage for LDCT screening of high-risk individuals [13] and integrated smoking cessation as part of screening. In fact, screening has created a “teachable moment” that can change smoking behavior [14–21]. The newly proposed USPSTF guidelines have suggested even broader entry criteria as does the European Union [11, 22].

The Veteran population has higher cigarette smoking rates than non-Veterans [23, 24] and lung cancer screening programs may be particularly beneficial in this population. The Veterans Health Administration (VHA) has begun to implement lung cancer screening programs that combine early detection with primary prevention (tobacco cessation) [25–27]. Between 2013 and 2015, VHA conducted the Lung Cancer Screening Clinical Demonstration Project at eight Veterans Affairs Medical Centers (VAMCs) [28]. After two years, participating VAMCs screened approximately 2,100 Veterans and identified 31 lung cancers, of which 64.5% were stage I [26]. Significant variability in implementation was observed between sites, especially around the processes of identifying eligible Veterans, identifying the staff who engaged in shared decision-making, assigning responsibilities to coordinators, trainings for primary care and radiology, communicating screening results to patients, and assigning responsibility for follow up of abnormalities not related to lung cancer (emphysema, coronary artery disease, etc.) [26]. This initial evaluation by VHA was limited in its exploration of local contextual factors as well as associations between processes of care and adaptations over time [26].

Lung cancer screening utilization in the United States is exceedingly low (<14%) [29–31], including in VHA [32]. This makes strategies to improve implementation critically important. Building upon VHA’s Lung Cancer Screening Clinical Demonstration Project, the Veterans Affairs Partnership to increase Access to Lung Screening (VA-PALS) initiative was developed by its leadership team with the overall vision to implement lung cancer screening programs that improve Veteran outcomes through evidence-based and high-quality early lung cancer detection and primary prevention [33–35]. VA-PALS was designed to consider the unique local environments, distinct culture, and varying levels of resources across VAMCs in disparate geographic locations (Figure 1). The overarching aims of VA-PALS are to:

1. Implement evidence-based lung cancer screening programs by developing and deploying the open-source VAPALS-ELCAP Management System adapted for the needs of the VA to track Veterans from the initial contact through a long-term screening and diagnostic process that could eventually lead to lung cancer diagnoses, providing training of navigators, radiologists and other team members, assessing screening protocol adherence, providing continuous quality assurance assessment, and assessing outcomes at the 10 high-volume VAMCs.

2. Perform a formal evaluation of the 10 nascent lung cancer screening programs to measure the impact of increasing Veterans access to lung cancer screening, rates of early detection, and opportunity for mortality reduction.

Fig. 1.

VA-PALS Initial Sites.

VA-PALS refers to Veterans Affairs Partnership to increase Access to Lung Screening.

Lung cancer screening implementation has not been rigorously studied using an implementation science theory-based approach [26, 36]. The purpose of this protocol to describe the scientific frameworks and protocol developed to inform and evaluate the implementation of lung cancer screening programs at VAMCs participating in VA-PALS (Figure 1). VA-PALS lends itself as a unique opportunity to explore theory-based relationships between types of strategies (the navigators as a team-based strategy, the phantoms as an intervention characteristic-based strategy, and the software management system as a workflow processes-based strategy) and their effects on evaluation constructs (reach, effectiveness, adoption, implementation and maintenance) across multiple settings. In addition, we will also validate an organizational readiness measurement tool [34] in a national cohort of VHA employees, assess reach at multiple points along the screening process, explore barriers of rural Veterans for future outreach interventions, and report adaptations in screening processes overtime in multiple programs. These data will likely generate new knowledge that will influence lung cancer screening program implementation more broadly within and outside the VHA.

2. Materials and Methods

2.1. Aim

The aim of this program evaluation is to develop and report lung cancer screening workflow and clinical outcome measures as well as contextual factors that influence implementation at sites participating in VA-PALS.

2.2. Program Status

This program evaluation was approved by VA Central Institutional Review Board (C-IRB E19–05) on September 23, 2019 and the VA Tennessee Valley Healthcare System Research & Development Committee on November 21, 2019. The VA Organizational Assessment Subcommittee approved the study on October 1, 2019 and the VA Office of Labor and Management Relations (national union approval) approved the study on January 27, 2020. This report was approved by the VA Office of Rural Health on March 17, 2020.

2.3. Implementation Strategies

Each VAMC participating in VA-PALS has a designated site champion and will receive the following: (1) a full-time lung cancer screening navigator (nurse practitioner [NP] or physician assistant [PA]) to support program development and coordinate screening-related care, (2) a comprehensive, open-source lung cancer screening management software system [33, 35], (3) training for navigators and radiologists, (4) expert guidance to standardize screening protocol adherence and LDCT imaging quality, and (5) a support network led by experienced screening leaders.

Decisions regarding individual site’s lung cancer screening program structure and workflow will be determined at the local level depending on the environment, resources available, and culture. As such, sites will be able to adapt their processes over time for different reasons (e.g., contextual pressures, leadership support, trial and error, etc.). These changes will be measured to enhance our understanding of how naturalistic implementation influences processes and the impact these changes may have on overcoming barriers.

2.4. Logic Model (Figure 2)

Fig. 2.

VA-PALS Program Evaluation Logic Model.

ELCAP refers to Early Lung Cancer Action Program; VAPALS-ELCAP refers to Veterans Affairs Partnership to increase Access to Lung Screening-Early Lung Cancer Action Program.

2.4.1. People.

VA-PALS will be overseen by an inter-professional leadership team of radiologists, a pulmonologist, a radiation oncologist, a screening navigator, and technology specialists at Atlanta VA Medical Center, Phoenix VA Health Care System, Early Lung and Cardiac Action Program at Mount Sinai Health System (MS-ELCAP), and Paraxial LLC who have expertise in lung cancer screening, treatment, research and healthcare computer science.[35] The software team is comprised of software developers with experience in VA networks and informatics experts at MS-ELCAP. The VA-PALS program evaluation team includes health services researchers at VA Tennessee Valley Healthcare System Geriatric Research, Education and Clinical Center and Vanderbilt University Medical Center working together with the VA-PALS leadership team and participating VAMCs. Participating VAMCs have or will have local teams that include a site champion, program director(s), screening program navigator, clinical providers, and software engineers.

2.4.2. Technology

The ELCAP Management System was designed to manage and organize lung cancer screening activities and has been in use since 1992 [35]. This software system has been used at 82 institutions in 10 countries and is a tool for navigators and radiologists to document, organize, and manage screening-related activities from the initial contact, screening follow up, evaluation of screening findings, and treatment of those diagnosed with lung cancer [2, 4, 35]. This software also serves as a database for improvement initiatives [35]. Other technological inputs include the CT scanners for screening examinations and small nodule phantoms of the CT scanning protocols for quality assurance [35]. A phantom is a specially designed object that can be scanned by a CT scanner and provides the information about the performance of the CT scanner and scanning parameters which can be used to optimize CT screening images.

2.4.3. Activities

2.4.3.1. Support Network Activities

The leadership team will provide guidance and support for each VA-PALS program through planned activities and site monitoring. It will provide standardized training for all newly hired navigators and radiologists at each VA-PALS site along with a support network for the navigators, site champions and program directors through regular conference calls, national workshops, and an operational guide to lung cancer screening. The leadership team will also manage a website (www.va-pals.org) to disseminate information and shared resources.

2.4.3.2. Quality Assurance Activities

The software team will translate the ELCAP Management System into the open source VAPALS-ELCAP Management System with guidance by the MS-ELCAP and leadership team. The software documents enrollment, screenings, follow-up, interventions, and treatment if diagnosed with lung cancer. It also provides management and quality assurance reports of screening protocol adherence and follow-up. This VAPALS-ELCAP Management System will then be integrated with the Veterans Health Information Systems and Technology Architecture (VistA) and the VHA’s electronic health system electronic medical record, Computerized Patient Record System (CPRS) [35]. It will be tested and piloted at the Phoenix VAMC prior to dissemination to other participating VA-PALS VAMCs. The leadership team will provide each VAMC with an imaging phantom and the software team will provide feedback on their scanners’ imaging quality [35]. Phantoms will measure imaging sensitivity, dynamic range, contrast/detail detectability, and spatial resolution. This calibration tool will ensure proper parameters are being used for LDCT imaging to assure high-quality assessment of lung nodules [35]. Local teams at each VAMC will use the VAPALS-ELCAP Management System and its quality assurance reports as well as the CT phantom assurance tools for program activities, adherence to the screening protocol, standardization of image parameters, and improvement initiatives.

2.4.3.3. Training Activities

Navigators will be trained by the MS-ELCAP and leadership teams on screening eligibility, shared decision-making, use of the management system, and coordination of screening-related activities. Radiologists will also be trained in the workflow and use of the VAPALS-ELCAP Management System.

2.4.3.4. Local Lung Cancer Screening Activities

Each site director will be tasked with hiring a lung cancer screening navigator (NP or PA), engaging their local stakeholders and administrators, developing a screening workflow, educating local providers and staff about the program, and reaching out to Veterans. Implementation of the lung cancer screening program will include the following activities: identify high-risk Veterans eligible for screening, perform shared decision-making, offer resources and/or smoking cessation treatment, perform the LDCT scan, ensure structured radiology reporting, facilitate any indicated additional workup, and schedule either repeat annual LDCT or work up of findings suspicious for lung cancer as appropriate.

2.4.4. Data Collection Activities

The primary clinical and implementation outcome measures were developed by the VA-PALS leadership team with the program evaluation team. The combined teams will work together to collect data from the management system, VHA’s Clinical Data Warehouse (CDW), surveys, and interviews as well as analyze results and share the data with the larger VA-PALS network. The program evaluation team will also participate in leadership and program meetings.

2.4.5. Short- and Long-term Outcomes

2.4.5.1. Short-term Outcomes:

The short-term outcomes consist of VA-PALS’s reach, adoption, effectiveness and implementation measures to be used within the first 24 months of the program, described in detail below and Table 1.

Table 1:

VA Partnership to increase Access to Lung Screening Program Evaluation Measures

Framework(s) Domains and Associated Measures*	Definition	Data Source
RE-AIM Reach/CFIR Implementation Measures (Short and Long-Term)
Veterans potentially eligible to participate (n)	Inpatient/outpatient Veterans per year per facility age 55–80; estimate eligible smoking history (~32% of age appropriate Veterans based on prior study)[26]	VINCI
Veterans approached to participate (n)	Veterans approached by LCS program	VAPALS-ELCAP
Veterans eligible and agreeing to participate after shared decision-making (n)	Eligible and agreeing	VAPALS-ELCAP
RE-AIM Effectiveness/CFIR Clinical Effectiveness Measures (Short and Long-Term)
Exam results based on Lung-RADS/I-ELCAP (n)	Screenings in each Lung-RADS/ELCAP category	VAPALS-ELCAP
Cancers detected (n)	Number of lung cancers detected	CDW or VAPALS-ELCAP
Pathologic/clinical stage of screen detected cancers	Stage of each cancer detected	CDW and chart review or VAPALS-ELCAP
Histologic type screen detected cancers	Type of each cancer detected	CDW and chart review or VAPALS-ELCAP
Non-lung abnormalities detected by screening (n)	Incidental findings detected classified by type	VAPALS-ELCAP
Downstream interventions (n, type)	PET scans, bronchoscopies, mediastinoscopies, biopsies	CDW or VAPALS-ELCAP
Treatments (n, type)	Surgeries, radiation therapy, systemic therapy	CDW or VAPALS-ELCAP
Smoking cessation referrals (n)	Referrals placed to smoking cessation program	CDW or VAPALS-ELCAP
Quit attempts (n)	Veteran attempts to quit smoking	VAPALS-ELCAP
Successful quit attempts (n)	Veteran self-reports successfully quitting for at least 6 months	VAPALS-ELCAP
Types of smoking cessation resources utilized (n)	Medications prescribed for smoking cessation (nicotine replacement, varenicline, bupropion)	VAPALS-ELCAP
Program referral to contact by navigator (n)	Days from referral date to program contact on intake form (for programs with referral orders)	VAPALS-ELCAP
Days from exam order to date of scan (n)	Days from order of LDCT screening to scan performance	VAPALS-ELCAP
Days from scan to intervention performed (n)	Days from a scan to a follow up intervention (PET, bronchoscopy, pulmonology consult, surgery)	VAPALS-ELCAP
RE-AIM Effectiveness/CFIR Clinical Effectiveness Measures (Long-Term)
Mortality	Survival rate of Veterans screened in	CDW and VAPALS-
RE-AIM Effectiveness/CFIR Clinical Effectiveness Measures (Short-Term)	lung cancer screening programs	ELCAP
Provider experience	Provider experience in lung cancer screening	Electronic survey
Veteran experience	Veteran experience in lung cancer screening	Electronic survey
RE-AIM Adoption/CFIR Implementation Measures (Short-Term)
VAMC location	City where VAMC is located	CDW/VHA website[52]
VISN	VISN where VAMC is located	CDW/VHA website[53]
U.S. Region	U.S. Census Region where VISN is located	CDW/VHA website[53]
Complexity Level	Complexity Level of VAMC***	CDW/VHA website[54]
First use of VAPALS-ELCAP software	Date of first use of VAPALS-ELCAP Management System	VA-PALS leadership
First use of phantom	Date of first use of phantom	VA-PALS leadership
Navigator position posted	Date the navigator position was posted	Interview with VA-PALS champion
Navigator official start date	Date the navigator started the position	Interview with VA-PALS champion
Navigator training date	Date the navigator attended I-ELCAP training	VA-PALS leadership
Organizational readiness	Organizational readiness for lung cancer screening in the first 12 months of screening program (reach, barriers, adaptations)	Electronic survey
Lung Cancer Screening Program Team characteristics	Roles and specialties of all persons involved in the operation of local screening programs	Interviews with navigators and champions
Degree of leadership buy-in	Degree of leadership support in form of resources, meetings, and communication in first 12 months	Interviews
Alignment with VISN priorities	Alignment with VISN priorities in first 24 months of program	Interviews with local program leaders
Unique provider referrals (n)	Unique provider referrals in months 1–24	CDW or VAPALS-ELCAP
Referring provider characteristics (n)	Referring provider type and specialty	CDW or VAPALS-ELCAP
RE-AIM Implementation/CFIR Implementation Measures (Short- and Long-term)
Navigator length of work in days (n)	Number of days each navigator works as the screening program navigator	Interviews with VA-PALS champion and leadership
Navigator workload (n)	Volume of Veterans encountered by each navigator	CDW or VAPALS-ELCAP
Navigator use of VAPALS-ELCAP software	Number of screening-related activities (approach Veterans, enroll Veterans, write notes, track screening results, manage follow-up procedures, manage annual screening follow-up) that navigator uses VAPALS-ELCAP and/or other tracking system	Interview with navigators for process maps
Radiologist use of VAPALS-ELCAP software	Proportion of radiologists using VAPALS-ELCAP software out of all radiologists who interpret LDCTs at each site	Interview with radiology champion
Phantom use	Dates phantom is scanned by CT and dates adjustments are made to CT imaging protocol	Interview with local radiology champion and VAPALS leadership
FRAME Measures (Short and Long-term)
Quality of LDCT exams	Phantom used, feedback incorporated into imaging protocol	Interviews with VA-PALS leadership
Process maps	Description of how each program performs: Veteran identification, eligibility confirmation, smoking cessation services, shared decision-making, screening exam performed, delivery of results, downstream activities for abnormal studies	Interviews with VA-PALS navigators
FRAME Measures (Long-term)
Adaptations to program	Individual site adaptations	Interviews with VA-PALS navigators
FRAME Measures (Short-term)
Navigator permanent position	Date that navigator position was made permanent	Interviews with VA-PALS champion
Barriers and facilitators to implementing lung cancer screening program	Qualitative data from in-depth interviews	Interviews with VA-PALS providers, staff, leadership
RE-AIM Maintenance/CFIR Implementation Measures (Short- and Long-Term)
Subsequent LDCTs ordered (n)	Number of subsequent annual LDCTs ordered after initial LDCT	CDW or VAPALS-ELCAP
Screening adherence	Percent Veterans returning for annual screening within 9–15 months of previous screening; sensitivity analysis using 6–18 months	CDW or VAPALS-ELCAP
Radiologist adherence to VAPALS-ELCAP software	Percent radiologists utilizing software after first 6 months	Interviews with VA-PALS radiologists
RE-AIM Maintenance/CFIR Implementation Measures (Long-Term)
Navigator retention	Number navigators retained in months 25–36 months, turnover at individual sites	Interviews with VA-PALS leadership and site champions
Unique provider referrals	Unique provider referrals in months 25–36	CDW or VAPALS-ELCAP
Unique Veteran referrals	Unique Veteran referrals in months 25–36	CDW or VAPALS-ELCAP
Process maps	Description of how each program performs: Veteran identification, eligibility confirmation, smoking cessation services, shared decision-making, screening exam performed, delivery of results, downstream activities for abnormal studies in months 25–36	Interviews with navigators
Sustained leadership buy-in	Amount and type of leadership support of lung cancer screening program in months 25–36	Interviews with leadership, VA-PALS champions
Fidelity measures	Fidelity measures defined above will be followed during months 25–36	Interviews with VA-PALS champions, leadership, and navigators; CDW

^*Domains are based on the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, the Consolidated Framework for Implementation Research (CFIR), and the Framework for Reporting Adaptations and Modifications to Evidence-based Interventions (FRAME) frameworks [38, 40, 46];

^**Short-term refers to ≤ 24 months since start of the program; Long-term refers to > 24 months since start of program. Providers interviewed include physicians and advanced practice providers in primary care, radiology, and the program clinical coordinators; VA-PALS refers to VA Partnership to increase Access to Lung Screening; VINCI refers to VA Informatics and Computing Infrastructure; CDW refers to the Veterans Health Administration’s Corporate Data Warehouse; SDM refers to shared decision-making; VAPALS-ELCAP refers to the VAPALS-ELCAP Management System to be developed by VA-PALS

^***VAMC complexity score consists of five complexity levels: 1a, 1b, 1c, 2, and 3, where 1a is the most complex and 3 is the least complex. This ranking system takes the following into consideration: (1) volume and patient case mix, (2) clinical services provided, (3) patient risk calculated from VA patient diagnosis, (4) total resident slots, (5) an index of multiple residency programs at a single facility, (6) total amount of research dollars, and (7) the number of specialized clinical services [54].

2.4.5.2. Long-term Outcomes

The long-term outcomes are the reach, implementation, effectiveness and maintenance measures to be used beyond 24 months since start of the program, described in detail below and in Table 1.

2.5. Target Population

The target population will include Veterans eligible for lung cancer screening, providers and staff involved in screening activities, and leadership at each VA-PALS site. Veterans eligible for lung cancer screening generally include those meeting the USPSTF criteria [12]; at least one site is using a more broad eligibility recommendation [37]. Providers involved in screening activities will include primary care providers (physicians, advanced practice providers, and physicians-in-training), radiology providers, and navigators for each program. Staff involved will include primary care nurses, schedulers, clinical application coordinators (CACs) and radiology technicians/technologists. Leadership will include the local program leadership as well as service leaders within primary care, radiology, and specialty services and executive leadership team.

2.6. Theoretical Frameworks

2.6.1. Implementation Framework

Implementation of an evidence-based practice into healthcare delivery depends upon multiple internal and external contextual factors that ultimately influence the processes and outcomes of healthcare delivery. We selected the Consolidated Framework for Implementation Research (CFIR) to guide this program evaluation based on this framework’s ability to provide structure to approaching the implementation of a complex, inter-professional program in a pragmatic manner [38]. The CFIR explores the characteristics of an evidence-based practice, the inner and outer settings of where the evidence-based practice is deployed, the characteristics of the individuals interacting with the evidence-based practice, and the process of implementing the evidence-based practice [38]. This mixed-methods program evaluation measures key elements of the five major CFIR domains: (1) characteristics of the evidence-based practice (image acquisition and CT standardization), (2) inner setting of each VA-PALS site (organizational readiness, resources) and (3) outer setting of VHA as a whole (policies), (4) the characteristics of individuals involved in the screening process (knowledge and beliefs, self-efficacy, and motivation), and the (5) process of implementation at each VA-PALS site (site process maps for implementation initially and adaptations over time) (Table 1). Thoroughly exploring each of these domains will inform future implementation strategies (Figure 3) [38, 39].

Fig. 3.

Consolidated Framework for Implementation Research & VA-PALS Program Evaluation. LDCT refers to low-dose computed tomography; VAMC refers to Veterans Affairs medical center.

2.6.2. Evaluation Framework – RE-AIM

The Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework was used to develop the program evaluation measures [40]. This evaluation framework has been used in the cancer screening setting [41] and provides a comprehensive structure to assess real-world clinical programs. VA-PALS evaluation measures reflect the activities at each VAMC and clinical and process outcomes of each lung cancer screening program. For each site, the date of hire of the navigator will be considered the program start date. Evaluation of all measures of reach, effectiveness and maintenance will begin at the program start date and continue for the next 36 months, as defined by the navigator hire date. The VA-PALS leadership and evaluation teams will compare measures (i.e. number of Veterans reached, screening utilization, etc.) over time in VA-PALS and non-VA-PALS VAMCs in an interrupted time series analysis. Furthermore, we will compare outcomes in months 1–24 to months 25–36 to evaluate sustainability of VA-PALS and non-VA-PALS VAMCs (Table 1).

2.6.2.1. Reach

The management system and the VHA’s Corporate Data Warehouse (CDW) will capture descriptive information (e.g. age, smoking history, rural status), method of contact for screening (referral from primary care, specialty care or smoking cessation, outreach or self-referral). In order to have a consistent denominator of the eligible population across all sites, we will estimate the number of Veterans potentially eligible for lung cancer screening according to the USPSTF criteria. As used before [32], we will obtain counts of Veterans between the ages of 55 through 80 seen at each VAMC each year. To calculate the percentage of age-appropriate Veterans with an eligible smoking history, we will multiply the age-appropriate population by 32% for each VAMC in each year; thirty-two percent was the percentage of age-appropriate Veterans who met the smoking eligibility criteria in the VHA’s Clinical Lung Cancer Screening Demonstration Project [26]. All of the components of the program’s reach will be designed to capture the number of eligible Veterans at each stage of the process including Veterans approached for screening, eligibility as well as agreement and refusal both before and after the shared decision-making process, and unique Veterans completing initial screens (Figure 4).

Fig. 4.

VA-PALS Program Evaluation Reach.

Eligible Veterans refers to Veterans meeting the United States Preventive Services Task Force eligibility criteria.

2.6.2.2. Effectiveness

The effectiveness of the program will be evaluated using data acquired from the CDW and the VAPALS-ELCAP Management System. The measures chosen are reflective of the results of screening, the frequency and type of follow-up procedures performed, the number of cancers detected, type and stage of cancers, treatment received, other abnormal findings not suspicious of lung cancer and their follow up (e.g. cardiovascular disease, COPD, etc.) [42], survival of screened Veterans, the number of smoking cessation quit attempts [14, 16], the type of smoking cessation services provided, wait times for services provided, and overall experiences of those involved in the screening process (see Qualitative Section) (Table 1). We will compare different measures of effectiveness (number of LDCT scans, cancers diagnosed, stage at time of diagnosis, etc.) before and after implementation of the navigator and VAPALS-ELCAP Management System using an interrupted time series analysis. Other measures of effectiveness (wait times for services provided, workflow process maps, and overall experiences of those involved in the screening process) will also be tracked over time using the CDW, process mapping and surveys.

2.6.2.3. Adoption

Adoption will be assessed by describing the characteristics of the VAMC facilities participating in VA-PALS. Organizational readiness for change will be measured using a validated survey tool to assess change valence, change commitment, and change efficacy [34, 43] to understand each VAMC’s inner setting (Table 1). Organizational readiness for change is defined as the “extent to which organizational members are psychologically and behaviorally prepared to implement organizational change.” [44] Change valence is defined as the organization’s members belief that pursuing change is beneficial and valuable to the organization [45]. To describe the program implementation process, we will obtain interviews from sites and program leadership: (1) first date of posting the navigator position for hire by human resources, (2) first day navigator starts clinical work, (3) date of navigator training, (4) date of installation of the VAPALS-ELCAP Management System, and (5) dates of imaging phantom use and CT scanner standardization. The number of unique providers and types of providers (specialty; physician, advanced practice provider) referring to the program within the first 24 months since hiring of the coordinator will be captured in the CDW and VAPALS-ELCAP Management System. Finally, to describe adoption at the site-level, we will capture alignment with VHA priorities through in-depth interviews of program leadership at each site.

2.6.2.4. Implementation

Implementation of lung cancer screening programs at the 10 VA-PALS sites will be captured via local program engagement to assess fidelity of the program (Table 1). Specifically, fidelity refers to the degree to which navigators, the management system, trainings, screening protocol adherence, and imaging phantoms are utilized at each site. Measures of fidelity will include the retention time of navigators, navigator workload, the number of radiologists who use the management system, the degree to which the navigators use the management system for screening related activities (approach/enroll Veterans, write notes, follow-up screening results, manage downstream follow-up, etc.), and the dates of phantom use and image standardization (Table 1). Measurement of each VAMC’s inner setting will assess team characteristics, the processes involved in screening, and barriers and facilitators. We will assess how smoking cessation is incorporated into each local program (who performs it, type of services performed, when it is performed within screening), how shared decision-making discussions are performed, and how and when results are disseminated to patients and referring providers (see section on Process Maps below). These measures will be primarily obtained through interviews with sites and supplemented with data from the VAPALS-ELCAP Management System (Table 1).

2.6.2.5. Maintenance

We will measure the extent to which VA-PALS is sustainable over time using a combination of data collected from the VA-PALS-ELCAP Management System, the CDW, and interviews with local program leadership. Maintenance measurements will consist of: number of subsequent screenings, screening adherence, screening workflow adaptations during months 25–36, the number of unique Veterans enrolled in each program during months 25–36, the number of unique providers who refer Veterans in each program during months 25–36, navigator retention, and sustained leadership buy-in (Table 1). We will also continue to track radiologists’ and lung cancer screening navigators’ continued adherence to the VA-PALS ELCAP Management System per the fidelity measures described in the Implementation Section above.

2.7. Process Maps and Adaptations

Each VA-PALS site will describe their initial processes of clinical workflow, which will be used to generate a process map at the start of implementation. These process maps will be generated via telephone interviews with each site’s lung cancer screening navigator. Each map documents the site’s method of conducting patient identification, eligibility confirmation, smoking cessation counseling, shared decision-making, appointment scheduling, LDCT appointment, communication of results and follow-up scans. Each site will be contacted every 6 months to review their process map and delineate workflow adaptations as implementation matures. Using the Framework for Reporting Adaptations and Modifications to Evidence Based Interventions (FRAME) [46], we seek to understand how and why processes are adapted as well as how these adaptations influence implementation outcomes over time. Adaptations will be noted across the following FRAME domains: 1) when and how over the course of implementation were modifications made, 2) were these changes planned and proactive (e.g., intentional adaptation) or unplanned and reactive (e.g., in response to other forces), 3) who decided changes were necessary, 4) what was modified, 5) at what level in the system of delivery were changes made, 6) what types of changes were made in content or context, and 7) the reasons modifications were made and what contextual drivers were at play. Once process maps are developed or revised, they will be reviewed with each site for accuracy. Adaptations will be correlated with implementation outcomes (workflow, barriers and facilitators).

2.8. Qualitative Evaluation

Qualitative data will supplement the quantitative measures to recognize elements that were successfully implemented, and elements that posed a challenge at each site (Table 2). In-depth interviews among providers, staff and leadership at five sites with different process maps will be used to assess barriers and facilitators around the main factors related to implementation. Key questions asked will be: (1) “Tell me about what role you currently have in the lung cancer screening program at your VA” (CFIR individual/team characteristics); (2) “What is difficult about implementing a lung cancer screening program?” (RE-AIM implementation); (3) “How well does lung cancer screening fit with existing processes and practices in your VA?” (inner setting, implementation climate, compatibility); (4) “Are meetings, such as staff meetings, held regularly to discuss work process and practices such as lung cancer screening?” (inner setting – network and communication); (5) “What could help you to continue to perform high-quality LDCT screening in your VA?” (RE-AIM maintenance). Targeted participants for interviews include primary care and radiology providers, staff, and leaders. All interviews will be audio-recorded and transcribed for subsequent thematic analysis, a standardized system that produces valid and reliable interpretations. All transcripts will be coded to establish a hierarchical coding system that categorizes comments contextually into themes that occur across multiple interviewees [47, 48].

Table 2:

Qualitative Data Collection

Interview Type and Framework(s) Domain*	Interviewee	Interview Items
Barriers and Facilitators
CFIR Individual Characteristics	Providers, Staff, Administrators	How long have you been in your role in radiology/primary care?
		Where else have you practiced/worked in the past 5 years? What percentage of your clinical practice live in a rural area? (providers only) Tell me about your thoughts on lung cancer screening? Tell me about your role in lung cancer screening. What have been the difficulties in performing lung cancer screening?
	Providers only	What percentage of your clinical practice live in a rural area?
CFIR Measures of Implementation; RE-AIM Implementation	Providers, Staff, Administrators	How long have you been in your role in radiology/primary care?
		What kinds of changes in the current support system could help you? What is working well with lung cancer screening at your VA?
	Providers only	What is working well for screening Veterans who live in rural areas? What has been difficult in screening Veterans from rural areas? What could help you perform lung cancer screening? What could help screen rural Veterans?
CFIR Process of Lung Cancer Screening	Providers, Staff, Administrators	How well does lung cancer screening fit within the existing work processes and practices in your VA?
CFIR Team Characteristics	Providers, Staff	Are meetings held regularly to discuss work processes and practices such as lung cancer screening?
CFIR Measures of Implementation; RE-AIM Maintenance	Providers, Staff, Administrators	What are your hopes for the future of lung cancer screening?
		What could help maintain the program?
Initial Process Map Interviews
CFIR Individual Characteristics	Program Navigators	Can you give me the date of the first day you worked as the program navigator?
		How many days a week do you work as the program navigator? Have you gone through navigator training with Mt. Sinai?
CFIR Team Characteristics	Program Navigators	How many navigators are at your site for the lung screening program? Were there any previous navigators? If so, how many and how long did they serve in this role if you know? What do you do when you are unsure of something? How are challenges handled by the team?
CFIR Process of Lung Cancer Screening	Program Navigators	Do you currently use the VAPALS-ELCAP Management System? If so, please provide the date of the first day you used it?
		Tell me about how Veterans are screened at your VA (program entry, eligibility verification, rural outreach, shared decision-making, scheduling smoking cessation, screening exam ordering, screening exam visit, type of standardized screening protocol, follow up on results, work up of suspicious findings, incidental findings, annual repeat scan, screening tracking and management of large panels of patients)
Initial and Follow-Up Process Map Interviews
CFIR Team Characteristics	Program Navigators	Please describe who is on your team, specifically each person’s role and specialty. Describe how you work as a team on a day-to-day basis and also as a program.
CFIR Inner Setting	Program Navigators	Do you interact with leaders at your VA besides your program director? If so, please describe. Does your VA provide you with resources to do your job? If yes, what resources? If no, what resources would help you do your job?
Follow-Up Process Map Interview
FRAME	Program Navigators	When and how were modifications made?
		Were these changes planned and proactive (e.g., intentional adaptation) or unplanned and reactive (e.g., in response to other forces)? Who decided changes were necessary? What was modified? At what level in the system of delivery were changes made (executive leadership, department/service line leadership, healthcare provider, or staff level)? What were the reasons modifications were made and what contextual drivers were at play?

^*Domains are based on the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, the Consolidated Framework for Implementation Research (CFIR), and the Framework for Reporting Adaptations and Modifications to Evidence-based Interventions (FRAME) frameworks [38, 40, 46]

3.0. Discussion

This report presents a structured and comprehensive evaluation protocol that will assess lung cancer screening program implementation across multiple VAMCs within an integrated healthcare system that has a decentralized leadership structure [49]. This affords an opportunity to evaluate lung cancer screening program implementation in disparate cultures and clinical environments where priorities and resources may vary. The protocol will assess clinical process adaptations over time and examine how these adaptations influence implementation. The measures described and the overall program evaluation will generate knowledge that may influence the implementation of lung cancer screening programs more broadly throughout the VHA and beyond. It may help address the current national estimates that demonstrate exceedingly low rates of lung screening across the US [29–32].

This program evaluation will advance the field of implementation science primarily through its theory-based approach. We will explore the interaction of strategies that act at the team, intervention and process of care levels (i.e. navigators, phantoms, management system) with constructs within implementation science frameworks, or perhaps, discover new constructs that are important in implementation of complex, imaging-based screening programs. Furthermore, we will be the first to validate an organizational measurement tool [34] in a large, VHA population. Finally, we will assess the Veteran reach of lung cancer screening at multiple points along the screening pathway and explore barriers of rural Veterans and adaptations to screening programs over time, areas that have not been well studied in the VHA.

There are several unique and important challenges in implementing lung cancer screening compared to other cancer screenings that may be contributing to delayed uptake since publication of the USPSTF recommendations in 2014. First, identification of eligible individuals at-risk is difficult because this step depends upon accurate smoking histories, which are often unavailable, recorded incorrectly or in an unstructured format in the medical record [26, 50]. Then, if eligible individuals are identified, screening participants and providers are encouraged (and required by CMS for reimbursement) to engage in shared decision-making during which the potential benefits and harms of lung cancer screening are discussed; this requires time as well as a clinician who is confident in their ability to perform a shared decision-making discussion [13, 51]. Furthermore, programs are encouraged to incorporate smoking cessation services for current smokers, which is critically important but adds time and requires clinician familiarity with this service [12, 13]. VA-PALS programs plan to address these challenges by shifting some of these tasks away from busy primary care clinicians, streamlining workflow, and coordinating downstream screening-related activities. This program evaluation can shed more light on this model.

There are several operational considerations in this program evaluation plan. Some measures of program effectiveness (clinical outcomes) rely upon the timing of the installation of the VAPALS-ELCAP Management System at all 10 sites. This is also during a time when the VHA is planning to transition its electronic health record system to Cerner. The VAPALS-ELCAP Management System will be open-source and compatible with Cerner, though such transition may interrupt its functionality. Next, although we describe a comprehensive, theory-based program evaluation, additional limitations may emerge given a reliance on surveys and in-depth interviews to capture patient and provider experiences. As with all survey and qualitative research, self-reported outcomes are limited by recall bias, social desirability bias, and selection bias. We will attempt to overcome these challenges by seeking a high response-rate with local champions and research incentives. We will also conduct a sufficient number of interviews to reach thematic saturation. Another consideration is that a few of the data fields within the VAPALS-ELCAP Management System allow for free text in which users can type their individual responses. The study team will need to analyze these data manually, which may lead to potential misclassification or misinterpretation. There are also some measures that are not directly captured by the VAPALS-ELCAP Management System, such as how communication of screening results are relayed to patients and providers. The study team will rely on site self-reporting for these measures. Our denominator is imperfect as some sites may use more broad eligibility criteria. Additionally, smoking histories are not accurately captured in the electronic health record. However, in order to have a consistent denominator across all VAMCs, we will use the same age criteria and estimate the proportion of those who are age appropriate for screening and meet smoking history criteria according to a prior study [26, 32]. Finally, the USPSTF is expected to expand the eligible screening population to include a younger age (starting at age 50) and less smoking history (at least 20 pack-years) in the coming months [22]. This will increase the reach of Veterans, making lung cancer screening a routine service to an even larger population.

In summary, we report a theory-based, comprehensive program evaluation protocol for a multi-site lung cancer screening initiative that describes implementation and clinical outcomes. The results from this program evaluation will generate new knowledge that may influence the implementation of lung cancer screening programs more broadly as well as advance our scientific understanding of implementation of image-based screening tools in diverse settings

Research data for this article.

Datasets used in this study will be available to VA researchers in the VA’s Corporate Data Warehouse upon appropriate VA ethics and data access approvals. De-identified, aggregated data from the VAPALS-ELCAP management and tracking system, interviews, and surveys will be available upon reasonable request to the corresponding author and with VA-PALS leadership approval.

Highlights.

• Veterans Affairs is implementing lung cancer screening programs nationally.

• Programs are provided a navigator, trainings, tools, and support network.

• Scientific frameworks inform the design of this program evaluation.

• Local environments and adaptations will be assessed through interviews and surveys.

• Implementation and clinical outcomes will be measured using a retrospective cohort.