VA Telederm study: protocol for a stepped-wedge cluster randomised trial to compare access to care for a mobile app versus a workstation-based store-and-forward teledermatology process

Abstract

Introduction

Teledermatology has emerged as an important strategy to enhance access to high-quality skin care. VA Telederm is a provider-facing, web-based mobile app designed to integrate into the existing teledermatology workflow in the US Veterans Health Administration (VHA). In this study, we will conduct a systematic evaluation of VA Telederm on access outcomes in VHA facilities using a pragmatic trial guided by clinical and operational leaders.

Methods and analysis

The study is a prospective, stepped-wedge cluster randomised trial with cross-sectional exposure and outcome measurement via retrospective database analysis of administrative records. Each cluster is a VHA facility deemed eligible for the trial. We assign the intervention using a cluster-level balanced randomisation scheme based on facility size, baseline teledermatology uptake and geographic location. The trial will test whether patients receiving dermatological care at participating facilities will have better access compared with patients receiving care through the current standard process. The primary outcomes proxy for patient-level access to dermatology services, including (1) consult completion time for teledermatology consults; (2) appointment completion time for new dermatology consults; and (3) travel distance for dermatology services. As secondary outcomes, we will assess facility-level adoption outcomes, that is, the number of dermatology encounters and the proportion of teledermatology consults out of all dermatology encounters. To account for secular trends in outcomes and for correlation across individuals within clusters, we will assess the impact of the intervention using generalised linear mixed regression models.

Discussion

Streamlining the current practice for store-and-forward teledermatology in the VHA can improve access to expert dermatological care for US veterans. The lessons learnt in this trial could validate the use of mobile technology for consultative store-and-forward dermatology in a large healthcare organisation. The results may also be of interest to other medical specialties assessing the merits of implementing mobile telehealth.

Protocol version

Version 3; 7 November 2018.

Trial registration number

NCT03241589; Pre-results.

Strengths and limitations of this study.

• The stepped-wedge cluster randomised design allows for strong causal inference.

• The national coverage ensures diversity of geographic settings and organisational cultures within a public integrated healthcare delivery system.

• The close partnership with clinical and operational leaders maximises the opportunity for rigorous implementation.

• The findings of the study may not be fully generalisable to other populations and healthcare delivery systems.

• The complex implementation process requiring buy-in from multiple stakeholders may produce variations across sites.

Introduction

Background and rationale

With the dermatology workforce facing a persistent shortage^{1 2} and misdistribution of providers and services across the country,³ access to dermatology services in the USA is severely lacking. Teledermatology has emerged as an important strategy to enhance access to dermatologic care. Asynchronous or store-and-forward telehealth (SFT) teledermatology transmits still digital photographs and textual information to dermatologists who need not be present at the same time and place, while live-interactive teledermatology uses real-time video interactions to exchange medical data and skin exams. SFT teledermatology has been shown to improve access to care in diverse populations and settings, enhancing patients’ ability to receive care^{4 5} and increasing service timeliness.⁶ Besides making care more accessible, SFT teledermatology may increase efficiency, thus allowing dermatologists time to provide more in-person visits to patients with more severe conditions.^{5 7} Teledermatology may also improve access for underserved populations, particularly by allowing providers in safety-net settings to prioritise patients with the most urgent and severe conditions.⁸ The few studies which have been conducted on the impact of mobile devices in provider-to-provider SFT teledermatology show significant decreases in wait times.^{9 10} However, these studies are limited by small sample sizes recruited from single, urban clinic locations.

Access to health services is a key priority for the Veterans Health Administration (VHA), both in terms of geographic proximity and timeliness of care.^{11 12} Veterans disproportionately live in rural areas, thus accentuating the problem of geographic access to specialty care,¹³ particularly for patients seeking care from specialty providers such as dermatologists who tend to cluster in urban metropolitan areas.² SFT teledermatology has grown rapidly in the VHA,¹⁴ resulting in enhanced access of both rural and non-rural veterans to skin care and more timely skin disease treatment.^15–17

Despite recent improvements in overall wait times, access to dermatology remains challenging, in part due to inconsistent implementation of teledermatology in the VHA. Data from fiscal year (FY) 2016 show wide variation in the uptake of SFT teledermatology consult use, from zero to about 50% of all consults in a given facility. One important impediment to the adoption of teledermatology by primary care clinics may be the inefficiency of the current workstation-based process, which involves several steps performed by VHA-trained imagers before the image can be reviewed by a dermatologist.

The convenience and capability of smart mobile devices (eg, tablet computers) to combine data acquisition (ie, photographs) with efficient communication strategies creates significant opportunities for the VHA to streamline teledermatology practice and expand it more widely. Capitalising on these opportunities, the VHA Office of Connected Care (OCC) developed a mobile app, VA Telederm, as a more facile option to increase SF teledermatology use among providers. VA Telederm is designed to integrate into the existing teledermatology workflow and is interchangeable with the conventional workstation-based process. Its target users are primary care providers (PCPs) and imagers. It recapitulates the same steps that are performed by these users on workstations, but using a streamlined graphical user interface, and it permits dictation of patient histories. Importantly, imagers are able to obtain or automatically transfer patient histories as well as capture skin images, and seamlessly upload images to the electronic health record using the app, instead of using a separate camera. Since no images reside on the mobile device, there is no need for imagers to perform the extra step of deleting camera images.

In this study, we will conduct a systematic evaluation of VA Telederm impact on dermatology access outcomes in VHA facilities using a clinically driven, pragmatic trial with a stepped-wedge cluster randomised trial (SW-CRT). To our knowledge, this is the first systematic study of teledermatology roll-out nationally, encompassing rural and urban facilities across the USA. The SW-CRT design allows for rigorous assessment of the causal impact of the VA Telederm app implementation on providers’ adoption of SFT teledermatology and the associated change in patients’ access to care in a large, integrated healthcare delivery system. This trial will help expand the evidence base for the effectiveness of provider-facing mobile apps in improving adoption and access to care by introducing subtle to moderate gains in user-friendliness and process efficiency. The comparator in our study is the current standard of care in the VHA facilities selected for the trial, which includes regular in-person dermatology visits and workstation-based SFT teledermatology.

The rationale for the study design evolved from the combination of several scientific and pragmatic considerations. Scientifically, the requirement for randomisation stems from the VHA leadership’s desire to produce the most rigorous evidence on which to base system policy.^{18 19} Thus, the pilot tests proposed initially were redesigned as a fully randomised trial. Pragmatically however, the OCC’s operational need to begin disseminating and testing the app as quickly as possible to sites that were ready for implementation brought the stepped-wedge design to the forefront as a more feasible way of rolling out the highly anticipated apps to providers. It became apparent that this design would allow evaluation of the app while gradually releasing it in the field, thus using the system’s limited capacity to solve the inevitable implementation challenges in a small number of facilities at any given time.

This paper provides the trial protocol following the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement, which provides recommendations for a minimum set of scientific, ethical and administrative elements that should be addressed in a trial protocol.^{20 21} The WHO data set as required by the SPIRIT statement is included in the online supplementary file. Given the absence of specific items in the SPIRIT statement for SW-CRTs, we also include more detailed items proposed in recent methodological papers on SW-CRT design, analysis and reporting.^22–24

Primary objective

Our primary objective is to assess the impact of the VA Telederm mobile app on teledermatology consult completion times, dermatology appointment completion times and travel distance in veterans seen in outpatient dermatology practice.

Secondary objectives

Our secondary objective is to determine if the VA Telederm app affects the number of dermatology encounters (instances of care) and adoption rates of teledermatology consults in outpatient dermatology practice in the VHA.

Specific hypotheses

The study will test two specific hypotheses:

1. The VA Telederm mobile app improves access to dermatology services among VHA patients, as measured by reduced teledermatology consult completion time, appointment completion times for new dermatology consults and travel distance for dermatology services. In addition, we measure exposure to dermatologic care by the total dermatology encounters (instances of care).

2. The VA Telederm mobile app increases the adoption of teledermatology consults in VHA dermatology clinics.

Trial design and study organisation

This study is a prospective, stepped-wedge cluster randomised superiority trial. VHA facilities are randomised to receive the intervention according to a constrained randomisation scheme that also ensures balance in key facility-level characteristics across the study sequences. Patients’ exposure to the intervention will be cross sectional (ie, patients are exposed when they visit a VHA facility for dermatology care) and outcome measurement will be done via retrospective database analysis of administrative records. The trial involves a partnership between VHA telehealth clinical operations, research and implementation scientists in the VHA.

Methods and analysis

Participants and setting

Each cluster participating in the trial is a VHA facility deemed eligible for participation, defined as having patients with at least one in-person outpatient visit for dermatology or teledermatology encounter in the Veterans Affairs (VA) during the course of the study. Each facility represents a VHA Medical Center providing care for a variable number of associated medical centres or clinics. The individuals receiving dermatological care at the eligible facilities, that is, patients with at least one in-person outpatient dermatology or teledermatology encounter in the VHA during the course of the study, will be automatically included in the study and will have their outcomes evaluated by retrospective, automated statistical analysis.

Eligibility criteria

We selected 36 eligible facilities using the inclusion and exclusion criteria presented in table 1, with the rationale for each criterion. Specifically, we included facilities located within the continental United States which had a dermatology clinic on-site (thus able to implement the intervention) and which only had a moderate penetration of SFT teledermatology. We excluded three facilities participating in a mixed-methods formative evaluation of the implementation to avoid contamination of the intervention.

Table 1.

Inclusion and exclusion criteria used for selecting the participating facilities and rationale for each

	Criterion	Rationale
Inclusion	1. Located within the continental United States.	VHA facilities outside the continental United States do not reliably report electronic medical record data to the CDW and/or do not have dermatology clinics.
	2. Greater than zero provider full-time equivalents practising dermatology.	Have a dermatology clinic on-site.
	3. Higher than 0.1% and at most 8.8% of total FY2016 dermatology encounters at the facility were for teledermatology readings.	Already performing some teledermatology consults at baseline using the existing store-and-forward technology, but their total teledermatology encounter rates were below the median; based on prior experience, these sites were judged to be good candidates for implementing the new mobile app.
Exclusion	Participating in pretrial mixed-methods formative evaluation.	Three facilities will participate in the formative evaluation, which will be conducted to inform the implementation of the mobile app; these facilities are located in Providence, RI, San Francisco, CA, and Denver, CO.

CDW, Corporate Data Warehouse; FY, fiscal year; VHA, Veterans Health Administration.

The current standard of care

Teledermatology services are currently provided in VHA facilities using a workstation-based SFT process. This process is consultative and consists of several stages, depicted schematically in figure 1. When a patient or the provider has a skin concern, a referring provider (typically a PCP) initiates the consult request for skin imaging in the electronic medical record, known as the Computerized Patient Record System (CPRS), which also prompts the PCP for pertinent medical history. On receipt of the imaging consult request, an imager trained in the VHA protocol schedules the patient for imaging and transmits information from the PCP’s consult request or, and if necessary, obtains further medical history from the patient according to a scripted set of questions that are recorded in a templated CPRS note. Usually imaging appointments are made for the same day when the imaging consult is placed, though in some situations the patient or the clinic may delay the appointment to a different day. The imager captures images of the patient’s skin using a standard digital camera and must then manually upload the photos and link them with the patient’s CPRS record. Finally, the imager then generates a new consult request to a teledermatology reader, typically a board-certified dermatologist. This imager must then manually delete the images to ensure that the patient’s privacy is protected. The reader reviews the history and images and writes a CPRS note that includes an impression and recommendations for the PCP, who is then responsible for enacting them. A face-to-face visit may also be recommended as follow-up.

Figure 1.

Current workstation-based teledermatology process in the Veterans Health Administration (VHA).

The intervention

The intervention studied in this trial consists of three elements, deployed under the umbrella of the Replicating Effective Programs framework:^25–27

1. The VA Telederm app is the key technology at the core of the intervention. The app has been developed by a federal contractor to be in strict compliance with VHA standards and regulations and is anticipated to be ready for national release by January 2019 by the OCC. Clinics within the participating facilities will be provided with mobile devices (ie, tablets) which can be used to download the app and perform the new imaging process. The app will not be available for download or use on any personal mobile devices, either of the physician or the patient, as the app is intended to be used only on secure government-issued devices. Teledermatology leads at each facility will provide guidance on how to test it prior to use, and each provider will first have to enrol their device with the VHA Mobile Health Services. The app will be implemented at the facility level (up to five tablets will be made available at no cost to each facility, though facilities may have additional tablets due to prior inventory or purchase) and will be available to referring providers in a participating facility after adoption during the corresponding trial step. However, use of the app will be at the discretion of providers and imagers, and thus not all patients may have the opportunity to receive care using the new app. Screenshots of the app’s interface are included in online supplementary appendix A.

   The VA Telederm app is intended to improve the existing teledermatology workflow by seamlessly integrating image capture and upload into the VHA electronic records system. The app’s target users are both referring providers and imagers. The app will allow referring providers to submit consults using touch screen entries and will permit dictation of patient histories. Imagers will be able to process those consults without transcribing or copying/pasting histories and to capture and upload images to the CPRS using the app, instead of using a separate camera. Thus, while it recapitulates the various steps that are performed by these users on workstations, the app will have a streamlined graphical user interface integrating these steps for an improved workflow. Since no images reside on the mobile device, there is no need for imagers to perform the extra step of deleting camera images.

2. Education programmes and resources specifically targeted to providers involved in the teledermatology process (PCPs, imagers and dermatologists) have been streamlined by the study team in collaboration with the OCC. The OCC will conduct training sessions for Telehealth Leads in each VHA Veterans Integrated Service Network (VISN) and the Facility Telehealth Coordinators (FTC) at each facility, who will then be responsible for training teledermatology providers. These sessions will be conducted using the existing process in the VHA by which new processes of care and guidelines are introduced. The core materials used for training are included in online supplementary appendix B.

3. Continuing support will be made available to the participating facilities to assist them with the adoption of the new process. Technical support will be provided via a 24/7 telephone hotline to assist with any app or device-related problems encountered by the providers. Implementation support will be provided by the VISN leads during monthly calls and via email by designated OCC staff. During these calls, providers will be able to share their experiences with the new app and will receive help to address any issues they have with the new care process.

Technical field testing

Before national release, the app will be field tested at the pilot VHA facilities in San Francisco, CA, Providence, RI, and Denver, CO. During this process, the VA Telederm app will be used by providers with the goal of identifying any technical issues with the mobile app or the new clinical process it requires. Any issues identified at this stage, such as software bugs, incompatibilities with the existing systems, or security vulnerabilities will inform any modifications necessary for the national roll-out of the intervention.

Formative evaluation

To complement and inform the randomised trial, we will also conduct a mixed-methods formative evaluation in the same facilities involved in the technical pilot field testing (ie, San Francisco, Providence and Denver). The evaluation will be guided by the Organizational Theory of Implementation Effectiveness, which is based on the work of Klein and Sorra²⁸ as modified by Weiner and colleagues.^28–31 The goal of this evaluation will be to understand the factors that may impact the organisational readiness for change (ORC)^{32 33} and process of implementation, how these factors change over time and how they are associated with successful implementation and sustainability of the app. The findings of the formative evaluation will inform the process of implementing the teledermatology apps during the randomised implementation nationally as well as the implementation of future mobile clinical applications.

In addition to their initial roles in field testing, the three pilot sites—San Francisco, Denver and Providence—are appropriate for evaluation since the teledermatology leadership is located at San Francisco and Providence VHA Medical Centers and Dermatology Field Advisory Council is located in Denver. These sites may have specific qualities that impact implementation. However, since they vary in terms of both organisation and location, lessons learnt will likely translate to the mix of other VHA facilities with dermatology programmes. Differences among the three sites include: (1) San Francisco and Denver have dermatology services, whereas Providence has dermatology as a section of the medicine service; (2) San Francisco and Denver do not use dermoscopy, whereas Providence does; and (3) each facility is in a different VHA information technology (IT) administrative region.

We will first identify baseline characteristics of each organisation and implementation teams that may impact implementation process and success. This will be followed by a qualitative and quantitative assessment of readiness to implement teledermatology; monitoring of the implementation process and progress through bimonthly site reports; as well as qualitative interviews approximately 6–8 months following initial implementation of the teledermatology app; and qualitative and quantitative evaluation of programme sustainability 1 year after the first use of the teledermatology application.

Baseline assessment

The three sites will be asked to identify individuals directly involved in planning and execution of app implementation, whose work or clinical decision-making may change as a result of app implementation. These sites will be asked to submit information on these core implementation team members and processes and size and composition of the medical centres and impacted clinical services. For each separate facility, individuals representing the core implementation team will participate in a group conference call with the goal of providing detail on the planned process of implementing the app. Written bimonthly (every other month) updates of the process will be provided by the sites.

Organisational readiness for change

Core implementation team members and clinical staff from impacted services will be surveyed using the validated Organization Readiness for Implementing Change instrument, a computer-based survey developed specifically to measure aspects of the Weiner theory of ORC.³⁴ This instrument examines perceptions of organisational-level change efficacy and commitment to newly implemented interventions. In addition, we will conduct semistructured qualitative telephone interviews of the core implementation team and clinical staff at each site to assess ORC and factors that are hypothesised to predict ORC.

Stages of Implementation Completion

We will also measure the advancement of the implementation process through monthly reports from the three sites. Implementation progress will be assessed using the Stages of Implementation Completion (SIC).³⁵ SIC enumerates key preimplementation, implementation and sustainability milestones. Dates by which specific implementation milestones are reached will be identified. This information will also enable us to examine if the degree of ORC is associated with the rapidity with which sites go through implementation steps. Bimonthly reports will also include assessment of barriers and facilitators identified through the ORC measurement process. Bimonthly information will be fed back to project and OCC leadership so that programme adjustments can be made.

Programme sustainability

At 6–8 months following the start of the implementation process at the three early adopter sites, we will conduct semistructured qualitative telephone interviews, ideally with the same individuals interviewed at baseline. The goal is to understand changes in the process of using the teledermatology app and the degree of implementation over time. At approximately 1 year, we will assess the sustainability of use of the mobile apps, using the Mancini and Marek’s Model of Community-based Program Sustainability as a conceptual guide.³⁶ Specifically, we will measure the six elements essential for sustainability (leadership competence, effective collaboration, demonstrating programme results, strategic funding, staff involvement and integration, and programme responsivity) using a modification of the validated Program Sustainability Index.^{36 37} At this stage, we will again conduct qualitative interviews with the individuals involved in planning and implementation.

To inform the national roll-out, we will continually analyse results and provide feedback to stakeholders. Rapid analysis approaches will generate preliminary findings to share among the research team, followed by in-depth content analysis.

Intervention timeline

Figure 2 illustrates the main features of our stepped-wedge design for the national roll-out, using the terminology proposed by Copas et al.²² For national roll-out, the app is targeted to sites that are clinically appropriate and best positioned to benefit from the app. The evaluation of the app will be structured as a prospective, SW-CRT, a form of randomised design which delivers the intervention to all participants in a staggered fashion over time. This design has a long history in statistical research³⁸ but has only more recently been used for programme evaluation.^{39 40} In brief, as detailed below, 36 participating facilities (or clusters of patients) will be randomly assigned to receive the intervention successively in sequences of six facilities each at the beginning of each trial step. Starting January 2019, one sequence will be exposed to the app, and at subsequent 3-month intervals an additional sequence will join the intervention until all groups are exposed starting April 2020. Therefore, the trial will consist of six steps, each one-quarter apart. Unexposed cohorts in the sequences that have not yet crossed over at any given time will serve as controls. Measurements of outcomes will be performed every quarter after the start of the trial, with the exception of the quarters reserved for implementation in each sequence commencing roll-out of the intervention. Additional measurements will be performed for two baseline quarters and two postroll-out quarters.

Figure 2.

Timeline and design features for the VA Telederm stepped-wedge cluster randomised trial (SW-CRT).

Outcomes

The primary study outcomes serve as proxies for access to dermatology services at the patient level, and will be used to test specific hypothesis (1) above. Table 2 specifies the outcome measures, the sources of data and other important variables used in the study. Specifically, using statistical techniques (presented in the Statistical methods section) we will assess changes in:

Table 2.

Study outcome measures and data sources

Measure/variable	Data sources	Coding notes
Consult completion time	VHA CDW	Total time in days from consult request date to consult completed date following Pizer et al 47
Appointment completion time for new patients	VHA CDW	Total time in days from appointment create date to appointment completed date following Prentice et al 48
Travel distance for VA care	VHA CDW OPES	Average driving distance from the centroid of the patient’s ZIP code of residence
Number of dermatologic encounters by type (in-person vs teledermatology) Percentage of dermatology encounters by type (in-person vs teledermatology)	VHA CDW	Volume of total dermatology visits, both in-person and via teledermatology (total and %); coded at facility level.

CDW, Corporate Data Warehouse; OPES, Office of Productivity, Efficiency and Staffing; VA, Veterans Affairs; VHA, Veterans Health Administration.

1. Consult completion time (continuous)—the interval between the time when a teledermatology consult is requested by a PCP until and when the dermatologist completes a note in the health record with his or her medical assessment.

2. Appointment completion time for new dermatology consults (continuous)—the interval between the time when a dermatology appointment (either in-person or teledermatology imaging) is requested and when it is completed.

3. Travel distance for dermatology services (continuous)—the distance between the centroid of the patient’s ZIP code of residence and the VHA facility to which he/she is receiving care.

We will also examine measures of teledermatology adoption at the facility level over time, such as the number and relative proportion of teledermatology consults among all dermatology encounters. These measures will be used to test specific hypothesis (2) above.

Outcomes will be extracted from the VHA’s Corporate Data Warehouse (CDW), which is regularly updated with information from individual electronic patient records at each clinic. Data on clinical full-time equivalents employed in dermatology will also be extracted from the VHA’s Office of Productivity, Efficiency and Staffing in order to monitor changes in the supply of dermatologists at the study facilities. We hypothesise the teledermatology apps will have a larger impact on veterans who live in rural areas. Consequently, models will also be secondarily stratified on urban/rural/highly rural status based on the Rural-Urban Codes assigned to the ZIP code of residence for each patient.

Sample size and power analyses

Our intended sample size is 16 000 individuals receiving care at the study facilities. We conducted power analyses separately for binary and continuous outcome measures under the parameters characterising trial design. We conducted all power analyses in Stata using the user-written package steppedwedge ⁴¹ following the authors’ guidance. We implemented the power analyses by using an incomplete design matrix with a 3-month transition period with no outcome measurement, consistent with our proposed roll-out. We verified our power analyses with analytical calculations performed using an alternative package written for the R statistical software⁴² and found the two methods to be highly consistent with each other. However, because the Stata package allowed for a user-specified design matrix with a 3-month transition period, which more closely reflects our intended roll-out approach, we chose it as our preferred power analysis method. We provide the full code for the power analysis in online supplementary appendix C.

We used a level of precision α=0.05 (probability of a type I error) and a minimum power level of 0.80, corresponding to probability of a type II error β=0.20. We also assumed a total number of clusters I=36 as per the study design and the number of baseline measurements B=2 (since we will measure two preroll-out quarters). Table 3 shows other key parameters used for selected power calculations and the corresponding statistical power level calculated with the steppedwedge commands. Because of the number of values the parameters can take leads to quite a large number of combinations, in multiway sensitivity analyses we allowed the parameters to vary within certain ranges we consider reasonable. For brevity, we show calculations that are at or near the predetermined limit of desired power (ie, 0.80) to illustrate potential cut-off values for our parameters.

Table 3.

Key results of power calculations under base assumptions

Outcome	Average cluster size (per measurement occasion)	Treatment effect (Δμ or OR)	Parameter in control group (SD)	Parameter in treatment group (SD)	ICC	Power
Consult completion time	120	Δμ=−0.10	μ0=27.8 (44.3)	μ1=25.0 (44.3)	0.25	0.800
	120	Δμ=−0.10	μ0=27.8 (44.3)	μ1=25.0 (44.3)	0.30	0.800
	60	Δμ=−0.15	μ0=27.8 (44.3)	μ1= 23.6 (44.3)	0.25	0.845
	50	Δμ=−0.15	μ0=27.8 (44.3)	μ1=23.6 (44.3)	0.25	0.775
	60	Δμ=−0.15	μ0= 27.8 (44.3)	μ1=23.6 (50.0)	0.25	0.798
	70	Δμ=−0.15	μ0=27.8 (50.0)	μ1= 23.6 (50.0)	0.30	0.812
Appointment completion time	200	Δμ=−0.05	μ0= 60.0 (50.0)	μ1= 57.0 (50.0)	0.20	0.930
	140	Δμ=−0.05	μ0= 60.0 (50.0)	μ1= 57.0 (50.0)	0.20	0.820
	300	Δμ=−0.05	μ0= 60.0 (75.0)	μ1= 57.0 (75.0)	0.20	0.800
	110	Δμ=−0.05	μ0= 60.0 (50.0)	μ1= 57.0 (40.0)	0.10	0.809
	240	Δμ=−0.05	μ0= 50.0 (50.0)	μ1= 47.5 (60.0)	0.10	0.814
	400	Δμ=−0.03	μ0= 60.0 (50.0)	μ1= 58.2 (50.0)	0.30	0.830
Travel distance (miles)	550	Δμ=−0.03	μ0= 13.0 (13.0)	μ1= 12.61 (13.0)	0.20	0.812
	300	Δμ=−0.04	μ0= 13.0 (13.0)	μ1= 12.48 (13.0)	0.20	0.800
	300	Δμ=−0.04	μ0= 13.0 (13.0)	μ1= 12.48 (13.0)	0.10	0.801
	300	Δμ=−0.04	μ0= 13.0 (13.0)	μ1= 12.48 (13.0)	0.30	0.800
	500	Δμ=−0.04	μ0= 13.0 (13.0)	μ1= 12.48 (20.0)	0.20	0.814
	1000	Δμ=−0.03	μ0= 10.0 (12.0)	μ1= 9.70 (15.0)	0.10	0.812
Proportion of teledermatology encounters	1700	OR=1.10	p0=0.03	p1=0.0329	0.05	0.809
	500	OR=1.10	p0=0.20	p1=0.2157	0.05	0.844
	500	OR=1.10	p0=0.03	p1=0.1089	0.25	0.837
	450	OR=1.20	p0=0.03	p1=0.0358	0.05	0.830
	150	OR=1.20	p0=0.10	p1=0.1176	0.10	0.831
	200	OR=1.30	p0=0.03	p1=0.0386	0.10	0.828

μ₀ is the mean of the outcome variable in the control group; μ₁ is the mean of the outcome variable in the treatment group; p₀ is the proportion of interest in the control group; p₁ is the proportion of interest in the treatment group; Δμ is the difference in means between the treatment and control arms, that is, the expected treatment effect to be detected for continuous outcomes; OR is the odds ratio between the treatment and control arms, that is, the expected treatment effect to be detected for the binary outcome.

ICC, intraclass correlation coefficient.

The average cluster size per measurement occasion K (this represents the number of relevant encounters in a measured period of time, ie, quarterly) was varied depending on the outcome, with the base value estimated using encounter data extracted from the VHA CDW. Specifically, for continuous outcomes the values are shown in the first three panels of table 3. We estimated reasonable baseline values via preliminary exploration of data from eligible sites for FY2016. We estimated an intraclass correlation coefficient (ICC) r=0.268 (95% CI 0.071 to 0.427) for continuous outcomes in the eligible sites by performing a postestimation procedure after estimating a mixed model with cluster-level random effects on baseline data on consult completion times (we used full year data with I=36 and total number of observations n=589 901). We used r≅0.25 as the base ICC value for our power calculations and varied it from 0.10 to 0.30 in our sensitivity analyses. The effect size Δμ was varied between −0.03% and −0.15% for the continuous outcomes (representing the per cent change in the baseline mean).

For the binary outcomes we assumed an average cluster size K=500 and found upper and lower values between 150 and 1700 depending on other key parameter values (fourth panel). Using FY2016 data, we estimated an average proportion of teledermatology encounters of p₀=0.03 (varied up to 0.20 to allow for the possibility of increasing teledermatology before the intervention begins roll-out). We estimated an ICC of r=0.136 (95% CI 0.091 to 0.190) for the binary outcome, and we used a baseline value r≅0.10 and varied it between 0.05 and 0.20 in the sensitivity analyses. The effect size was expressed as an OR for the binary outcome and was varied between 1.10 and 1.30, which was considered clinically meaningful but also conservative for the analysis.

Recruitment

The eligible facilities were contacted by the operations partner (ie, OCC) in November 2017 to confirm their participation in the trial and to identify staff, including the Clinical Applications Coordinators and providers, who will need training in order to implement the intervention. Moreover, each facility’s FTC and the associated overseeing VISN telehealth lead will be notified by email of the mobile app’s implementation 1–3 months prior to the implementation date assigned to their specific site. These have been and will continue to be supplemented by announcements of the app and trial during weekly national VISN lead and FTC conference calls with OCC, and monthly conference calls by OCC’s teledermatology leads with the field. The FTC, supported by the VISN leads, will be responsible for disseminating information about the app to all clinical and allied support staff and recruiting their support, including informatics and IT staff. One month prior to implementation, a conference call will be held with the FTC and VISN telehealth leads to review the app and its implementation.

Assignment of intervention

We used a constrained randomisation procedure to assign the order in which the facilities will receive the intervention. Given the small number of clusters included in the trial, this procedure avoids the potential imbalance in critical facility characteristics across the trial sequences simply due to chance. In these situations, constrained randomisation has been shown to perform better in achieving baseline balance on several potential confounders than simple randomisation, matching or stratification.^{43 44} This procedure is described briefly below and more details are provided in online supplementary appendix D.

We followed a two-stage procedure in a similar vein to the approach proposed by Bertsimas et al, ⁴³ which entails first an allocation of study units (facilities) to sequences such that the difference between the sequences is minimised (the optimisation stage) followed by random assignment of the order in which the sequences will receive the intervention (the randomisation stage). Bertsimas et al developed their procedure in the context of a parallel randomised controlled trial with multiple treatments and a small number of units in each treatment arm. To our knowledge, this is the first time this procedure is adapted to an SW-CRT. Besides balancing sequence characteristics, we also randomise sequence to the order in which they receive (the same) treatment, as opposed to different treatments. Either way, the goal of systematically decreasing the differences between sequences while preserving the random component is achieved.

Site characteristics used for the optimisation stage consisted of two continuous variables and one categorical variable. Specifically, the two continuous variables were the size of dermatology practice (measured by the number of dermatology encounters in the baseline year) and the level of teledermatology activity (measured by the percentage of teledermatology consults of all dermatology appointments in the baseline year). The categorical variable was the geographic location (determined by one of the five VHA administrative regions encompassing each facility). These characteristics were chosen as they are likely to affect the implementation of the intervention and its impact on outcomes. For example, larger facilities may have more resources for implementation and stronger incentives to increase efficiency. Similarly, facilities that are already extensively providing teledermatology may be more effective in implementing it compared with facilities with lower uptake. Finally, facilities in different geographic areas differ in their practice patterns and constraints and may have systematic differences in preintervention outcome trends.

The sequences of facilities in order of implementation are shown in online supplementary appendix E. Balance across sequences in the number of visits and the percentage of teledermatology encounters is shown in online supplementary appendix F, while balance in geographic location is shown in online supplementary appendix G. We also considered balancing on facility complexity level, a VHA-specific measure that indicates the relative size and complexity of clinical services and administrative structures of a given VHA treatment facility. However, because this measure is highly correlated with the number of dermatology patients at each facility, we did not end up using this measure for balancing. Nevertheless, this procedure achieved reasonable balance on this measure by virtue of balancing on dermatology practice size (data not shown).

Blinding

The healthcare professionals (PCPs and imagers) involved in the study will not be blinded to the intervention, as it is impossible to conceal the use of the app on a mobile device compared with the current workstation workflow. The veteran patient experiences a different imaging device depending on whether the VA Telederm app is used versus a traditional workstation and auxiliary camera, so they are also not blinded to the process. However, other than the imaging technique, the consult generation process is relatively transparent to the veteran. Furthermore, data collection will be performed passively from patient records.

Data collection and management

Outcome data will be collected by the research team via automated extraction from the CDW. Data will be stored on the Department of Veterans Affairs Informatics and Computing Infrastructure (VINCI) and only accessible by the research team. VINCI is a VA Health Services Research & Development (HSR&D) resource centre that provides a secure, central analytic platform for performing research and supporting clinical operations activities. The platform includes a cluster of services for securely hosting suites of databases integrated from national data sources (such as the VHA CDW). VINCI servers for data, applications and virtual sessions are physically located at the VA Austin Information Technology Center (AITC) located in Austin, Texas. AITC hosts a secure enclave of high-performance servers and high-speed storage and has multiple layers of security and disaster recovery to prevent data loss.

VINCI maintains compliance with the guidelines established by the VHA policies and regulations. VA-credentialed research staff will be granted access to the study data along with tools for analysis and reporting in the secure virtual working environment through a certified VHA network computer. This computing environment will enable uniform security standards for access, a common point of entry for all investigators who use the data, and consistent control of data quality.

Study data will be kept in accordance with the Department of Veterans Affairs Record Control Schedule 10-1. Storage and transfer of any Personally Identifiable Information or Protected Health Information will be performed in accordance with applicable VA policies and directives, state and federal regulations, and applicable statutes including the Health Insurance Portability and Accountability Act. Standard data quality checks such as examination of outliers or significant changes over time will be conducted to identify potential problems with the data extracted from CDW. Analytical files will be built in the VINCI secure environment and will be analysed on the VINCI servers. On completion of the research project, the study principal investigators and the VA Information Security Officer will ensure that data containing sensitive, confidential information will be returned to the VA and removed from all servers, desktops, removable storage devices, and so on.

Statistical methods

We will employ data analysis strategies that account for the causal structure implied by our trial design and mitigate its potential shortcomings. Two related issues which may confound the treatment effect are the within-cluster correlation and potentially significant secular trends in the outcomes of interest given the long duration of the trial (2.5 years). In fact, the exposure of each cluster to both the control and intervention allows us to partially exploit the within-cluster variance towards estimation, which renders this type of trial less sensitive to the ICC. To ensure that these confounding factors are properly handled, we will analyse the data using several model specifications.²⁴ This will also allow us to explicitly test some of the assumptions underlying our empirical model.

Our first analysis relies on an intent-to-treat approach, in which we will directly assess the impact of being randomised to implement the VA Telederm app on the following specialty care access outcomes: consult completion time, appointment completion time and travel distance for VHA care. This model yields an estimate of the average effect of being randomised to receive the VA Telederm app (average treatment effect). From a policy perspective, this effect can be interpreted as the efficacy of deploying an app in real-world outpatient clinics, where overall uptake to clinical practice is likely less than 100%.

Specifically, we will estimate generalised linear mixed models of the form: ${\displaystyle Y_{iqt}=F\left(\mu +{\alpha }_q+{\beta }_t+\theta V_{qt}+X_{iq}\right)+e_{iqt}}$,

where

${\displaystyle Y_{iqt}}$ is the outcome for patient ${\displaystyle i}$ in cluster ${\displaystyle q}$ treated in quarter ${\displaystyle t}$;

${\displaystyle \mu }$ is the outcome in the first observation;

${\displaystyle {\alpha }_q\sim N\left(0,{\varphi }^2\right)}$ is the random effect for clusters (VHA facilities);

${\displaystyle {\beta }_t}$ is a fixed effect adjusting for being in quarter ${\displaystyle t}$;

${\displaystyle V_{qt}}$ is a fixed effect for whether or not facility ${\displaystyle q}$ was randomised to the intervention in quarter ${\displaystyle t}$;

${\displaystyle \theta }$, the coefficient of interest, is the effect of being randomised to receive the intervention on the outcome;

${\displaystyle X_{iq}}$ are fixed effects adjusting for demographic characteristics of patient ${\displaystyle i}$ in cluster ${\displaystyle q}$, that is, age, gender, ethnicity and rurality; and

${\displaystyle e_{iqt}}$ is the error term for each dermatology encounter.

Depending on the distribution of the outcome variable, the function F(.) is either the identity function (for continuous, normally distributed outcomes like travel distance) or exponential function (for highly skewed and always positive outcomes like completion times for appointments and consults and time to follow-up).

This type of model, proposed for the analysis of SW-CRTs by Hussey and Hughes,⁴⁰ clearly involves several important underlying assumptions, such as a common underlying piecewise secular trend across all clusters, a constant change in this common trend as a result of the intervention and an identical correlation between two observations in a cluster irrespective of treatment and time duration between the observations.²⁴ In secondary analyses, we will relax these assumptions in order to assess whether they impact the results. For example, we will allow the secular trends to vary by strata of clusters, such as VHA administrative regions or VHA VISN (using a fixed-effect interaction between time and stratum), or even by clusters (by adding a random interaction between time and cluster, and thus allowing intracluster correlation to vary by time period). Similarly, we will test models allowing for treatment effect heterogeneity across strata of clusters or across time (using either fixed or random effects), with the important caveat that some of these models will be estimable only on data collected in time periods in which there are both treated and control clusters.²⁴

In addition to the intent-to-treat analysis, we will also assess the impact of the intervention using an instrumental variable-based two-stage residual inclusion procedure. In this approach, we will estimate two parameters of interest. First, we are concerned with the effect of the randomisation on uptake of the apps, as a factor leading to teledermatology adoption in the sites receiving the intervention. This effect can be obtained by estimating the following first-stage model:

$${\displaystyle Ap{p}_{iqt}=\mathrm{L}\mathrm{o}\mathrm{g}\mathrm{i}{\mathrm{t}}^{-1}\left(\mu +{\alpha }_q+{\beta }_t+\theta V_{qt}+X_{iq}\right)+e_{iqt}}$$ (1)

Second, we are interested in the average effect of the treatment among compliers (patients who only receive the treatment as a direct result of their exposure to the intervention), referred to as the local average treatment effect. This effect better reflects the efficacy of teledermatology compared with regular practice and can be estimated using the following second-stage model:

$${\displaystyle {\mathrm{Y}}_{iqt}=F\left(\nu +a_q+{\delta }_t+Ap{p}_{iqt}+X_{iq}+{\hat{e}}_{iqt}\right)+{ϵ}_{iqt}}$$ (2)

where ${\displaystyle {\hat{e}}_{iqt}}$ is the predicted residual from estimating equation (1). Estimating this effect would allow future work to investigate why the intervention works and why uptake of the intervention varies across facilities. Pointing out that, for example, access can be improved significantly provided that the leadership of a healthcare system can ensure uptake of the app, would be important for future policy decisions.

Continuous monitoring of implementation

Implementation will also be assessed at all participating sites by monitoring intermediate milestones and quantitative indicators of implementation that are available in CDW as well as from OCC’s own telehealth database and Web and Mobile Solutions (WMS) mobile device procurement programme. Randomised sites, in addition to the three sites in the formative evaluation, will be asked to complete a bimonthly implementation site report monitoring key milestones, collected electronically via the VHA intranet. Sites will be sent email reminders 2 weeks and 1 week prior to, and 1 week after the due date, with follow-up via phone call, if necessary. Collection of these data will be descriptively summarised every quarter (3 months) to understand how rapidly sites meet key milestones as a result of the OCC implementation process, to correlate the milestones to the number of patients serviced via the apps (ie, reach) and to allow for stratified analyses of main quantitative study results by degree of implementation based on reaching milestones to determine if the apps are more effective among sites that have reached more implementation milestones. The study will not have a separate data monitoring committee, due to the low risk of the intervention and its minimal interference with patient care. Since we do not anticipate any adverse effects to be reported, a data monitoring committee is not necessary.

Patient and public involvement

The VA Telederm mobile app is designed to be used by primary healthcare providers and by imaging staff respectively to order and to process teledermatology consults prior to being read by a dermatology reader. The app interchangeably substitutes for these steps in VHA’s existing teledermatology process using its electronic health record. While the app is intended to make teledermatology services available to more patients, patients are not the actual users of the app. For all of these reasons, it is anticipated that the patient experience itself will not be affected by the use of the VA Telederm app, and no patients or patient advocacy groups were consulted in the design of the app or this trial. The development of the research questions and outcome measures was informed by prior scientific literature (including work published by the authors) on the impact of teledermatology on access and on patient wait time measures for primary and specialty care services. For similar reasons, there are no plans to disseminate the results of this trial directly to patients or patient groups, and the burden of the intervention was not assessed by patients.

Ethics and dissemination

Research ethics approval

The research has been approved by the Institutional Review Board (IRB) at VHA Boston (IRB Project No 3069), which has designated the study as exempt since it involves collection and analysis of data in a way that subjects cannot be identified, either directly or through identifiers linked to the subjects.⁴⁵ Specifically, since the app will be implemented within the process of care in the VHA, the data in the cluster randomised trial will be deidentified and collected retrospectively in the administrative database. The research team will query the relevant database tables and extract the data necessary for the proposed analyses and will conduct the research in a secure environment following all required procedures for protection of privacy and confidentiality. For the purposes of the Ottawa statement, the research participants in this study are the patients receiving dermatology care at the eligible facilities, since they will be affected by the change in the healthcare delivery process.⁴⁶ However, the study interventions and data collection procedures pose no more than minimal risk.

The research components of the formative evaluation have been approved by the IRBs at the VHA Durham, San Francisco, and Providence facilities, respectively. In this study component, the research participants are the VHA employees involved in the implementation, from whom informed consent will be obtained before being interviewed or surveyed.

Protocol amendments

Any modifications to the protocol which may impact on the conduct of the study, including study objectives, design, patient population, sample sizes, study procedures or significant administrative aspects, will require a formal amendment to the protocol. All such amendments will be agreed on by the study investigators and approved by the IRB prior to implementation and notified to the VHA OCC and HSR&D. Administrative changes of the protocol, such as minor corrections that have no effect on the way the study is conducted, will be agreed on by the study investigators and documented in a memorandum. The IRB may be notified of these administrative changes at the discretion of the study investigators.

Dissemination policy

The project team comprised specialist clinicians, academic researchers and experts in implementation science. This provides the project with access to a wide range of channels for results dissemination to policymakers, researchers and system stakeholders. The results of the study will be published in academic peer-reviewed journals and presented at professional conferences. Additionally, VHA leadership will be briefed on the preliminary as well as final study findings in order to inform future VHA policy regarding teledermatology. Communications with VHA leadership will be facilitated by two members of the research team who also serve as the clinical leads for teledermatology in the OCC (DHO and MAW). No use of professional writers will be made. Only investigators involved in the study planning, design or analysis will be eligible for authorship of study communications.

Participant-level data will not be made available to the public due to privacy and confidentiality concerns, but statistical code may be shared by request from the study authors. Study investigators and approved study personnel will be the only individuals who can access the final trial data set.

Patient and public involvement

Patients were not involved in the development of the research question or the outcome measures.

Discussion

Study impact and importance

At the end of the study period, the trial will document the effectiveness of mobile SFT teledermatology in enhancing veterans’ access to dermatology services. Moreover, it will produce a comprehensive understanding of the factors that lead to successful mobile telehealth implementation and adoption. The results will be of significance to the VHA as it develops and implements other mobile telehealth programmes, and more generally to other healthcare organisations planning for large-scale telehealth interventions.

In particular, the study will allow us to assess whether web-based mobile teledermatology apps improve access to expert dermatology services by decreasing consult times, reducing appointment completion times for new patients, increasing instances of dermatologic care and reducing the distance travelled by patients to receive dermatology services.

Strengths

This study has several important strengths. First, the stepped-wedge design of the trial will allow us to assess with increased confidence the causal impact of the new teledermatology intervention, by assigning the order of the intervention in a randomised fashion. Stepped-wedge designs allow clusters to be compared with other sites and to also serve partially as their own control, thus permitting us to account for outcome time trends for each participating facility. Second, the constrained randomisation scheme will ensure that imbalance in measured facility characteristics due to sheer chance will not bias the findings. This bias is an important concern in trials in which randomisation is performed for a small number of clusters. Third, in contrast with previous studies which were mainly conducted on small and relatively homogeneous samples, our study includes individuals accessing care in facilities throughout the USA. This includes rural and urban facilities serving patients characterised by geographic and socioeconomic diversity.

Fourth, the close partnership with clinical and operations leaders will ensure that all eligible sites receive the intervention and that clinician buy-in, which is crucial for the success of the intervention, is maximised.

Finally, the pilot testing and formative evaluation will ensure that implementation issues are addressed early by learning from the early test sites. In this way, any early issues with on-the-ground implementation can be mitigated.

Limitations

The study also has several limitations. First, implementation will likely vary by facility depending on the local culture, resources, efficacy and engagement of local leadership. Although we will conduct a formative evaluation in three facilities in order to inform the national roll-out of the intervention, it is likely that we cannot ensure uniform implementation across facilities. For example, some of the participating sites will use dermatoscopes that attach to imaging devices in order to collect high-quality photos. Since the cost of dermatoscopes is not covered by the OCC or the VHA Mobile Health Services office, there will likely be some variation in the quality of the images across sites.

Second, the impact of the app will depend on the effectiveness of the education and support provided to providers and to the extent to which the providers find the new process more intuitive and easy to use. Although we have adapted previously used training materials developed by OCC, there is still potential for inconsistent dissemination and support for the app.

Finally, the implementation requires the cooperation of multiple stakeholders at the national, regional and local levels in order to ensure proper training, education and support for providers in their adoption process. It is inevitable that some confusion or improper deployment will occur at least initially, which may affect the implementation process. Moreover, although we have allowed for 3 months for implementation in each facility, the possibility for longer delays still exists.

Generalisability

The study findings may not be generalisable outside the VHA, which has a different institutional structure from most other practices in the USA. The findings may also not be generalisable to healthcare systems outside the USA or other teledermatology mobile apps.