Conceptualizing and measuring integration of telehealth and in-person services from the provider’s perspective: development of the integration of telehealth and in-person services (ITIPS) survey

Abstract

Objective:

Hybrid in-person and telehealth work environments are now common among health care providers. When in-person and telehealth services are not well integrated, provider workload could increase, negatively affecting provider satisfaction and burnout and hindering implementation of interventions aimed at improving quality. A lack of measures of telehealth integration has hindered studies of such impacts. This article presents the Integration of Telehealth and In-Person Services (ITIPS) survey, developed to assess telehealth integration and its facilitators from the provider’s perspective. The ITIPS survey represents a meaningful step toward measuring integration in hybrid healthcare environments, including indicators of the extent of telehealth integration and factors promoting telehealth integration.

Materials and Methods:

Using an exploratory sequential mixed-methods design, this study included semi-structured interviews and a participant-driven, modified-Delphi survey development approach consisting of 2 rounds of Qualtrics surveys to obtain participant feedback. Some participants engaged in all phases, whereas others participated either in interviews or in the modified Delphi surveys.

Results:

Interviews identified multiple indicators of telehealth integration related to decisions about the visit modality, provider and staff workflows, as well as influencing factors such as leadership priorities related to quality and access, physical space, scheduling systems, and staff support. Our study yielded a survey with 22 items measuring the extent of telehealth integration in a practice environment and 31 items assessing factors influencing the level of telehealth integration.

Conclusion:

This study presents the ITIPS survey, which has undergone assessments for content validity and is ready for psychometric assessment for additional types of validity.

Background

Advancements in technology, broadband access, and value-based care have facilitated the growth of telehealth.¹ When appropriate for both patient and provider, telehealth can enhance access to care and complement traditional modes of healthcare delivery.² However, the disruption of the COVID-19 public health emergency led to rapid, widespread telehealth adoption,³ often without systematic planning. Currently, hybrid in-person and telehealth work environments are common for many health care providers; however, the extent to which in-person and telehealth encounters are integrated effectively is largely unknown.⁴

We conceptualize telehealth integration as the extent to which traditional, in-person services and telehealth services are structured and coordinated to promote quality of care for patients and efficient workflows for providers. We posit that telehealth integration should be a key consideration when implementing telehealth services. In other words, telehealth implementation processes should account for the interaction between in-person and telehealth modalities from the perspective of providers working within a hybrid environment. Effective integration of telehealth with in-person care could reduce fragmentation, improve care coordination, promote value-based care, and minimize duplicative services.² For example, Tang et al. recently examined the effect of providers switching from telehealth to in-person and found that telemedicine visits occurring immediately following an in-person visit had higher no-show rates and lower patient-reported ratings of their providers.⁵ These findings may suggest a lack of effective telehealth integration within a provider’s workflow. However, overall the extent and effects of telehealth integration remain under studied—likely due, at least in part, to a lack of provider-focused measures of telehealth integration.⁶ Health systems, payers, and policymakers need such research to understand impacts of telehealth integration and to inform strategic efforts.

Integration of care has long been a focus of inquiry. For example, Singer et al. developed the Comprehensive Theory of Integration, which identifies types of integration (structural, functional, normative, interpersonal, and process).⁷ Other models, such as Primary Care Behavioral Health and the Collaborative Care Model, guide integration of behavioral health into primary care.⁸ Furthermore, several efforts have created patient-centered care integration measures. For example, Friedberg et al. developed the Patient Perceptions of Integrated Care survey to assess aspects such as coordination, continuity, and patient-centeredness of care.⁹ Similarly, the Pediatric Integrated Care Survey measures caregiver experiences across dimensions such as access, communication, family impact, and team functioning.¹⁰ These complement other measures like the Coordination of Care Survey¹¹ and the Child CAHPS survey¹² but none focus on telehealth integration.

Within the implementation science literature, Glasgow et al. developed a framework for promoting rapid research of telehealth interventions.¹³ Their framework calls for the more intentional contextual assessment of factors that may affect implementation of these interventions. Measuring the concept of telehealth integration is important for understanding organizational context in a hybrid environment. Furthermore, as telehealth utilization increases for some services, providers and patients may become more comfortable with using telehealth in general, perhaps contributing to more types of services being offered via telehealth, even if they originally were conceived as being delivered in-person. Effective telehealth integration could be a key indicator of the feasibility and perhaps sustainability of delivering a new service via telehealth. However, there is no agreed-upon, standardized tool for assessing the extent of integration of telehealth and in-person services (ITIPS) from a provider perspective.

This article describes an innovative approach to the development and content validation of a pragmatic survey instrument that quantifies, from the provider’s perspective, how well telehealth services are integrated with in-person services—the ITIPS survey. Specifically, we engaged providers and health system administrators to (1) identify factors promoting telehealth integration; (2) identify direct indicators of telehealth integration; and (3) assess face validity of survey items designed to measure these factors and indicators. To build evidence of validity and reliability of ITIPS, future work should include cognitive testing and psychometric analyses.

Methods

Our research team included 3 faculty members (C.S., S.K., and D.M.) and 1 doctoral student (S.T.). All have experience with telehealth. Two members (C.S. and S.T.) have extensive experience in qualitative methods. C.S. also has experience in survey development.

This study employed a participatory survey-development process using an exploratory sequential mixed-methods design.¹⁴ More specifically, we conducted semi-structured interviews to identify telehealth-integration concepts to measure and 2 rounds of surveys using Qualtrics Software (Qualtrics International) to assess survey items we developed to measure the concepts (see Figure 1). Some participants were included in all phases of the data collection; others participated either in interviews or in the modified Delphi surveys. Participation in each phase was determined by our sampling strategy (eg, desired professional roles of participants) and participant availability. The study was reviewed and considered exempt by the UNC-Chapel Hill IRB (# 23–2589).

Figure 1.

Development process flow diagram.

Semi-structured interviews

Recruitment and sample

The initial phase involved conducting semi-structured interviews to identify dimensions of telehealth integration and factors that promote or hinder effective telehealth integration. Potential participants received information about study goals via email and were recruited primarily from the Supporting Pediatric Research on Outcomes and Utilization of Telehealth (SPROUT) network, which includes over 120 institutions.³ We also used targeted recruiting within the Doxy.me Research network contacted individuals known to the research team to have relevant roles and experience. We focused on telehealth in pediatrics and maternal health, due to the focus of the SPROUT network and to narrow the clinical scenarios that participants would be considering when assessing the survey items.

Our final sample included 14 participants. Participants included 12 pediatric and maternal health providers, 1 non-clinical telehealth administrator, and 1 non-clinical telehealth researcher. Some providers also held administrative roles and/or were clinician-researchers. In addition to specialty, criteria used when sampling providers included experience working in a hybrid (in-person and telehealth) environment, geographic location within the U.S., and whether providers cared for rural patients, urban patients, or both. An advantage of drawing upon the SPROUT network for recruitment was its national coverage and its inclusion of providers with substantial telehealth experience. The participants in our sample were from organizations located in 11 different states, representing the South, West, Northeast, and Midwest Census regions. Nine of the 14 represented provider organizations affiliated with academic medical centers. The majority of the practice sites were located in a metro area, but many cared for patients residing in both metro and nonmetro areas.

Data collection

We developed a semi-structured interview guide, informed by relevant literature, including the SPROUT Telehealth Evaluation and Measurement Framework and the Comprehensive Theory of integration by Singer et al.^3,7 (see Appendix for Interview Guide). C.S. and S.T. conducted the interviews via Zoom with audio recording, each lasting approximately 45 minutes. Interviews began with participants describing what effective telehealth integration means to them, what they observe (or would observe) when it occurs, and a scenario illustrating effective integration. If participants did not discuss how modality decisions should be made, ideal workflows, and accessibility of patient information, then we asked directly about these topics. To round out the discussion, we asked questions broadly informed by the types of integration identified by Singer et al.⁷ Finally, we asked selected participants to discuss the financial costs associated with factors promoting telehealth integration.

Because our interviews were semi-structured, we did not ask each participant to answer every question in our interview guide; instead, we allowed participants to identify the indicators/factors they perceived to be most salient. We expected that direct indicators of telehealth integration and factors affecting telehealth integration could span the categories of integration identified by Singer et al.⁶ Furthermore, at times, participants naturally differentiated indicators from factors; however, we did not ask participants to categorize each issue they identified, due to time constraints and the cognitive demand that exercise would require. Clarifying these distinctions occurred in our analysis.

Data analysis

Interviews were recorded and professionally transcribed. C.S. and S.T. employed a rapid-analysis approach, coding transcripts using Microsoft Excel, to map transcript excerpts to topics (ie, codes). Although a rapid approach has some potential tradeoffs (eg, less in-depth interpretive analyses), we found it appropriate for various reasons, including the need for timely results to inform development of the survey within the project period, a clearly defined purpose of categorizing indicators of telehealth integration and factors affecting that integration, and a small team with substantial qualitative experience.¹⁵ We used memos to assist with specifying the topics and to inform development of topic summaries, which we generated. Based on the summaries, we organized topics into 2 categories—indicators of telehealth integration and factors affecting integration (ie, environment for integration)—and identified subtopics of indicators/factors within each category. In summary, our approach involved identification of concepts and activities, with a focus on comprehensiveness within the sample, and included both description (eg, to identify a workflow within a well-integrated environment) and categorization (eg, to categorize factors affecting integration).

Modified Delphi surveys

Recruitment and sample

Our sample for the modified Delphi surveys included 7 of the interview participants and 8 new participants, for a total of 15 individuals. Three participants in our interview sample were not invited to participate in the modified Delphi phase; these participants provided useful insights during the interviews regarding telehealth integration from an operational and/or cost perspective, but they did not perform a clinical role using telehealth or have experience needed for developing survey items. Four others who were invited to participate in the Delphi phase were unavailable. New participants we identified using an approach similar to the interview phase of the project. Of the 15, 14 were clinicians and 1 was a maternal and child health researcher. The participants were from organizations located in 9 different states (including the District of Columbia), representing the South, West, Northeast, and Midwest Census regions. All represented provider organizations affiliated with academic medical centers, which enabled most participants to speak about caring for both rural and urban patients.

Data collection

Based on our interview findings, we developed survey items in Qualtrics for the first round of the Delphi survey. The 15 participants were asked to prioritize dimensions of factors that affect telehealth integration (ie, environment for integration) and direct indicators of telehealth integration. They also were asked to assess the relevance (using a Likert scale) and clarity (using open-ended comments) of draft measures for each factor and indicator. Factors affecting integration included reimbursement for services delivered via telehealth, leadership prioritization of telehealth integration, quality improvement efforts that include improving telehealth, adequate staffing to support a hybrid in-person/telehealth model, telehealth training for providers and staff, and providers having a convenient space to conduct telehealth visits. Indicators of telehealth integration were organized into phases: modality decision (ie, in-person or telehealth) at the time of scheduling, starting a telehealth visit, experience during a telehealth visit, and concluding a telehealth visit.

The second round of the modified Delphi survey involved participants re-evaluating the revised measures from the first round. For this round, we received responses from 13 of the 15 participants in our sample. Participants were provided a draft of the survey as it would appear to an actual survey respondent, with survey items organized into sections for each factor affecting integration (eg, reimbursement) and each indicator of integration. For each section of the survey, participants were asked to review the survey items and provide open-ended feedback about (1) the usefulness of the survey items for understanding the intended concept and (2) suggestions for improving the wording of the question stems or response options.

Data analysis

In round 1, the feedback obtained was analyzed to (1) assess prioritization and specification of the subcategories of factors affecting integration and indicators of integration and (2) refine survey items measuring each factor and indicator. More specifically, Likert scores assessing the importance of factors and indicators were analyzed descriptively to identify which were perceived by participant to be higher priorities for measurement. In our results, we report which factors were rated as most important, as determined by counts of “extremely important” and “moderately important” categories. Although we believe these ratings of importance were useful for the development process, we did not exclude entire factors that were rated as less important because the observed effect of factors on integration is a question to be addressed in future research using data collected with ITIPS.

To inform decisions about removing and/or revising individual survey items, we analyzed both Likert scores assessing relevance of the survey items to their respective factor/indicator and open-ended comments about the items. Revised items were then assessed in round 2. For example, 2 items received Likert scores of “Not at all relevant” from 4 respondents and were not included in the next round for consideration, whereas new survey items were added for the next round based on suggestions entered in the open-ended comments.

In round 2, analysis was strictly qualitative, with open-ended comments reviewed to identify which individual survey items were deemed clear and appropriate and which required further refinement as well as any potential dimensions within the factors/indicators that were not adequately measured. This process yielded the final survey presented in this paper.

Results

Semi-structured interviews

Our analysis revealed insights into how participants discussed telehealth integration and factors promoting it. Although our goal was to identify telehealth integration from a provider’s perspective, some participants emphasized patient experience, suggesting beliefs that patient experiences are at least as important as provider experiences. Related to patient experiences, there was some variability in the extent to which participants highlighted the role of telehealth in population health, such as differences in health care access or disparities in health outcomes. These discussions informed our understanding of factors affecting telehealth integration, specifically population-level quality improvement efforts such as patient-centered care and organizational factors such as leadership priorities that highlight telehealth and in-person care being well integrated and complementary rather than siloed. Participants discussed several other factors as enablers of effective telehealth integration, including physical space that allows appropriate privacy and convenience for telehealth visits, telehealth technology that minimizes glitches, and staffing that minimizes provider burden when switching between in-person and telehealth visits.

Discussions about indicators of integration were categorized as either decisions about whether a visit would be delivered via telehealth, provider and staff workflows related to a telehealth visit, or challenges specific to telehealth visits (eg, problems encountered). With respect to workflows, many participants compared activities for a telehealth visit to an in-person visit regarding efficiency of the task or challenges faced with activities in a hybrid environment. Our initial approach was to organize these issues by phases of a visit: (1) scheduling, (2) initiating a telehealth visit, (3) interactions with the patient during the visit, and (4) concluding the visit (see Figure 2).

Figure 2.

Indicators of telehealth integration.

Some participants reported costs of offering telehealth at all, as opposed to costs of integration. This may be due to many practice sites having not conceptualized or measured costs specifically attributable to integration. Furthermore, there was acknowledgment that providing both in-person and telehealth services carries higher costs than offering either one alone. Specific to integration costs and consistent with the first phase of the visit described above, participants identified costs related to time spent preparing for and making this decision about visit modality, including training staff involved with the decision and providing the patient with information about the decision. Costs associated with furnishing telehealth-capable rooms conveniently located to in-person patient rooms was an infrastructure-related integration cost. Relatedly, provider time transitioning between modalities was identified, for example, walking to the room for telehealth visits (particularly if it is not conveniently located to the in-person visit rooms), logging into the telehealth system, and any time spent waiting for the patient that would not occur in an in-person visit, which one participant suggested could be measured with time and motion studies. Also identified were staff time and expertise related to ensuring integration of technology (eg, telehealth and EHR platforms) as well as staff time dedicated to supporting patients with connecting to telehealth visits (eg, troubleshooting technical problems). Arguably, these could be viewed as costs for delivering any telehealth rather than as integration costs; however, this staff time mitigates waiting time or other inefficiencies for providers working in a hybrid environment. Similarly, some sites may require more sophisticated video conferencing technology than others, for example, larger health systems handling complex cases in which multiple providers in different locations are on the same call with a patient. In other words, providing well integrated in-person and telehealth services may require additional staff activities and potentially more expensive technology or systems integration than providing telehealth alone.

Delphi round 1

Likert scale prioritization of factors/indicators and open-ended comments

The first round of the Delphi survey revealed varied responses and recommendations across multiple domains. With respect to the environment for telehealth integration, domains assessed included reimbursement, leadership priorities, quality improvement, supportive staffing, training, and physical space for telehealth. Reimbursement and leadership priorities were regarded as the 2 most important domains, with staffing also regarded as extremely or moderately important by all but one respondent. Quality improvement, training, and physical space were rated as less important, with at least 2 respondents rating the domain as slightly or not at all important (see Supplementary Table). Key recommendations from the open-ended comments in this round of feedback were the need to add greater specificity and coverage of reimbursement issues; clarify levels of leadership (ie, system vs practice level); include items to assess technology more explicitly; specify quality measurement initiatives more explicitly; and reduce duplication in the indicators of integration items.

Delphi round 2

Based on feedback from the first round, survey items measuring indicators of telehealth integration were reorganized to reduce duplication. Specifically, they were not organized into the phases—modality decision at the time of scheduling, starting a telehealth visit, experience during a telehealth visit, and concluding a telehealth visit. Instead, to streamline the survey and reduce survey burden, the items were organized by the type of response option, and the number of indicator items was reduced from 31 to 22 (see Table 1 for revised telehealth integration indicator items).

Table 1.

Revised survey items for integration indicators based on round 2 feedback

INDICATORS OF TELEHEALTH INTEGRATION (part 1)
Response Options:
Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree, Don’t Know/Unsure
1	In our clinic, providers are able to document guidance about whether a patient’s follow-up visit can be scheduled for telehealth.
2	In our clinic, there are standardized criteria for providers to determine whether a visit can be scheduled for telehealth.
3	When scheduling a visit, patients are involved in decisions about whether their visit will be in person or telehealth.
4	In our clinic, there are standardized criteria for schedulers to use when determining if a telehealth visit is appropriate.
5	When scheduling a visit through the patient portal, patients have the option to select telehealth.
6	The patient portal provides guidance to patients about determining whether telehealth is appropriate for a particular visit.
7	Prior to a visit, our providers can change the modality from telehealth to in-person and keep the original visit date/time.
8	Prior to a visit, our providers are able to change the modality from telehealth to in-person without causing a significant delay in care for the patient.
9	Our clinic’s scheduling system is well designed for providers who complete a combination of in-person and telehealth visits.
10	For providers in our clinic, patient information is equally accessible for in-person and telehealth visits.
11	For providers in our clinic, the documentation process is equally efficient for in-person and telehealth visits.
INDICATORS OF TELEHEALTH INTEGRATION (part 2)
Response Options (unless otherwise indicated):
Never, Rarely, Sometimes, Often, Always, Don’t Know/Unsure
1	In our clinic, providers log on to the telehealth system and have to wait for the patient to join the visit.
2	In our clinic, patients are well prepared for their telehealth visit.
3	In our clinic, providers have to troubleshoot technical problems at the beginning of telehealth visits.
4	In our clinic, providers have access to the patient information they need prior to beginning a telehealth visit.
5	In our clinic, we conduct patient visits via telehealth that would have been completed better in person.
6	In our clinic, we conduct in-person visits that could have been completed equally as well via telehealth.
7	In our clinic, providers are able to meet their clinical goals during telehealth visits.
8	In our clinic, providers are able to meet patient needs during telehealth visits.
9	In our clinic, patient care is delayed because patients must wait for an in-person visit when their needs could have been resolved sooner with a telehealth visit.
10	For telehealth visits in our clinic, the level of care coordination support provided by support staff is: Less than for in-person services About the same as in-person services More than for in-person services Don’t know/unsure
11	Which is true in your clinic? The amount of administrative/logistical responsibilities for providers is similar for in-person and telehealth visits Providers perform more administrative/logistical responsibilities when completing a telehealth visit Providers perform more administrative/logistical responsibilities when performing an in-person visits Don’t Know/Unsure

Regarding the “environment for integration” section, the number of factors affecting telehealth integration was reduced from 6 to 5, with training and physical space being combined into one section called space, technology, and training. Furthermore, the leadership priorities factor was divided into 2 subcategories (health system leadership and practice-level leadership). Overall, there were several comments about needing additional items to capture the environmental constructs, leading to an increase in items from 17 to 31 (see Table 2 for revised environmental factors items).

Table 2.

Revised survey items for environmental factors based on round 2 feedback.

RESPONSE OPTIONS (unless otherwise indicated) Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree, Don’t Know/Unsure
ENVIRONMENTAL FACTORS
Reimbursement
1	Reimbursement levels from our most common payers are sufficient to support remote patient monitoring.
2	Reimbursement levels from our most common payers are sufficient to support audio-only calls with patients.
3	Reimbursement levels from our most common payers are sufficient to support emails with patients.
4	Reimbursement levels from our most common payers are sufficient to support telehealth for substance-use services.
5	Reimbursement levels from our most common payers are sufficient to support telehealth for mental health services.
6	Within my practice environment, for telehealth services that are billable, reimbursement for them is equal to comparable in-person services.
7	Uncertainty about future reimbursement for telehealth services hinders my practice’s ability to integrate telehealth and in-person services.
8	Uncertainty about whether telehealth services will be reimbursed for a particular patient hinders my practice’s ability to integrate telehealth and in-person services.
9	Variability in reimbursement for telehealth services across payers hinders my practice’s ability to integrate telehealth and in-person services.
Leadership Priorities
1	Leaders in our health system commit sufficient resources to telehealth.
2	Leaders in our health system commit sufficient resources to integration of telehealth with in-person services.
3	Leaders in our health system commit sufficient resources to addressing health disparities.
4	Leaders in our health system commit sufficient resources to delivering health care services that are responsive to individual patient preferences.
5	People in leadership positions within my practice environment commit sufficient resources to telehealth.
6	People in leadership positions within my practice environment commit sufficient resources to effective integration of telehealth with in-person services.
7	People in leadership positions within my practice environment commit sufficient resources to addressing health disparities.
8	People in leadership positions within my practice environment commit sufficient resources to delivering health care services that are responsive to individual patient preferences.
Quality Improvement Efforts
1	We have a performance indicator to measure the time between a patient making an appointment and completing the visit.
2	We have a performance indicator to measure the time patients spend waiting past the appointment time.
3	We have performance indicators to measure adherence to evidence-based/consensus-based clinical practices.
4	We have performance indicators to measure patient safety.
5	We have performance indicators to measure patient health outcomes.
6	We have performance indicators to measure patient satisfaction with telehealth visits.
7	We have performance indicators to measure provider satisfaction with telehealth visits.
8	We have performance indicators to measure health disparities.
9	We have performance indicators to measure the number of patients seen during a specific time period.
10	In our clinic, quality improvement (QI) methods are used to improve telehealth services in the following areas (Check all that apply): Timeliness of care Adherence to evidence-based or consensus-based clinical practices Patient safety Patient health outcomes Patient satisfaction Health disparities Clinical team workflow efficiency Provider satisfaction Other _______ None of the Above
Supportive Staffing for a Hybrid Model
1	In our clinic, providers have the option to work outside of the clinic when completing telehealth visits.
2	In our clinic, support staff develop plans in collaboration with providers to support care delivery in a hybrid model.
3	In our clinic, support staff have protocols for supporting providers working in a hybrid model.
4	In our clinic, support staff prepare the computer, camera, and other needed technology for a telehealth visit prior to the provider logging in for the visit.
5	In our clinic, support staff are able to troubleshoot problems with the computer, camera, and other needed technology for a telehealth visit prior to the provider logging in for the visit.
6	For in-person visits in our clinic, support staff update the medication list with a patient at the beginning of the visit.
7	When the provider is located in our clinic, support staff update the medication list with a patient at the beginning of a telehealth visit.
8	When the provider is located outside of our clinic, support staff update the medication list with a patient at the beginning of a telehealth visit.
9	For in-person visits in our clinic, support staff schedule follow-up visits with patients at the end of the visit.
10	When the provider is located in our clinic, support staff schedule follow-up visits with the patient at the end of a telehealth visit.
11	When the provider is located outside of our clinic, support staff schedule follow-up visits with a patient
12	Overall, in our clinic, support staff effectively support the work of providers who see patients in a hybrid model.
Space, Technology, and Training
1	When at the clinic, providers have access to physical space that meets their needs to conduct telehealth visits.
2	In our clinic, the telehealth platform can be accessed through the electronic health record.
3	In our clinic, the electronic health record and telehealth platform work well together.
4	In our clinic, the electronic health record supports in-person visits and telehealth visits equally well.
5	In our clinic, providers receive adequate training on communicating with patients during telehealth visits.
6	In our clinic, providers receive adequate training on the technology used to conduct telehealth visits.

Unlike in round 1, most of the comments in this round were suggestions for minor edits to wording to existing items, which were incorporated into the final version. Some comments regarding environmental factors led to more substantial revisions, compared to those about integration indicators. Many comments simply affirmed the clarity and/or importance of the survey items. In the environmental factors section, there were 4 notable comments focusing on reimbursement items. One related to variability in reimbursement across payers and uncertainty about reimbursement for telehealth services for a particular patient. We agreed that these are important aspects of uncertainty not captured in the existing items. We therefore added 2 items to capture this and removed one that assessed the difficulty of predicting future reimbursement. The other 3 comments related to the issue that some telehealth activities are not reimbursed at all, which makes answering about “sufficiency” of reimbursement difficult. We decided there is a need for providing additional guidance to respondents that “sufficiency” refers also to instances in which no reimbursement is received; that is, 0 is a reimbursement amount.

Discussion

Providers in well-integrated hybrid environments may be better positioned to implement innovations within their practice, benefiting from more efficient provider workflows and patient-centered care modality decisions. While the influence of integration may be most important for telehealth-delivered interventions, it could also extend to initiatives seemingly unrelated to telehealth by shaping providers’ overall workload. Currently, such relationships are open questions for future studies. Furthermore, psychometric assessment of the ITIPS survey is needed to assess its ability to support such future research.

We view telehealth integration as aligned with the structural characteristics construct within the inner setting domain of the Consolidated Framework for Implementation Research (CFIR), spanning physical, information technology, and work infrastructure subconstructs.¹⁶ In a hybrid care environment, telehealth integration represents an interrelationship between these elements. As such, it may function as “a higher-order construct,” like implementation climate and organizational readiness for change, encompassing multiple CFIR dimensions.¹⁶ The current iteration of the ITIPS survey fills a gap in implementation science by offering a pathway to measure this construct. As mentioned above, psychometric assessment of ITIPS is needed to identify whether additional modifications are needed.

Telehealth integration also has implications for tailoring implementation strategies.¹⁷ This includes both capacity-building strategies, which aim to improve general implementation readiness, and innovation-specific strategies, which address barriers to a particular intervention.¹⁸ For example, sites with low levels of telehealth integration may benefit from capacity building strategies, such as telehealth-related training or technical assistance with assessing and improving workflows. Sites implementing specific innovations may require EHR modifications to support hybrid documentation and coordination.

Finally, our interviews revealed insights into the types of costs related to integrating in-person and telehealth services. These insights are particularly useful for implementation of clinical interventions that may be delivered both in-person and via telehealth or that require multiple providers working in separate hybrid environments (ie, located in different sites) to convene via telehealth simultaneously with the same patient.

As mentioned above, this version of the ITIPS was developed using a literature-informed, data driven process. Although our approach was systematic, we acknowledge there are limitations to the current version of the ITIPS. Our current efforts establish content validity and buy-in for our measures using a modified-Delphi process with a panel of experts. Perhaps the most notable limitation is the need to establish reliability, validity, and the relationships among the survey items. A next step in future research could involve pre and pilot testing and psychometric analyses to examine the structure of ITIPS and its ability to measure telehealth-integration concepts. Furthermore, analyses of discriminant validity could be useful, particularly for constructs within the environment for integration.

Conclusion

This paper presents results from an exploratory sequential mixed-methods study to develop a survey (ITIPS) for providers about the extent of integration between in-person and telehealth services in their practice and factors that promote such integration. ITIPS could support future research on effects of telehealth integration on healthcare delivery and implementation of new interventions.

Supplementary Material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.