Telemedicine Implementation for Safety Net Populations: A Systematic Review

Abstract

Background:

Telemedicine systems were rapidly implemented in response to COVID-19. However, little is known about their effectiveness, acceptability, and sustainability for safety net populations. This study systematically reviewed primary care telemedicine implementation and effectiveness in safety net settings.

Methods:

We searched PubMed for peer-reviewed articles on telemedicine implementation from 2013 to 2021. The search was done between June and December 2021. Included articles focused on health care organizations that primarily serve low-income and/or rural populations in the United States. We screened 244 articles from an initial search of 343 articles and extracted and analyzed data from N = 45 articles.

Results:

Nine (20%) of 45 articles were randomized controlled trials. N = 22 reported findings for at least one marginalized group (i.e., racial/ethnic minority, 65 years+, limited English proficiency). Only n = 19 (42%) included African American/Black patients in demographics descriptions, n = 14 (31%) LatinX/Hispanic patients, n = 4 (9%) Asian patients, n = 4 (9%) patients aged 65+ years, and n = 4 (9%) patients with limited English proficiency. Results show telemedicine can provide high-quality primary care that is more accessible and affordable. Fifteen studies assessed barriers and facilitators to telemedicine implementation. Common barriers were billing/administrative workflow disruption (n = 9, 20%), broadband access/quality (n = 5, 11%), and patient preference for in-person care (n = 4, 9%). Facilitators included efficiency gains (n = 6, 13%), patient acceptance (n = 3, 7%), and enhanced access (n = 3, 7%).

Conclusions:

Telemedicine is an acceptable care modality to deliver primary care in safety net settings. Future studies should compare telemedicine and in-person care quality and test strategies to improve telemedicine implementation in safety net settings.

Introduction

In response to the COVID-19 pandemic, health care delivery systems rapidly implemented telemedicine systems to facilitate remote care for the safety of their patient populations.^1–4 It resulted in a multifold increase in telemedicine adoption for patient care across diagnoses and demographic groups.^5–9 Evidence in published peer-reviewed literature indicates that telemedicine is an acceptable and welcomed modality for care, as patients have reported high levels of satisfaction,^5,9 and health care settings and providers have expressed interest in maintaining telemedicine as a care modality postpandemic.^10–12

Ideally, health care organizations would first plan for the implementation of an innovation, including assessing the resources required, addressing user needs, and mitigating obstacles, to promote sustained and effective implementation.^13–15 However, the crisis of COVID-19 pandemic spurred rapid adoption without an intentional planning phase, which may have resulted in less-effective implementation or potential negative impacts on patients, particularly in Federally Qualified Health Centers (FQHCs) and other safety net settings that disproportionately serve historically marginalized and minoritized populations.

Prepandemic only 43% of FQHCs used telehealth to provide care,¹⁶ which increased to 98% with the onset of the COVID-19 pandemic.¹⁶ Widespread telemedicine adoption has the risk of exacerbating existing health care inequities if it is not effectively implemented or does not reach underserved populations and communities.^4,17–20 In addition, while safety net settings are key health care settings to ensure equitable access to quality health care, their patient population might be more likely to be at risk of being left behind by the “digital divide”—a division between people who have access to and use digital media and those who do not.²¹

To address these gaps, we compiled evidence about primary care telemedicine implementation and effectiveness in safety net settings. We conducted a systematic review on telemedicine implementation for patients in FQHCs, rural health centers (RHCs), community health centers (CHCs), and academic medical centers. Our objective was to assess evidence about barriers and facilitators at the organizational level for synchronous phone or video visits to provide primary care services. The purpose of this work was to provide insights to CHCs about previous work examining telemedicine implementation to provide quality and effective care for their patient population.

Methods

We systematically searched PubMed and reviewed quantitative and qualitative peer-reviewed articles published from January 1, 2013 to December 31, 2021. The initial date restriction was set since pre-2013 articles were less likely to be relevant to modern telehealth implementation due to advances in technology and policy. Inclusion criteria consisted of peer-reviewed original research articles using quantitative and/or qualitative methods: (1) describing and analyzing telemedicine implementation where telemedicine was defined as either synchronous phone or video provider–patient appointments, and (2) focused on health care settings that predominantly serve low income and/or rural populations in the United States, including FQHCs, RHCs, CHCs, academic medical centers, and safety net hospitals. Articles that examined behavioral and mental health and pharmacy settings were also included. However, we excluded articles reporting exclusively on Veterans Affairs (VA) health organizations due to the potential limited generalizability of VA telemedicine study results to safety net settings focused on populations with low-income and/or limited English language proficiency.

We also excluded articles published in a language other than English and articles reporting on nonreimbursable appointment types defined by the Centers for Medicaid and Medicare Services (CMS), for example, peer-to-peer educator appointments, and dental appointments.

Data collection

The search was completed between June and December 2021 and yielded a total of n = 343 articles. Search terms used in the review are presented in Supplementary Appendix SA1. Duplicates were removed and remaining articles (n = 244) underwent title and abstract screening followed by full-text screening to identify potentially relevant articles that met inclusion criteria. To supplement the search, we also reviewed articles (n = 17) from a special issue focused on vulnerable populations published in Telemedicine and e-Health in August 2021.²² Articles were only included for final analysis based on independent review by two team members (n = 45). Figure 1 is a PRISMA flow diagram^23–26 detailing the review process.

Fig. 1.

PRISMA flow diagram for the selection of articles. From: Page et al.²⁵

Data extraction and analysis

Each article underwent extraction and quality assessment using relevant Critical Appraisal Skills Program (CASP) Checklists²⁷ by two reviewers. The extraction tool is presented in Supplementary Appendix SA2. Any disagreements between reviewers were resolved by a senior consensus reviewer. Abstracted data included the study design, telemedicine implementation barriers or facilitators, target population demographics (with a specific focus on a subset of further marginalized or at-risk populations, e.g., non-White, elderly, rural, specific comorbidities), study location, time period, intervention details, telemedicine modality (phone, video, or both), and results. If an article contained a study design or was guided by an implementation science theory, model, framework, or concept, this was also recorded during the extraction phase. All screening, extraction, and quality assessments were performed using Covidence software to streamline data collection and analyses.²⁸

Results

After the full screening process, n = 45 articles were identified and included in the final sample.^4,29–72 In terms of study design, n = 36 (80%) articles were observational in nature^{4,29–35,37–52,54,55,57,59,60,62,64,65,68–71} and n = 9 (20%) were randomized controlled trials.^{36,53,56,58,61,63,66,67,72} All articles met the CASP Checklist requirements for being methodologically sound and relevant according to the review's intended purpose.

Most articles focused on video (n = 35, 78%)^{4,29,31–35,37–46,49,50,52–54,56–58,60,62–67,69–71} or audio (n = 24, 53%)^{4,29,32–34,37–46,51,55–57,61–63,66,72} telemedicine modalities. Other modalities mentioned were remote monitoring (n = 6),^{31,48,49,56,57,60} virtual reality(n = 1)³⁶ and store-and-forward (n = 1).⁵⁴ Twenty-three (51%) articles analyzed data collected before the COVID-19 pandemic^{36,48,50,52–69,71,72} and 22 (49%) analyzed data collected during the pandemic.^{4,29–35,37–43,45–47,49,51,70,73} Study design and telemedicine modality details are included in Table 1.

Table 1.

Telemedicine Modalities, Implementation Description, Intervention Components, and Sustainability by Study Design (N = 45)

FIRST AUTHOR, YEAR (REF)	WHAT DID IMPLEMENTATION CONSIST OF?	DURATION OF THE INTERVENTION	WAS THE DESCRIPTION OF THE INTERVENTION CLEAR?	SUSTAINABILITY/FOLLOW-UP	FIDELITY/EVEN IMPLEMENTATION
Randomized controlled trials
Anderson, 201072	Audio	Temporary	Very clear	≥6 months	Different across groups
Coker, 201967	Video	Temporary	Very clear	≥6 months	Different across groups
Davis, 201066	Video, audio	Temporary	Somewhat clear	≥6 months	Different across groups
Fortney, 201363	Video, audio	Temporary	Very clear	≥6 months	Different across groups
Grubbs, 201861	Audio	Temporary	Very clear	≥6 months	Different across groups
Mittal, 201458	Video, other	Temporary	Very clear	≥6 months	Different across groups
Pyne, 201556	Video, audio, remote monitoring	Temporary	Very clear	≥6 months	Different across groups
Richter, 201553	Video	Temporary	Very clear	≥6 months	Different across groups
Rosal, 201436	Other	Temporary	Very clear	≥3 but <6 months	Different across groups
Observational studies
Adams, 202131	Video, remote monitoring	Routine	Very clear	Not clear	Different across groups
Armstrong, 201171	Video	Routine	Very clear	≥6 months	Different across groups
Barney, 202070	Video	Routine	Not clear	<3 months	Same across all groups
Caton, 202145	Video, audio, not clear	Routine	Somewhat clear	<3 months	Different across all groups
Chang, 20214	Video, audio, not clear	Routine	Not clear	<3 months	Different across groups
Childs, 202132	Video, audio	Routine	Very clear	Not clear	Same across all groups
Clifton, 200369	Video	Routine	Very clear	<3 months	Same across all groups
Coffman, 201668	Not clear	Routine	Not clear	Not clear	Unclear
Dayal, 2019 (Neurology)65	Video	Routine	Very clear	≥6 months	Different across groups
Dayal, 2019 (JAMA)64	Video	Routine	Very clear	≥6 months	Same across all groups
Dunham, 202129	Video, audio	Routine	Very clear	<3 months	Same across all groups
Franciosi, 202130	Not clear	Routine	Very clear	≥6 months	Same across all groups
Friesen, 201562	Video, audio	Temporary	Very clear	≥6 months	Same across all groups
Futterman, 202047	Not clear	Routine	Not clear	<3 months	Unclear
Hernandez, 201660	Video, remote monitoring	Routine	Very clear	≥6 months	Same across all groups
Howren, 202135	Video	Routine	Not clear	Not clear	Same across all groups
Khoong, 202141	Video, audio	Routine	Somewhat clear	<3 months	Different across all groups
Lin, 201859	Not clear	Routine	Somewhat clear	Not clear	Unclear
Mammen, 202057	Video, audio, remote monitoring	Temporary	Very clear	≥3 but <6 months	Same across all groups
Mills, 2021 (AHA)40	Video, audio	Routine	Not clear	<3 months	Different across groups
Mills, 2021 (Telemedicine)46	Video, audio	Routine	Somewhat clear	Not clear	Same across all groups
Nguyen, 202133	Video, audio	Routine	Somewhat clear	<3 months	Same across all groups
Nies, 202134	Video, audio	Routine	Somewhat clear	<3 months	Same across all groups
Parnell, 202055	Audio	Routine	Very clear	Not clear	Same across all groups
Patton, 202139	Video, audio	Routine	Very clear	<3 months	Different across groups
Phenicie, 202143	Video, audio	Routine	Not clear	≥3 but <6 months	Unclear
Shin, 201454	Other	Routine	Very clear	≥6 months	Different across groups
Simon, 202137	Video, audio	Routine	Not clear	<3 months	Unclear
Spinelli, 202051	Audio	Routine	Somewhat clear	<3 months	Different across groups
Tolou-Shams, 202138	Video, audio, training	Routine	Not clear	<3 months	Unclear
Uscher-Pines, 2020 (Psychiatric)52	Video	Routine	Somewhat clear	Not clear	Same across all groups
Uscher-Pines, 2020 (Substance Abuse)50	Video	Routine	Somewhat clear	Not clear	Different across groups
Uscher-Pines, 202144	Video, audio	Routine	Not clear	<3 months	Unclear
Vilendrer, 202049	Video, remote monitoring	Routine	Very clear	≥6 months	Same across all groups
Volcy, 202142	Video, audio	Routine	Somewhat clear	<3 months	Unclear
Zakaria, 201948	Remote monitoring	Routine	Very clear	≥6 months	Unclear

Telemedicine acceptance

Reviewed articles reveal high acceptance and belief in the ability of telemedicine to provide quality care beyond the COVID-19 pandemic among clinicians.^34,57 In particular, smartphone-based telemedicine for asthma care had high acceptability among clinicians⁵⁷ and a national survey (with a sample of n = 157 FQHC clinicians) found 80% believed in the efficacy of telemedicine to deliver quality care postpandemic as either a supplement or substitute for in-person care for certain patient populations (e.g., patients with mobility issues).³⁴

Effectiveness of telemedicine-based care

Of the articles performing analysis of the quality of care and patient outcomes provided by telemedicine (n = 35), n = 21 (60%) articles found positive results,^{29,31,34,38–40,42,46–48,55–57,60,62–67,69} n = 1 (3%) study had null results,⁷² and n = 13 (37%) articles had mixed results.^{30,32,33,36,37,41,43,51–53,58,61,71} Another n = 10 articles were descriptive only in nature and did not test a hypothesis or analyze patient, clinician, or organizational outcomes.^{4,35,44,45,49,50,54,59,68,70} Since a variety of settings are captured under the term “safety net,” we also analyzed differences in outcomes by federal designation (FQHC, CHC, RHC, academic, safety net) according to the articles. Of the articles with positive results, n = 7 examined FQHCs,^{34,56,62,63,66,67,69} n = 4 CHCs,^31,40,46,67 n = 7 academic settings,^{29,39,47,48,60,64,65} and n = 4 safety net settings.^38,42,55,57 The study with null results examined an FQHC.⁷² Of the articles with mixed results, n = 7 examined FQHCs,^{37,43,52,53,58,61,71} n = 2 CHCs,^36,52 n = 4 academic settings,^30,33,36 and n = 4 safety net settings.^32,41,51,53

Finally, of the descriptive articles, n = 7 examined FQHCs,^{35,44,45,50,54,59,68} n = 3 CHCs,^45,50,68 n = 2 RHCs,^45,68 n = 2 academic settings,^49,70 and n = 1 safety net settings.⁴ A brief summary and valance of each article's conclusions are presented in Table 2.

Table 2.

Summary and Valance of Conclusions by Article

ARTICLE	VALENCE OF CONCLUSIONS	SUMMARY OF TELEMEDICINE-RELATED CONCLUSIONS	PRE- OR DURING PANDEMIC DATA	INCLUDED POPULATIONS OF INTEREST	SUBGROUP ANALYSES	SETTING FEDERAL DESIGNATION
Adams, 2021	Positive	Telemedicine increased access and provided similar patient satisfaction to in-person visits for psychiatric care.	During	African American, Latino/Hispanic	—-	CHC
Anderson, 2010	Null	Telephonic disease management support did not improve clinical or behavioral outcomes compared with usual care.	Pre	Limited English proficiency	Limited English proficiency, Depression status, Education	FQHC
Armstrong, 2011	Mixed	Teledermatology increases access for patients, but improvements in reimbursement, design, communication, and training are needed to sustain virtual services.	Pre	—-	—-	FQHC
Barney, 2020	N/A	Telemedicine was feasible and acceptable to patients, but future analysis is needed to analyze concerns about privacy, quality of care, and health disparities.	During	—-	—-	Academic
Caton, 2021	N/A	There was high adoption of telemedicine for treatment in opioid use disorder in California, but impact on patient outcomes remains unclear.	During	—-	—-	FQHC, CHC, RHC
Chang, 2021	N/A	Telemedicine adoption was high during the COVID-19 pandemic in New York City, but was less likely to be adopted and faced more barriers to implementation in communities with high social vulnerability.	During	—-	Social Vulnerability Index score	Safety net
Childs, 2021	Mixed	Telemedicine increased appointment attendance rates compared with in-person services, but these effects were observed differentially across racial/ethnic groups, potentially exacerbating disparities.	During	African American, Latino/Hispanic	African American, Latino/Hispanic, Insurance type, Intensive outpatient program type	Safety net
Clifton, 2003	Positive	Telepharmacy had high acceptance among patients of CHCs and increased access to medications and pharmacy services.	Pre	65 and older	65 and older	FQHC
Coffman, 2016	N/A	In 2014, a nation survey found only 15% of family physicians reported using telemedicine with users more likely to be employed in federally designated “safety net” clinics and HMOs.	Pre	—-	—-	FQHC, CHC, RHC
Coker, 2019	Positive	Children in a telemedicine-enabled referral process in community mental health clinics were three times more likely to complete initial screening visits than usual care with higher satisfaction scores.	Pre	African American, Latino/Hispanic	—-	FQHC. CHC
Davis, 2010	Positive	Diabetes self-management education delivered through telemedicine was effective in improving metabolic control and reducing cardiovascular risk in a population that was primarily rural and composed of racial/ethnic minorities.	Pre	African American	—-	FQHC
Dayal, 2019 (Neurology)	Positive	Compared with in-person visits, telemedicine increased attendance of outpatient pediatric neurology and was more likely to be used by patients with nonprivate insurance, lower education, and lower household income.	Pre	—-	Insurance type, Patient distance to nearest hospital	Academic
Dayal, 2019 (JAMA)	Positive	Telemedicine reduced hospital utilization for pediatric neurology compared with in-person care.	Pre	—-	—-	Academic
Dunham, 2021	Positive	Telemedicine allowed the respectful and Equitable Access to Comprehensive Health care (REACH) Program to maintain uninterrupted care to patients during the COVID-19 pandemic with a hybrid of telemedicine and in-person appointments.	During	African American, Latino/Hispanic	—-	Academic
Fortney, 2013	Positive	Telemedicine patients of rural FQHCs had better outcomes across multiple aspects of collaborative care for depression than patients receiving practice-based care.	Pre	African American	—-	FQHC
Franciosi, 2021	Mixed	Telemedicine reduced no-show rates, but increased the proportion of younger, English-speaking patients in many specialties. Some specialties also saw an increase in the percentage of white patients with telemedicine, and primary care and adult nonsurgical providers saw an increase in Medicare patients.	During	African American, Latino/Hispanic, Asian, Limited English proficiency	African American, Latino/Hispanic, Asian, Insurance type	Academic
Friesen, 2015	Positive	Qualitative interviews with key participants at CHCs showed telelactation sessions were easy to implement, widened the client base, increased access, and reduced mothers' anxiety about the birthing process and hospital experience.	Pre	African American	—-	FQHC
Futterman, 2020	Positive	Telemedicine allowed for appropriate continuation of satisfactory prenatal care with no impact on patient-perceived satisfaction of care during the COVID-19 pandemic.	During	African American, Latino/Hispanic, Limited English proficiency	African American, Latino/Hispanic, Limited English proficiency	Academic
Grubbs, 2018	Mixed	Despite telemedicine being more effective than usual care overall, telemedicine was a less-effective avenue for veterans receiving care for depression in the VA compared with FQHC patient populations receiving similar care, putting veterans at higher risk of nonresponse than FQHC patients.	Pre	African American	Veteran status, Gender	FQHC
Hernandez, 2016	Positive	Telemedicine was feasible to implement for children presenting to non-children's hospital EDs and allowed for effective collaboration between physicians to provide adequate and timely treatment.	Pre	—-	—-	Academic
Howren, 2021	N/A	A brief quality improvement study concluded that older, rural adults showed a low willingness to use telemedicine to access mental health services.	During	—-	—-	FQHC
Khoong, 2021	Mixed	Safety net patients are interested and able to complete video visits, although many face barriers related to internet and mobile data access.	During	African American, Latino/Hispanic, Asian, 65 and older, Limited English proficiency	African American, Latino/Hispanic, Asian, 65 and older, Limited English proficiency	Safety net
Lin, 2018	N/A	A study that outlines common policy-level facilitators and barriers to telemedicine adoption.	Pre	65 and older	African American, Latino/Hispanic, Asian, 65 and older, Limited English proficiency	FQHC
Mammen, 2020	Positive	Smartphone-based telemedicine improved clinical asthma management, adherence to guidelines, and patient outcomes with high levels of patient and clinician acceptability.	Pre	African American, Latino/Hispanic, Asian	—-	Safety net
Mills, 2021 (AHA)	Positive	A survey among 587 predominantly low-income and minority patients with hypertension in Louisiana and Mississippi found that the COVID-19 pandemic reported high rates of protective practices to prevent the spread of COVID-19 and of access to quality health care during the pandemic either in-person or by telemedicine. In addition, patients are willing to return to their clinics for health care.	During	African American, Latino/Hispanic	African American, Latino/Hispanic, 65 and older	CHC
Mills, 2021 (Telemedicine)	Positive	Telemedicine provided an efficient way to screen for and provide education on COVID-19, as well as providing a secure alternative to in-person care. Increased telemedicine use was also associated with decreased burnout among primary care residents.	During	—-	—-	CHC
Mittal, 2014	Mixed	Telemedicine was not differentially associated with outcomes of a depression treatment intervention in an underserved population compared with in-person care, but the intervention yielded low treatment response rates for both in-person and virtual interventions.	Pre	—-	Caucasian vs not Caucasian, Income, Insurance type, Education	FQHC
Nguyen, 2021	Mixed	Implementation of telemedicine in free clinics may be feasible, but more solutions for patients with smartphone-only internet access are needed.	During	African American, Latino/Hispanic	African American, Latino/Hispanic	Academic
Nies, 2021	Positive	The majority of surveyed clinicians in FQHC settings believed telemedicine would be useful for providing care after the COVID-19 pandemic is over.	During	—-	—-	FQHC
Parnell, 2020	Positive	Through the implementation of virtual postoperative visits for laparoscopic cholecystectomy patients, clinic efficiency improved by increasing new patient encounters, decreasing postoperative volume, and trending toward increased operations scheduled without compromising patient safety.	Pre	—-	—-	Safety net
Patton, 2021	Positive	Hybrid telemedicine provided many benefits to pregnant patients diagnosed with substance use disorder and yielded overwhelmingly positive responses to implementation.	During	African American, Latino/Hispanic	—-	Academic
Phenicie, 2021	Mixed	Telemedicine helped overcome access barriers for rural patients without compromising patient satisfaction. However, older patients were less satisfied with telemedicine than their younger counterparts.	During	African American, Latino/Hispanic	65 and older, Limited English proficiency	FQHC
Pyne, 2015	Positive	Telemedicine-based collaborative care for depression in rural FQHCs was found to be more cost-effective than a similar in-person model.	Pre	African American	—-	FQHC
Richter, 2015	Mixed	Compared with telephone counseling to help rural patients quit smoking, integrated telemedicine increased utilization of cessation pharmacotherapy and produced higher participant satisfaction, but phone counseling was significantly less expensive.	Pre	Latino/Hispanic	—-	Safety net, FQHC
Rosal, 2014	Mixed	It was feasible to deliver diabetes self-management interventions to inner city African American women through virtual worlds with outcomes comparable to in-person interventions, but the virtual intervention was more expensive and was slightly less effective at A1c and depression reduction.	Pre	African American	—-	Academic, CHC
Shin, 2014	N/A	Thirty-seven percent of respondents to a national survey of FQHCs provided some type of telemedicine service. FQHCs that provide at least one telemedicine service are more likely to be located in rural areas and FQHCs that provide two or more telemedicine services are more likely to have generous state and local funding.	Pre	—-	—-	FQHC
Simon, 2021	Mixed	Despite increasing volume of telemedicine visits, FQHCs saw a drop in services provided and delays of routine care during the COVID-19 pandemic.	During	—-	—-	FQHC
Spinelli, 2020	Mixed	Despite higher-than-expected telemedicine utilization in San Francisco, the odds of viral nonsuppression after the start of COVID-19 was 31% higher than prepandemic, with homeless individuals facing the highest odds of negative impact.	During	—-	African American, Latino/Hispanic, Asian, Homelessness	Safety net
Tolou-Shams, 2021	Positive	At an urban safety net hospital providing child mental health services during the COVID-19 pandemic, no-show rates significantly declined after the implementation of telemedicine and service delivery volume was unchanged compared with pre-COVID-19 in-person visits.	During	—-	—-	Safety net
Uscher-Pines, 2020 (Psychiatric)	Mixed	Among community mental health centers, most used telemedicine in adjunct with in-person care. Most health centers planned to continue using telemedicine, but noted less patient engagement, challenges sharing information within care teams, and greater inefficiency.	Pre	—-	—-	FQHC, CHC
Uscher-Pines, 2020 (Substance Abuse)	N/A	Eight out of 22 health centers in 14 states reported offering tele-opioid use disorder treatment, with medication management as the most commonly cited use. Usually, telemedicine was only offered after an in-person consultation and leading barriers included regulations on the prescribing of controlled substances, including buprenorphine, and difficulties in sending laboratory results to distant (prescribing) providers.	Pre	—-	—-	FQHC, CHC
Uscher-Pines, 2021	N/A	Despite primary care visit volume declining in FQHCs during the COVID-19 pandemic, behavioral health visit volume remained stable primarily because telemedicine replaced in-person visits (particularly by telephone).	During	—-	—-	FQHC
Vilendrer, 2020	N/A	In an analysis of three institutions during the beginning of the COVID-19 pandemic, all were able to adopt inpatient video calls. Rapid deployment was facilitated by direction from executive leadership, leveraging off-the-shelf hardware, vendor engagement, and clinical workflow integration.	During	—-	—-	Academic
Volcy, 2021	Positive	A majority of patients, faculty, and residents in internal and family medicine reported positive perceptions of telemedicine in a survey conducted after the start of COVID-19.	During	—-	—-	Safety net
Zakaria, 2019	Positive	An urban safety net hospital found an increase in access and efficiency of dermatology after the implementation of teledermatology.	Pre	African American, Latino/Hispanic, Asian	—-	Academic

CHC, community health centers; FQHC, federally qualified health centers; RHC, rural health centers.

Patient and clinician satisfaction

Articles reveal that telemedicine is found to be acceptable from the perspective of safety net patients, with a high level of interest among clinical providers to use telemedicine to deliver primary care services across settings (n = 4 FQHCs, n = 1 CHCs, n = 2 academic, n = 1 safety net).^{34,39,41,62,67,69,70} From an administrative standpoint, telemedicine can be successfully implemented in these traditionally resource-constrained settings for primary care delivery with the added potential benefits of enhancing team coordination and increasing efficiency.^{29,33,36,38–41,43,45–48,51–53,55,56,60–63,65–67,69,71} One study suggests telemedicine implementation may reduce clinician burnout through increased workflow efficiency while providing care to underserved patients.⁴⁶ One pre/post comparison found that the same clinicians had, on average, a two-point reduction on an abbreviated Maslach Burnout Inventory post-telemedicine implementation survey, with reductions in burnout associated with less emotional exhaustion and depersonalization.⁴⁶

Another potential strength of utilizing telemedicine is that it may help to bridge the gap between urban and rural patients, where the convenience and elimination of travel times can help to improve care access for rural patients, reduce existing pre-telemedicine gaps in care utilization, and decrease missed appointments.^{43,53,54,56,63,66}

However, telemedicine may not be an appropriate substitute for in-person care in all scenarios and with all patient populations.^{4,30,35,43,51,61} For example, certain patients in safety net settings, like older patients in rural regions, may have lower interest in and satisfaction with telemedicine modalities.^35,43 Additionally, while several articles found parity between telemedicine and in-person patient outcomes,^{55,57,60,61,66,72} Rosal et al. found that a telemedicine intervention improved A1c control and depression reduction compared with baseline, but less so than an in-person intervention for inner city African American women.³⁶

Grounding in implementation science theories, model, frameworks, and concepts

Only n = 3 (7%) articles grounded their research in a previously published implementation theory, model, or framework, using a total of four frameworks and models.^45,66,67 The four frameworks and models utilized were the Exploration, Preparation, Implementation, and Sustainment (EPIS) Framework⁷⁴; the Donabedian Model⁷⁵; the Health Belief Model⁷⁶; and the Transtheoretical Model.⁷⁷ All three articles utilized these frameworks to guide intervention development, maximize the potential for impact, and analyze aspects of the setting that facilitate adequate supports and resources for success. While one study was descriptive in nature and used the EPIS framework to guide thinking about clinic categorization for opioid use disorder treatment,⁴⁵ two others focused on patient outcomes and demonstrated improved diabetes outcomes⁶⁶ and increased patient access to community mental health clinics.⁶⁷

Twenty (44%) articles examined the implementation of a telemedicine program longer than 3 months.^{30,36,43,48,49,53,54,56–58,60–67,71,72} Sixteen (36%) articles gave the same intervention to all patients^{29,30,32–35,46,49,52,55,57,60,62,64,69,70} compared with 20 (44%) articles that studied inconsistent interventions across patients and/or sites.^{4,31,36,39–41,45,50,51,53,54,56,58,61,63,65–67,71,72} Nine (20%) articles had unclear/missing descriptions of the studied telemedicine program.^{37,38,42–44,47,48,59,68} Thirty-four (76%) examined a long-lasting intervention^{4,29–35,37–52,54,55,59,60,64,65,68–71} and 11 (24%) implemented telemedicine as a trial only lasting the study period.^{36,53,56–58,61–63,66,67,72} Details on the implementation and sustainability of identified telemedicine interventions are presented in Table 1.

Barriers and facilitators to implementation

A third (n = 15) of the articles explicitly mentioned barriers to implementation^{4,29,35,42,43,50,52,55,59,60,63,67,69–71} and n = 8 (18%) mentioned facilitators.^{31,32,39,45,60,62,67,71}

Identified barriers included: billing/administrative workflow disruption (n = 9, 20%),^{4,43,50,52,55,59,60,63,71} lack of quality broadband access (n = 5, 11%),^{4,52,59,69,71} low patient acceptance/preference for in-person care (n = 4, 9%),^4,35,50,52 clinical workflow disruption (n = 4, 9%),^50,52,69,70 lack of technical/implementation expertise (n = 4, 9%),^4,59,70,71 lack of interpretation or translation of service (n = 3, 7%),^4,43,55 and lack of regulatory support (n = 3, 7%).^50,59,69 Two articles, respectively, also mentioned patient digital literacy,^4,29 clinician/staff training and resource requirements,^70,71 privacy concerns,^52,70 and safety/quality of care concerns.^4,45 A 2020 qualitative analysis of n = 11 CHCs and n = 9 FQHCs in 14 states and their telemedicine capacity for mental health services highlighted most of these barriers, including difficulty sharing information, assessing a patient's physical state, and establishing rapport with patients through a virtual medium, especially with “warm hand-offs” that are central to behavioral health.⁵²

Identified facilitators to telemedicine implementation were efficiency gains (n = 6, 13%),^{39,45,60,62,67,71} patient acceptance (n = 3, 7%),^39,62,67 and enhanced patient access (n = 3, 7%).^39,62,71 Two articles reported telemedicine being inexpensive/more cost-effective compared with in-person appointments.^62,71 One article, respectively, mentioned the availability of training for clinicians and/or staff (n = 1, 2%),³¹ and reimbursement/payment for telemedicine appointments (n = 1, 2%).³² One 2021 article noted that some facilitators essential to enhancing digital health equity for substance use treatment and prenatal care delivery during the COVID-19 pandemic included reduced patient transportation and childcare needs for appointments, which reduced missed appointment rates and facilitated the treatment of pregnant and postpartum patients diagnosed with opioid use disorder.³⁹

Of articles examining FQHCs, n = 6 identified barriers related to billing/administrative workflow disruption,^{43,50,52,59,63,71} n = 4 a lack of quality broadband access,^52,59,69,71 n = 3 low patient acceptance/preference for in-person care,^35,50,52 n = 3 clinical workflow disruption,^50,52,69 n = 2 lack of technical/implementation expertise,^59,71 and n = 1 a lack of interpretation or translation of service.⁴³ All three articles identifying a lack of regulatory support were conducted in FQHCs.^50,59,69 Of articles examining CHCs, n = 2 identified barriers related to billing/administrative workflow disruption,^50,52 n = 1 a lack of quality broadband access,⁵² n = 1 low patient acceptance/preference for in-person care,⁵⁰ and n = 2 clinical workflow disruption.^50,52 For articles analyzing academic settings, n = 1 identified barriers related to billing/administrative workflow disruption,⁶⁰ n = 1 clinical workflow disruption,⁷⁰ n = 1 lack of technical/implementation expertise.⁷⁰

Among articles conducted in safety net settings, n = 2 identified barriers related to billing/administrative workflow disruption,^55,60 n = 1 lack of technical/implementation expertise,⁴ and n = 2 a lack of interpretation or translation of service.^4,55 No articles that explicitly mention barrier to telemedicine implementation were conducted in RHCs.

Of articles that identified efficiency gains as a facilitator to telemedicine implementation, n = 4 included FQHCs in the sites,^45,62,67,71 n = 2 CHCs,^45,67 n = 1 RHCs,⁴⁵ and n = 2 included an academic setting.^39,60 Of articles that identified patient acceptance as a facilitator to telemedicine implementation, n = 2 included FQHCs in the sites,^62,67 n = 1 CHC,⁶⁷ and n = 1 included an academic setting.³⁹ Both articles reporting telemedicine being inexpensive/more cost-effective compared with in-person appointments were conducted in FQHCs.^62,71 The article mentioning the availability of training for clinicians and/or staff was conducted in a CHC,³¹ and the article identifying reimbursement/payment for telemedicine appointments as a facilitator of telemedicine implementation was conducted in a safety net setting.³²

Full details of barriers and facilitators are presented in Supplementary Appendix SA3.

Diversity, equity, and inclusion of telemedicine implementation in safety net settings

While examining these articles for diversity, equity, and inclusion considerations, we found that in the demographics sections of these articles only n = 19 (42%) included African American/Black patients,^{29–33,36,39,41,43,46–48,56,57,61–63,66,67} n = 14 (31%) LatinX/Hispanic patients,^{29–33,39,41,43,46–48,53,57,67} n = 4 (9%) Asian patients,^30,41,48,57 n = 4 (9%) patients 65 years of age or older,^41,43,59,69 and n = 4 (9%) patients with limited English proficiency.^30,41,47,72

While extracting data on subgroup analyses of groups of special interest, n = 8 (18%) included special analyses for African American/Black patients,^{30,32,33,41,46,47,51,59} n = 8 (18%) LatinX/Hispanic patients,^{30,32,33,41,46,47,51,59} n = 4 (9%) Asian patients,^30,41,51,59 n = 5 (11%) patients 65 years of age or older,^{41,43,46,59,69} n = 4 (9%) patients with limited English proficiency,^41,43,59,72 n = 4 (9%) patients without private insurance,^30,32,58,65 n = 2 (4%) patient with varying levels of education,^58,72 and n = 7 (16%) for other subgroups, which included n = 1 (2%) study each examining gender,⁶¹ veteran status,⁶¹ homelessness,⁵¹ social vulnerability,⁴ low income,⁵⁸ distance from nearest clinic,⁶⁵ and patients with depression.⁷² Analyses of these subgroups yielded group-specific barriers and facilitators, such as enhanced interest in video visits through smartphone applications by non-English-speaking patients; technology use barriers for older, non-English-speaking, Black or LatinX patients, and low interest and satisfaction in telemedicine for rural older adults, despite high telemedicine adoption and satisfaction among rural patients overall.^41,43,59

Overall, n = 24 (53%) articles^{29–33,36,39,41,43,46–48,56,57,61–63,66,67} explicitly included a population of interest in the demographics section and a smaller subset of n = 15 (33%) articles^{30,32,33,41,46,47,51,59} performed a subgroup analysis on a population of interest.

Supplemental analysis of 2022 articles

In an attempt to keep up with rapidly evolving evidence, we also analyzed articles published in 2022 through February 2023 outside of the systematic review process. The lead author used the same search terms as the main systematic review to find articles on PubMed that met the inclusion criteria of the study and summarized the evidence to provide additional understanding. Articles summarized through this process did not undergo the full screening and abstraction phases of the systematic review.

In 2022 through February 2023, 12 articles fit our inclusion criteria of our systematic review.^78–89 Of these, six articles examined telemedicine's interaction with racial and ethnic disparities,^{78–81,88,89} one study examined disparities between limited English-proficient patients and English-speaking patients,⁸⁶ four examined disparities between rural and urban populations,^80,81,84,87 and three examined disparities within vulnerable safety net populations in general.^82,83,85 Similar to the findings in this review, patients who did use telemedicine missed fewer appointments regardless of race/ethnicity. One study also highlighted the importance of audio-only visits for Asian American patients of FQHCs.⁸⁹

These articles also had mixed results when examining rural–urban disparities, finding that rural patients had lower telemedicine adoption and higher rates of missed appointments, but also finding a dose–response between rurality and telemedicine adoption, with increased likelihood of telemedicine adoption being associated with greater geographic distance from the nearest urban center regardless of race/ethnicity of the patient. These articles also found that when rural patients used telemedicine, it helped mitigate rural–urban disparities in missed appointments. Of these more recent articles, only one study used an implementation science framework,⁹⁰ which could have helped improve evidence about telemedicine acceptance among racial and ethnic minority groups. Of the 39 recent articles,^{38,78,80–115} all articles had at least one major positive finding regarding telemedicine's ability to improve patient satisfaction, decrease missed appointments, and potentially mitigate disparities for safety net patients.

Discussion

This systematic review found that telemedicine is an acceptable care modality for primary care services in safety net settings and has the potential to positively impact patient outcomes and improve quality of care through enhanced efficiency and ease of access to care. As evidence has accumulated rapidly in the past few years, empirical trends are beginning to emerge.

We found that only a fifth of included safety net telemedicine articles were randomized controlled trials. Given the expense and complexity of implementing changes to telemedicine for experimental purposes and the need to rapidly respond to COVID-19 in the past (as n = 30, or 66.7% of the articles included were published during the pandemic), randomized controlled trials are often infeasible. Observational articles benefit from their pragmatism and enhanced generalizability.¹¹⁶ Most (76%) of included articles also analyzed a routine implementation of telemedicine, rather than a trial with a set start and end date, providing enhanced learnings about adaptations and practical uses and design considerations. More evidence gathered by quasi-experimental methods paired with natural experimental articles could build the knowledge base needed on telemedicine implementation in safety net settings while maintaining feasibility that may not be possible in these settings with randomized controlled trials.

Despite gaps that exist in the form of health disparities and the “digital divide,” 21 articles in this review highlight the utility of telemedicine to serve the needs of safety net populations. Discovering ways to successfully integrate telemedicine into administrative and clinical workflows and finding ways to support telemedicine use among patients with limited access to broadband and limited English proficiency through policy or organizational-level interventions may address existing barriers to telemedicine adoption and sustainability in safety net settings. For instance, Khoong et al., found that non-English speakers in an urban safety net setting were more interested in video visits than English speakers and perceived language barriers as easier to overcome with visual cues.⁴¹ Clinics that use interpretation services for visits with non-English-speaking patients may find it useful to explore directing these patients to video-based telemedicine services and offer video interpreters, helping to overcome barriers to telephone-based interpretation also observed in Parnell et al.⁵⁵

However, as with a few other articles in our review, such as Barney et al., privacy concerns over the ability to create a quiet and isolated location for appointments as well as data security were a barrier to patient acceptance of telemedicine.⁷⁰

Other articles included in our review demonstrate that telemedicine is an acceptable and effective method to deliver health care to safety net populations. Adams et al., studied the implementation of a telemedicine clinic in an urban setting for patients experiencing homelessness.³¹ This clinic allowed for remote assessment of patients by a group of 10 rotating family medicine resident physicians with medical students staffing the sites within the community. Medical students helped with administrative and logistical tasks, such as entering patient record data, obtaining vital signs, and assisting patients with the remote stethoscope, ophthalmoscope, and dermatoscope offered at the remote site. Through this established drop-in telemedicine clinic and training for staff, the drop-in center was able to meet patient needs while reducing emergency department utilization and providing patients needing a referral with appointments time within 24 h of their visit to the drop-in clinic. This set-up maintained flexibility for clinicians, while helping to overcome barriers to access such as lack of housing and broadband access and maintaining high levels of satisfaction among clinicians and patients.

High rates of clinician acceptance of telemedicine in safety net settings were also exhibited in two articles previously mentioned,^34,57 as well as an association between reduced burnout among residents and telemedicine adoption.⁴⁶ Clinician buy-in to telemedicine is a key facilitator of overall telemedicine adoption and another factor that points to the potential longevity of telemedicine in safety net settings.

Similar to many of the reviewed articles, prior systematic reviews of health information technology-based interventions have found that consideration of and adaptation to existing workflows is an important factor that influences implementation success.^117,118

These reviews also highlight the importance of policy support at both the organizational (e.g., training, management support, resource availability, supporting infrastructure) and government (e.g., reimbursement, incentives, supportive policy) levels,^118–122 thus signaling the importance of continued support for telemedicine reimbursement and policy that allows for flexibility. This support includes the outer context of telemedicine implementation, which includes ensuring patient access to broadband internet and digital devices capable of video and telephone-based care. The Lifeline Program for Low-Income Consumers,¹²³ a federal program that subsidies internet bills and payments for electronic devices is an example of a policy that provides a supportive outer context for telemedicine implementation. We see the importance of these supports echoed in the articles included in our review that describe facilitators and barriers to implementation.^{4,29,31,32,35,39,42,43,45,50,52,55,59,60,62,63,67,69–71}

Childs et al., mentions changes to CMS reimbursement policies during the pandemic as a key support for providing telemedicine to vulnerable and high-risk populations.³² Armstrong et al., also cited lack of reimbursement and inability to incorporate telemedicine into a sustainable business model as major barriers to sustainable teledermatology as reported by interviewed dermatologists.⁷¹ One of the greatest advantages of telemedicine that could facilitate its use is that it helps eliminate rather than add, such factors as mentioned by Patton et al.³⁹ Patients who have limited time in their schedules to seek health care such as individuals working multiple jobs, who are parents or caregivers, or who reside in rural areas far away from the nearest health care facility, may see enhanced value from no longer needing to travel or obtain childcare to attend a medical appointment.

Very few articles assessed sustainability of telemedicine, and all but one of the reviewed articles that assessed sustainability were published post-2020. Given recent developments in policy, telemedicine is likely to remain an option for patients moving into the future. It is important that the sustainability of various types of telemedicine implementations are understood to conserve resources and maximize impact for health organizations, care teams, and patients. Similarly, it is important to understand how the use of telemedicine systems may differentially impact patients of color, older patients, and other marginalized patients. There has been recent investment by the U.S. federal government to fund research in safety net and other settings, which may indirectly capture learnings relevant for effective telemedicine implementation for marginalized populations.¹²⁴ Many of the studies funded by these investments are included in this review.

However, without explicit inclusion and analysis, we are not able to confidently identify specific strategies for effective implementation and sustainability of telemedicine interventions that help rather than create barriers for marginalized populations. Given the low proportion of published research studies that primarily focus on telemedicine implementation among patients of color, patients with limited English proficiency, and other at-risk groups, there is risk of exacerbating inequities in access to care and quality of care unless more evidence is generated that can be used to better support telemedicine use for these subpopulations. Even if telemedicine use is sustained and shown to be efficacious, it is critical that health care organizations do not further exacerbate already existing health disparities through the implementation of novel forms of care delivery. Rigorous diversity, equity, and inclusion research in academic medicine is central to creating the knowledge base needed to meaningfully address systemic gaps in health services' administration and medicine at large.¹²⁵

Few articles reported on barriers (n = 15) and facilitators (n = 8) to telemedicine implementation, and even fewer (n = 3) were grounded in implementation theories or frameworks. First, reporting on barriers and facilitators could reveal key lessons to improve the optimization of telemedicine implementation. The low level of articles reporting on barriers and facilitators limits the ability to translate research findings into practice and spread the use of evidence-based practices, a central aim of implementation science.¹²⁶

Relatedly, using implementation frameworks, models, and theories can provide valuable context and standardization, while reducing the research-practice gap and move the field toward an integrated body of knowledge.¹⁵ This could be especially useful when trying to address inconsistencies and gaps in telemedicine use and effectiveness for historically marginalized groups. Frameworks such as the Consolidated Framework for Implementation Research and others may also help frame important considerations such as creating a supportive outer context, establishing the proper information technology infrastructure for implementation, and engaging with and adapting to stakeholder needs and preferences.¹⁴

More specialized frameworks such as the Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies, can help developers of novel telemedicine technologies consider how these technologies can better fit into workflows and the overall health care organization while assessing potential aspects of the technology that could lead to inequalities in access, uptake, and use by different demographic groups and addressing them in early stages.¹²⁷

Finally, when analyzing for differences across the domains presented in our results by the type of settings (FQHC, CHC, RHC, academic, safety net), we found that among studies reporting facilitators and barriers to telemedicine implementation, various types of safety net organizations experience similar facilitators, barriers, and factors that influence acceptance and satisfaction among patients and clinicians. However, none of the articles examining patient and clinician satisfaction and only one of the articles that identified facilitators or barriers examined RHCs. The lack of research regarding satisfaction and implementation facilitators and barriers in RHCs highlights a need for further evidence generation regarding factors that influence telemedicine implementation in rural settings. Given the distribution of studies in our categorization of valence of outcomes, any evidence about whether telemedicine effectiveness varies setting type are inconclusive.

Recent articles published in 2022 have continued to highlight disparities in telemedicine adoption with non-Hispanic White patients being much more likely to adopt and use telemedicine than patients of other racial and ethnic backgrounds. Five recent articles found that established relationships between clinicians and patients was a key facilitator of telemedicine adoption.^{79,96,98,110,114} Of particular note, however, is one study that found high telemedicine satisfaction across racial/ethnic groups, but low satisfaction among Hispanics.⁸⁸ In a positive trend, 18 articles that examined telemedicine implementation more recently had a minimum 9-month period,^{78–85,90,92,100,101,105,110,111,113,115} which is an improvement compared with the articles we included in the review. Having a longer study period will provide greater insight as to how telemedicine can be sustained in safety net settings.

Limitations of this systematic review include the use of a single database to gather articles for analysis. However, the risk of missing relevant articles published in the literature is low given the comprehensive nature of PubMed on the topic of medicine and health services research. As with all systematic reviews, publication bias¹²⁸ has the potential to impact the findings of this study. Research with statistically significant positive findings is more likely to be published than articles with null or negative findings. The prevalence of articles with positive findings may still be an indicator of the potential promise of telemedicine, but also the shortcomings of telemedicine implementation in CHCs may not be accurately reflected by the available literature.

This systematic review also only analyzed articles published through 2021. However, the inclusion of articles with data collected prepandemic and during the pandemic is a strength of this review.

Future research should compare the effectiveness of telemedicine and in-person quality of care and strategies to implement telemedicine for safety net populations. Future studies should also address the gap in adoption and patient perceptions of quality of care provided through telemedicine between non-Hispanic White patients and patients of other racial/ethnic backgrounds.

Conclusion

Telemedicine is a potentially promising vehicle to improve primary health care access and quality for patients in safety net settings, especially for younger, rural populations. This review aims to synthesize evidence that could help ensure safety net populations are not left behind as telemedicine is sustained and expanded. Positive results presented in this systematic review suggest telemedicine could provide quality primary care that is potentially more accessible and affordable by adding more flexibility necessary for patients that have tight constraints on their time and resources, such as eliminating the need for travel or childcare. Articles also show high rates of adoption and acceptability among safety net patients, as well as high acceptance and perceived usefulness among clinicians who work in safety net settings.

However, a lack of articles with negative and/or null results should be taken with caution, as this may be the result of publication bias. Since policies supporting the continual use of telemedicine beyond the COVID-19 pandemic are gaining support and are being implemented across states, more evidence is needed to ensure that gaps in care quality and accessibility for vulnerable and underserved populations are not exacerbated.

Authors' Contributions

A.A.T.: conceptualization (support), data curation (lead), formal analysis (lead), investigation (lead), methodology (equal), writing—original draft (lead), writing—review and editing (lead), and project administration (lead). M.M.: formal analysis (support), and investigation (support). G.C.: formal analysis (support), and investigation (support). J.L.F.: conceptualization (lead), and methodology (equal). D.D.P.: conceptualization (support), methodology (equal), and supervision (support). H.P.R.: conceptualization (support), supervision (lead), funding acquisition (lead), writing—original draft (support), and writing—review and editing (support).

Disclosure Statement

No competing financial interests exist.

Funding Information

This work was funded by a grant from the Center for Information Technology Research in the Interest of Society (CITRIS) and the Banatao Institute. A.A.T. received support from the Agency for Health Care Research and Quality (Grant No. T32HS022241).

Supplementary Material

Supplementary Appendix SA1

Supplementary Appendix SA2

Supplementary Appendix SA3