Patient satisfaction with telemedicine in acute care setting: A systematic review

Abstract

Background:

Telemedicine has revolutionized health-care services with its unprecedented abilities to connect patients with health-care professional across the distances. Patient satisfaction is an important measure of the quality and effectiveness of health-care services.

Aim:

The goal of this systematic review is to investigate patient satisfaction with telemedicine in acute care setting.

Methods and Results:

Four sources of data were searched PubMed, Cumulative Index of Nursing and Allied Health Literature, Scopus, and Web of Science. We used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis as our basis of organization. Our analysis has showed that acute telemedicine was effective in managing a broad spectrum of acute medical conditions while achieving high levels of patient satisfaction.

Conclusion:

Patient satisfaction is a complex product of expectations and experiences. Furthermore, it is an important indicator of the quality of the service. Despite the challenging nature of acute medicine, telemedicine services were successful in improving the quality of the service and achieving high levels of patient satisfaction.

Relevance for Patients:

Telemedicine is rapidly evolving as an essential component of our healthcare system. Implementing telemedicine in acute care is a relatively new concept and patient satisfaction in these settings needs to be evaluated.

1. Introduction

Telemedicine is defined as the use of communication technology to provide patients with medical information and services [1]. Telemedicine has emerged during the past two decades as a potential solution for many problems facing health-care systems around the world. Besides its inherent abilities in connecting patients to health-care providers across distances, it has proven its efficacy in reducing costs of medical services, in-hospital admission, and readmission rates while improving outcomes and patient satisfaction [2-4].

Despite the challenging complexity and ambiguity, patient satisfaction is an indispensable measure of the quality of any health-care service, including telemedicine services [5]. This importance arises from the fact that satisfied patients are both more likely to comply with their treatment plans as well as cooperate with their health-care providers [6]. In addition, satisfaction with a particular service increases chances of reuse of that service in the future, as well as recommending that service to others [7]. Finally, satisfaction with a service is a reflection of its quality. The health-care service has several quality attributes such as art of care, technical competency, accessibility, finances, physical environment, provider availability, continuity of care, and efficacy. Although it is essential to continuously assess and revise each of these attributes objectively, evaluation of patient satisfaction can give a holistic view of the quality of the system.

Since its introduction, telemedicine has been mostly utilized to serve patients with chronic medical conditions. However, the last few years have witnessed more utilization of telemedicine in acute medical care to solve some of the most complex challenges in this field such as emergency department (ED) overcrowding, the lack of health-care access in remote or underserved areas, and the high expenses of inpatient care.

ED offers various clinical services to a broad spectrum of clinical conditions from benign to life threatening making it liable to overcrowding. Emergency department overcrowding is defined by the American College of Emergency Physicians as a condition where the need for emergency services exceeds available resources for patient care in the emergency department and mainly results from ED boarding [8]. ED boarding is the practice of holding patients in the ED after admission to the hospital until an inpatient bed is available [9].

ED boarding is a major cause of ED overcrowding and results mainly from lack of institutional capacity relative to the number of cases rather than under-staffing, flaws in ED design, or poor performance of ancillary service as was previously believed [10]. ED overcrowding increases morbidity and mortality for both boarded and ED patients; it also increases the length of stay for admitted patients and decreases both patient and staff satisfaction [11,12].

The cost of inpatient care is high and accounts for a large percentage of total health-care spending: US health-care spending has increased to 3.8 trillion USD in 2019 and hospital care services accounts for about 31% of the total expenditure [13]. In general, the total expenditure on health care in the United States is expected to rise from 17.9% of the country’s gross domestic product (GDP) in 2017 – 19.7% by the end of 2028; an increase that is 1.1% faster than growth in GDP [14].

Although the utilization of telemedicine in outpatient settings has been associated with high levels of patient satisfaction [4], the fundamental difference between the acute and outpatient care mandates separate evaluation of patient satisfaction. Therefore, we conducted a systematic literature review to investigate the association between the utilization of telemedicine in the acute care setting and patient satisfaction.

2. Methods

2.1. Information sources, search strategy, and eligibility criteria

We used four sources of data for our search: PubMed (2010 – 2021), Cumulative Index of Nursing and Allied Health Literature (CINAHL) (2010 – 2021), Scopus (2010 – 2021), and Web of Science (2010 – 2021). We used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) as our basis of organization. The following Medical Subject Headings terms guided our search strategy: (“telemedicine” OR “homecare services”) AND (“patient satisfaction”). Search terms were adapted according to each database. Details on search terms used for every data base are provided in the Supplementary File.

Eligibility criteria were (1) published between 2010 and 2021, (2) English language, (3) humans only, (4) full text available, and (5) covering telemedicine in acute care setting and at least one measure of patient satisfaction.

2.2. Study selection and data collection process

Each author independently performed the search and removed the duplicated using EndNote (Clarivate Analytics). After filtering the studies based on titles; abstracts were then screened according to the aforementioned eligibility criteria. The remaining studies were screened based on full-text readings.

2.3. Data items and summary measures

We included all studies that covered both telemedicine in acute care and at least one measure of patient satisfaction such as communication, convenience, safety, privacy, likelihood to reuse the service, and likelihood to recommend the service to others. Studies that failed to cover both topics were excluded from the analysis.

2.4. Quality and risk of bias assessment

The quality and risk of bias of the included studies was assessed using the National Heart, Lung, and Blood institute (NHLBI) quality assessment tools. In 2013, NHLBI developed a group of quality assessment tools to assess studies’ internal validity. The tools were designed to and tested to detect potential flaws in study methods or implementation. Two authors used the appropriate tool for each study according to the study design and provided the results for each study in the Supplementary File.

3. Results

3.1. Study selection, study characteristics, and results of individual studies

The initial search revealed 307 results, after title and abstract screening, 283 results were excluded. Twenty-four papers underwent full-text screening resulting in 12 papers being included in the final review (Figure 1).

Figure 1. Preferred reporting items for systematic reviews and meta-analysis flowchart.

We have summarized the results of our analysis in three tables. Table 1 contains a summary of acute conditions treated, the technology used, the service provided through telemedicine, and a summary on patient satisfaction. Table 2 summarizes the type of questionnaire used for every study.

Table 1. Summary of the included studies.

Author, date, and country	Acute conditions managed	Technology used	Service provided through telemedicine	Home care services provided	Summary
Hernandez et al.[15], May 2018, Spain	Acute illness, exacerbation of chronic conditions, early discharge, post-surgical care	Video conferences through a digital platform	Remote monitoring, remote consultation	Home-based care plan, including daily nurse visits, physician’s visits, and laboratory tests such as ABG, blood analytics, and forced spirometry	High patient and provider satisfaction rates (98%) reported
Jakobsen et al.[16], May 2015, Denmark	Acute exacerbation of COPD	Touch screen with a Webcam	Remote monitoring and remote consultation	Oxygen therapy, aerosolized medications, steroids, sedatives	High satisfaction rates (100%) among patients and care providers
Mashru et al.[17], January 2017, Canada	Musculoskeletal infections, skin and soft-tissue infections, respiratory infections, and acute rheumatic fever	Video conferencing	Remote consultation	N/A	High patient satisfaction (98% overall satisfaction) was reported.
McIntosh et al.[18], December 2014, USA	Acute pediatric conditions as otitis media, conjunctivitis, and upper respiratory tract infections	Portable telemedicine units which include peripheral devices, connected to a laptop computer, that enable acquisition of high-resolution images, and lung sounds	Remote consultation through videoconference visits and store-and-forward (asynchronous) visits	N/A	Almost all survey respondents were satisfied or highly satisfied with neighborhood visits (97.6%) and endorsed greater convenience than alternatives (94.5%).
Mendez et al.[19], August 2013, Canada		RP-7 robot (In Touch Health Inc., Santa Barbara, CA, USA)	Remote consultation	N/A	High degree of patient, nurse, and physician satisfaction was reported.
Polinski et al.[20], August 2015, USA	Pharyngitis, sinusitis, otitis media, otitis externa, upper respiratory infections, bronchitis, allergic rhinitis, influenza	Video monitor with two-way audio and visual capabilities	Remote consultation	N/A	High degree of patient satisfaction was reported
Shah et al.[21], April 2013, USA	Cough, shortness of breath, muscloskeletal pain, face swelling, and chest pain	Electronic stethoscope, digital otoscope, high-resolution camera, and web camera linked to laptop, scanner, and printer	Remote consultation	N/A	High satisfaction among patients and providers
Summerfelt et al.[22], October 2015, USA	Acute exacerbation of COPD or CHF, asthma, DVT, and pneumonia	Central station and home monitoring station to allow virtual visits. Wireless devices for biometric measures (e.g., blood pressure)	Remote consultation and remote monitoring	Home visits by nurses and physicians, diagnostic procedures such as USS, and therapy including IV fluids and oxygen therapy	High satisfaction rates among patients. There was statistically significant better satisfaction with staff, convenience for caregivers, and comfort, convenience, and safety than in control group
Sykora et al.[23], September 2020, Czech Republic.	Not reported	Audiovisual calls between paramedics and physicians to evaluate low urgency calls	Remote consultation	Treatment on site for eligible cases by paramedics	Audiovisual consult improved the subjective feelings of safety by emergency physicians, but not of patients or paramedics
Teot et al.[24], December 2019, France	Surgical and traumatic wounds	Videoconference through a web platform	Remote consultation	Wound dressing and examination	The overall satisfaction of the patient and caregiver was high
Thomas et al.[25], October 2017, USA	Pediatric mental health emergencies	Videoconference	Remote consultation	N/A	Providers and patient caregivers reported high satisfaction with overall acceptability, effectiveness, and efficiency of telepsychiatry
Vitacca et al.[26], May 2010, Italy	Acute respiratory distress in patients with amyotrophic lateral sclerosis	Telephone calls	Remote consultation	Home visits for mechanical in-exsufflation, and manually assisted coughing	All patients were satisfied and 75% of patients were extremely satisfied with the service. About 86% considered the intervention effective while 14% considered it somewhat effective

Table 2. Type of questionnaire and aspects of patient satisfaction evaluated for every study.

Study	Type of survey conducted	Aspects of patient satisfaction evaluated	Comments
Hernandez et al.[15]	Self-completed questionnaire	Treatment received, likelihood to participate again	Validated questionnaire Qualitative data regarding the adoption of technology in the context of the service was assessed by the Method for Assessment of Telemedicine applications (MAST)
Jakobsen et al.[16]	Self-completed questionnaire	Technology, communication, convenience of equipment	Non-validated user satisfaction questionnaire
Mashru et al.[17]	Self-completed questionnaire	Technology, communication, privacy, staff, likelihood to recommend service, likelihood to reuse the service, overall satisfaction	Validated questionnaire
McIntosh et al.[18]	Self-completed questionnaire (by the parent or guardian)	Convenience, overall satisfaction, likelihood to reuse the service for the child, likelihood to use telemedicine by caregiver	Non-validated questionnaire The questionnaire was developed from parent focus groups, from key informant interviews with parents, health-care providers, and staff, and from an instrument previously used with school telemedicine service
Mendez et al.[19]	Self-completed questionnaire	Technology, communication, likelihood to reuse the service	The study depended on a combination of non-validated questionnaire and qualitative interviews to evaluate the satisfaction
Polinski et al.[20]	Self-completed questionnaire	Technology, treatment, staff, communication, convenience, likelihood to reuse service, likelihood to recommend service, preference of telehealth compared to traditional visits.	Non-validated questionnaire
Shah et al.[21]	Interview questionnaire	Communication, perceived value, unmet expectations, diagnosis certainty, staff training, technical issues	From the interviews, 196 discrete statements were identified. Thirty-one codes were developed and assigned to the various statements. These statements were then organized into eight themes and three overarching domains
Summerfelt et al.[22]	Self-completed questionnaire	Patients: convenience, safety, staff, communication, overall satisfaction, likelihood to reuse service, likelihood to recommend service to others. Caregiver/family member: Ease for family members, travel time, and missed work	Validated survey (Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS] survey)
Sykora et al.[23]	Self-completed questionnaire	Overall satisfaction	Non-validated questionnaire
Teot et al.[24]	Not reported	Overall satisfaction
Thomas et al.[25]	Self-completed questionnaire (by parent or guardian)	Technology, communication, comfort, likelihood to reuse service, likelihood to recommend to others, reduced unnecessary travel, missed days from work/school, or delays for next available appointment	Validated questionnaire

3.2. Synthesis of results

All the studies included in this review offered telemedicine services to acutely ill patients. A wide range of acute conditions were managed through telemedicine in the reviewed studies including acute respiratory tract infections [27,28], skin and soft-tissue infections [28], acute rheumatic fever [27], acute pediatric conditions [27], acute exacerbations of COPD and CHF [29-31], acute respiratory distress in patients with amyotrophic lateral sclerosis [32], surgical and traumatic wounds [33], medical emergencies in elderly [21], mental health emergencies in children [34], and post-surgical care [29].

The included studies reported providing telemedicine services in urban and rural [26,35] health-care settings. One interesting study reported achieving telemedicine presence using a robot in a distant rural underserved area [35]. In addition, the included studies provided telemedicine services to pediatric [27,34], adult [28-32], and elder populations [21]. This diversity is important as it proves that telemedicine can be implemented in different healthcare settings and adopted by diverse patient populations.

The services provided through telemedicine included remote consultation for diagnosis, management plan, decisions on referrals, admission to a hospital or discharge, and remote monitoring for follow-up.

The most frequently used modality of telemedicine was videoconference as it was used in eleven out of 12 studies [27-31] [21,31-37]. In ten studies, videoconference was used to connect patients with health-care providers, and in one study, it was used to connect paramedics to emergency physicians [37]. Only one study used telephone calls instead of videoconference [32]. In that study, patients with amyotrophic lateral sclerosis used phone calls to report symptoms of acute respiratory distress to health-care professionals and based on the call medical assistance was offered to the patients at their homes (Figure 2).

Figure 2. (1) Patients with amyotrophic lateral sclerosis suffering acute respiratory distress would request help through a phone call. (2) A specialized team heads to the patient home. (3) Respiratory assistance including assisted ventilation is provided at patient’s home. Created with Biorender.com.

The fact that videoconferencing is the most frequently utilized modality in the included studies is not surprising. Videoconferencing is very convenient in terms of being an effective alternative to the traditional face-to-face appointments. This advantage becomes particularly useful in delivering health care to distant communities or when social distancing is preferred as we have witnessed during the SARS-CoV-2 pandemic [11].

Telemedicine visits took place in patients’ homes in seven studies. In the other five studies, a neighborhood service center [27], a clinic [36], a hospital [34], a senior living facility [21], and a rural clinic run by nurses [35] hosted the service.

Patients were offered in-person visits or other in-person medical services in six studies [29-33,37]. These services include home visits by health-care professionals, laboratory testing, imaging studies, oxygen therapy, intravenous (IV) fluids, and wound dressing [29-33,37].

Eleven out of 12 studies reported evaluation of patient satisfaction using questionnaires [21,27-32,34-37], one study did not report the modality used for evaluation [33] (Table 2). Out of the 11 studies, nine studies used self-completed surveys for evaluation, while two studies used interview questionnaires [32]. Seven studies out of 12 evaluated provider satisfaction, six studies reported using questionnaires while one study did not report the modality used [33].

Most studies reviewed used telephones, mobile phones, computers, and tablets to connect patients to care providers. One study used a home monitoring station to allow virtual visits as well as wireless devices for biometric measures as blood pressure and oxygen saturation [31]. Another study described the use of portable telemedicine units which included peripheral devices, connected to a laptop computer that enabled acquisition of high-resolution images of tympanic membranes, eyes, throat, or skin as well as audio files of lung sounds [27]. An electronic stethoscope, a digital otoscope, a high-resolution camera, and a web camera were also used to obtain patients’ vitals and other diagnostic information and send them to care providers [21]. An interesting study reported the use of a robot to achieve tele-presence in a rural community; the robot was successfully used for diagnosing and monitoring the patients (Figure 3).

Figure 3. In one program, patients from a rural area received telemedicine services in a rural clinic run by nurses. The virtual visits were carried out through a robot. Created with Biorender.com.

Technology and communication were the most frequently evaluated attributes in the included studies. In general, technology and communication were very well-received by patients and care providers (>90% in all studies). The good impression among patients including elderly [21] holds high promises of more implementation of technology in acute care.

Most patients said that they would reuse and/or recommend the service. One study asked parents/caregivers of children who received telemedicine service if they would consider telemedicine for their own care and the responses were highly positive (78% agreed) [27]. Caregivers and family members also expressed their satisfaction with telemedicine as it reduced travel time and days missed from work [31].

Convenience is a major determinant of quality from the patient’s perspective. Convenience was tested in two studies and generally received an excellent feedback (94.5% and 95%) [27,36]. In one study, convenience was the most important factor in deciding to use telemedicine (85% of patients said that it was very important while only 0.8% said that it was not at all important) [27]. In the same study, satisfaction with convenience was a strong predictor of being satisfied with telemedicine as those who were satisfied with convenience had more than 2.3 times the odds of liking telehealth [27].

Although privacy, safety, and staff training were less frequently evaluated, they received positive feedback in general (>90% in both studies) [28,31]. Telemedicine was also effective in reducing cost, unnecessary travel, missed days from work or school, and delays to next appointment [31,34]. Overall satisfaction was the sole measure of patient satisfaction in two studies [33,37] and was tested among other aspects in most of the studies. The overall satisfaction was high in all except one study [37].

3.3. Quality and risk of bias assessment

The quality and risk of bias assessment is provided in details in the Supplementary File. In general, three studies were of poor quality, five studies were of fair quality, and the rest of the studies were of good quality.

Potential sources of bias included restriction to a certain area (geographically limited) [27,28,30], limited generalizability [37], age bias [29], gender bias [21,28,30], non-randomization [29], small sample size [30], and interviewer bias [21,32].

4. Discussion

Satisfaction with a health-care service – and probably any service – is a combination of expectations and actual experiences [38]. In other words, to achieve consumer’s satisfaction, the service offered to a patient should meet his expectations.

Expectations are beliefs, created, and sustained by a cognitive process; these expectations, however, determine satisfaction which is an affective state [39]. Patients’ expectations can be divided into four distinct types: Ideal, predicted, normative, and unformed. Ideal expectations, as the name implies, refer to an idealistic state of beliefs; in other words, the patient’s imagination of how a perfect service should look like [39,40]. Predicted expectations are simply what the patient is expecting to happen in the real world, and it is based on a source of knowledge which can be the patient’s own previous experiences, a family member, or a friend experience or even the media [39]. Normative expectations are situational, they imply what the patient think should happen while receiving the service. Finally, unformed expectations refer to a state of inability to form any thoughts about the health-care service to be received; patients could be too afraid or anxious to formulate an expectation. In this state, patients would perceive any service as accepted regardless of quality [40]. These expectations are not constant, they evolve and interact with each other as the patient’s experience goes on which make them very difficult to predict.

While it seems impossible to predict or control patients’ expectations, it looks more plausible to identify the attributes of those expectations. In theory, if we identify the attributes that form patients’ expectation, we could fulfill these attributes to achieve the highest satisfaction with the service. Mahon [41] was one of the first researchers to try to identify these attributes. She reported eight attributes: The art of care, technical quality of care, accessibility, finances, physical/organizational environment, availability of providers and resources, continuity of care, and efficacy. Janet et al. [5] identified provider attitude, technical competence, accessibility, and efficacy as the major attributes of patient satisfaction.

Based on the aforementioned factors, to achieve high levels of satisfaction, the proposed health-care service should be humane, technically competent, easily accessible, and affordable. It should provide patients with best possible outcomes, guarantee continuity of care, facilitate delivering service to their homes with minimal bureaucracy, or organizational complications [42]. Telemedicine has the potential to improve most of these aspects and, thus, provide patients with better overall experience and higher levels of satisfaction than the usual hospital setting [5].

Designing a valid questionnaire is not easy; even the simplest surveys require trained personnel, generous resources, and ample time [6]. This fact is reflected in the included studies where most of the included studies utilized non-validated tools to assess their outcome which is a major drawback (Table 2). Self-completed surveys and interview surveys are the most commonly used surveys in health-care sector, and they were the most commonly utilized in this review (Table 2). Self-completed surveys are generally preferred as they guarantee standardization of items among patients [6]. They are also less liable to interviewer bias and could be conducted at a significantly lower cost [6]. Despite this fact, two of the included studies depended on interview questionnaires to evaluate patient satisfaction [21,32]. In general, a well-designed survey should evaluate as many patient satisfaction attributes as possible and provide the decision maker with information that are easy to interpret and build actions on [43].

Telemedicine is more frequently implemented in care of patients with chronic medical conditions, usually for the purpose of remote monitoring and follow-up. In these settings, telemedicine is frequently integrated with home care services to provide chronic patients with hospital care level at home. These models have shown great success in managing complex clinical scenarios as heart failure and debilitating neurological diseases while lowering cost and hospitalization rates [2,3]. These programs are usually well-received by patients and achieve high levels of patient satisfaction [4]. However, using telemedicine in delivering acute care is fundamentally different and patient satisfaction in such settings should be approached differently and distinguished from patient satisfaction in the setting of chronic disease.

Telemedicine has the potential to be the next breakthrough in acute medicine as it extends the boundaries of the practice across distances. In our analysis, telemedicine was effectively used to manage a broad spectrum of acute medical conditions from pharyngitis to acute exacerbations of COPD and psychiatric emergencies (Figure 4). Telemedicine reduced the cost of medical service (13, 14), readmissions, ED visits [31], unnecessary travel [31], missed days from work and school [34], and unnecessary referral and hospitalization. Telemedicine provided patients with acute care at homes, senior living facilities [28], rural community clinics [35], and geographically isolated areas [28]. It has improved outcomes, communication with the medical staff, and the quality of service.

Figure 4. Pediatric patients with psychiatric emergencies presenting in the emergency department were offered either a regular face to face visit or a virtual visit. Created with Biorender.com.

The studies included used different modalities to evaluate patient satisfaction. Most studies used self-completed questionnaires (Table 2) which are superior to interview questionnaires that were used in two studies. Technology, communication, likelihood to recommend/reuse, and overall satisfaction were the most frequently evaluated attributes of telemedicine. Most patients embraced telemedicine services and reported high levels of satisfaction.

Provider satisfaction was also evaluated in seven studies and similar attributes were measured. Physicians, nurses, paramedics, and technicians reported high levels of satisfaction with telemedicine. Most of the providers expressed their willingness to reparticipate in a telemedicine service. However, some concerns about the inability to physically examine the patient were reported [28]. In addition, some technical issues like the quality of the images or the weight of the equipment were reported [21]. Despite these concerns, the general trend among the vast majority of providers was very positive.

Despite the promising results of telemedicine in acute care, it still faces significant challenges. One major obstacle is the resistance to change by both providers and patients. This is particularly true for older patients and providers who might face difficulties using modern technology [44,45]. Cost and reimbursement seem to be another challenge to implementing telemedicine. For example, in the United States, Medicare does not reimburse very much in the fee-for-service system, and reimbursement is limited to nonmetropolitan areas, and to certain current procedural terminology codes. Many of these restrictions result from fears that telemedicine either will allow providers to abuse the health-care system or will lead to overutilization and drive up costs [46]. Furthermore, the level of patient education could be problematic, particularly in the developing world as handling technology requires a minimum level of literacy. Moreover, regulations and legalizations might be a significant challenge in some instances. For example, in the United States, some medical boards require an in-person consultation before initiating any telemedicine service [46]. Finally, the lack of evidence on many of telemedicine interventions might delay the implementation of these interventions on a large scale.

We believe this review provides a contribution to the growing evidence on the effectiveness and efficiency of telemedicine. It proves that acute medicine can substantially benefit from the technological advance while achieving high levels of patient satisfaction. To the best of our knowledge, this is the first systematic review to measure this association.

5. Conclusion

Telemedicine is more frequently used in managing patients with chronic medical conditions and was found to achieve high levels of patient satisfaction. This systematic review aims at evaluating patient satisfaction when telemedicine is implemented in the acute care.

Technology, communication, convenience, and safety received highly positive feedback from patients. The overall satisfaction in most of the studies was excellent among patients.

Supplementary File

Database search

PubMed

Search terms: (Patient Satisfaction[MeSH Terms]) AND (telemedicine[MeSH Terms])

2010 – 2021

Results: 1557

CINAHL

Search terms: Telemedicine AND Patient Satisfaction

2010 – 2021

Results: 976

Scopus

Search terms: Telemedicine AND “Patient Satisfaction“

2010 – 2021

Article

Results: 5051

After duplicate removal: 4261

Quality assessment

Hernandez et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?			☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?			☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?	☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure, or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?		☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?	☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?	☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Good
Rater #2: Good

Jakobsen et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the study described as randomized, a randomized trial, a randomized clinical trial, or an RCT?	☑
2. Was the method of randomization adequate (i.e., use of randomly generated assignment)?	☑
3. Was the treatment allocation concealed (so that assignments could not be predicted)?	☑
4. Were study participants and providers blinded to treatment group assignment?	☑
5. Were the people assessing the outcomes blinded to the participants’ group assignments?		☑
6. Were the groups similar at baseline on important characteristics that could affect outcomes (e.g., demographics, risk factors, and comorbid conditions)?	☑
7. Was the overall dropout rate from the study at endpoint 20% or lower of the number allocated to treatment?	☑
8. Was the differential drop-out rate (between treatment groups) at endpoint 15 percentage points or lower?		☑
9. Was there high adherence to the intervention protocols for each treatment group?	☑
10. Were other interventions avoided or similar in the groups (e.g., similar background treatments)?			☑
11. Were outcomes assessed using valid and reliable measures, implemented consistently across all study participants?	☑
12. Did the authors report that the sample size was sufficiently large to be able to detect a difference in the main outcome between groups with at least 80% power?		☑
13. Were outcomes reported or subgroups analyzed pre-specified (i.e., identified before analyses were conducted)?		☑
14. Were all randomized participants analyzed in the group to which they were originally assigned, that is, did they use an intention-to-treat analysis?	☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Fair
Rater #2: Fair

Mashru et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?			☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured before the outcome (s) being measured?		☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?			☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?		☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?			☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?		☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Poor
Rater #2: Poor

McIntosh et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?			☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?		☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?		☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?	☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?		☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?			☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?		☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Poor
Rater #2: Poor

Mendez et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?			☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?		☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?		☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure, or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?			☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?			☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?		☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Poor
Rater #2: Poor

Polinski et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?	☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?	☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?	☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?			☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?	☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?		☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Fair
Rater #2: Fair

Shah et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?	☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?			☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?		☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?		☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure, or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?			☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?			☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?		☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Fair
Rater #2: Fair

Summerfelt et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?	☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured before the outcome (s) being measured?	☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?	☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure, or exposure measured as continuous variable)?			☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?		☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?	☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?	☑

Quality rating (good, fair, and poor)
Rater #1: Fair
Rater #2: Fair

Sykora et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the study described as randomized, a randomized trial, a randomized clinical trial, or an RCT?	☑
2. Was the method of randomization adequate (i.e., use of randomly generated assignment)?	☑
3. Was the treatment allocation concealed (so that assignments could not be predicted)?	☑
4. Were study participants and providers blinded to treatment group assignment?		☑
5. Were the people assessing the outcomes blinded to the participants’ group assignments?		☑
6. Were the groups similar at baseline on important characteristics that could affect outcomes (e.g., demographics, risk factors, and comorbid conditions)?	☑
7. Was the overall dropout rate from the study at endpoint 20% or lower of the number allocated to treatment?	☑
8. Was the differential dropout rate (between treatment groups) at endpoint 15 percentage points or lower?	☑
9. Was there high adherence to the intervention protocols for each treatment group?	☑
10. Were other interventions avoided or similar in the groups (e.g., similar background treatments)?		☑
11. Were outcomes assessed using valid and reliable measures, implemented consistently across all study participants?	☑
12. Did the authors report that the sample size was sufficiently large to be able to detect a difference in the main outcome between groups with at least 80% power?	☑
13. Were outcomes reported or subgroups analyzed pre-specified (i.e., identified before analyses were conducted)?	☑
14. Were all randomized participants analyzed in the group to which they were originally assigned, that is, did they use an intention-to-treat analysis?	☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Good
Rater #2: Good

Teot et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the study described as randomized, a randomized trial, a randomized clinical trial, or an RCT?	☑
2. Was the method of randomization adequate (i.e., use of randomly generated assignment)?	☑
3. Was the treatment allocation concealed (so that assignments could not be predicted)?	☑
4. Were study participants and providers blinded to treatment group assignment?		☑
5. Were the people assessing the outcomes blinded to the participants’ group assignments?		☑
6. Were the groups similar at baseline on important characteristics that could affect outcomes (e.g., demographics, risk factors, and comorbid conditions)?	☑
7. Was the overall dropout rate from the study at endpoint 20% or lower of the number allocated to treatment?	☑
8. Was the differential dropout rate (between treatment groups) at endpoint 15 percentage points or lower?	☑
9. Was there high adherence to the intervention protocols for each treatment group?	☑
10. Were other interventions avoided or similar in the groups (e.g., similar background treatments)?	☑
11. Were outcomes assessed using valid and reliable measures, implemented consistently across all study participants?	☑
12. Did the authors report that the sample size was sufficiently large to be able to detect a difference in the main outcome between groups with at least 80% power?	☑
13. Were outcomes reported or subgroups analyzed pre-specified (i.e., identified before analyses were conducted)?	☑
14. Were all randomized participants analyzed in the group to which they were originally assigned, that is, did they use an intention-to-treat analysis?	☑

^*CD: Cannot determine, NA: Not applicable, NR: Not reported

Quality rating (good, fair, and poor)
Rater #1: Good
Rater #2: Good

Thomas et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the study question or objective clearly stated?	☑
2. Were eligibility/selection criteria for the study population pre-specified and clearly described?	☑
3. Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest?	☑
4. Were all eligible participants that met the pre-specified entry criteria enrolled?	☑
5. Was the sample size sufficiently large to provide confidence in the findings?	☑
6. Was the test/service/intervention clearly described and delivered consistently across the study population?	☑
7. Were the outcome measures pre-specified, clearly defined, valid, reliable, and assessed consistently across all study participants?	☑
8. Were the people assessing the outcomes blinded to the participants’ exposures/interventions?		☑
9. Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis?	☑
10. Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided P values for the pre-to-post changes?		☑
11. Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time series design)?		☑
12. If the intervention was conducted at a group level (e.g., a whole hospital, a community, etc.) did the statistical analysis take into account the use of individual level data to determine effects at the group level?	☑

^*CD, Cannot determine; NA, Not applicable; NR, Not reported

Quality rating (good, fair, and poor)
Rater #1: Good
Rater #2: Good

Vitacca et al.
Criteria	Yes	No	Other (CD, NR, NA)*
1. Was the research question or objective in this paper clearly stated?	☑
2. Was the study population clearly specified and defined?	☑
3. Was the participation rate of eligible persons at least 50%?			☑
4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria for being in the study pre-specified and applied uniformly to all participants?	☑
5. Was a sample size justification, power description, or variance and effect estimates provided?		☑
6. For the analyses in this paper, were the exposure (s) of interest measured prior to the outcome (s) being measured?		☑
7. Was the timeframe sufficient so that one could reasonably expect to see an association between exposure and outcome if it existed?	☑
8. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome (e.g., categories of exposure or exposure measured as continuous variable)?		☑
9. Were the exposure measures (independent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
10. Was the exposure (s) assessed more than once over time?		☑
11. Were the outcome measures (dependent variables) clearly defined, valid, reliable, and implemented consistently across all study participants?	☑
12. Were the outcome assessors blinded to the exposure status of participants?		☑
13. Was loss to follow-up after baseline 20% or less?	☑
14. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure (s) and outcome (s)?	☑

^*CD, Cannot determine; NA, Not applicable; NR, Not reported

Quality rating (good, fair, and poor)
Rater #1: Fair
Rater #2: Fair

Funding

No funding was used to complete this work.

Conflicts of Interest

The authors declare no conflicts of interest.