Skip to main content
PMC home page
Elsevier Sponsored Documents logoLink to Elsevier Sponsored Documents
. 2023 Jul 26;5(8):e534–e544. doi: 10.1016/S2589-7500(23)00092-4

The global effect of digital health technologies on health workers’ competencies and health workplace: an umbrella review of systematic reviews and lexical-based and sentence-based meta-analysis

Israel Júnior Borges do Nascimento a,b,c, Hebatullah Mohamed Abdulazeem d, Lenny Thinagaran Vasanthan e, Edson Zangiacomi Martinez f, Miriane Lucindo Zucoloto f, Lasse Østengaard g,h, Natasha Azzopardi-Muscat c, Tomas Zapata c, David Novillo-Ortiz c,*
PMCID: PMC10397356  PMID: 37507197

Summary

Systematic reviews have quantified the effectiveness, feasibility, acceptability, and cost-effectiveness of digital health technologies (DHTs) used by health-care workers. We aimed to collate available evidence on technologies’ effect on health-care workers’ competencies and performance. We searched the Cochrane Database of Systematic Reviews, Embase, MEDLINE, Epistemonikos, and Scopus for reviews published from database inception to March 1, 2023. Studies assessing the effects of DHTs on the organisational, socioeconomic, clinical, and epidemiological levels within the workplace, and on health-care workers’ performance parameters, were included. Data were extracted and clustered into 25 domains using vote counting based on the direction of effect. The relative frequency of occurrence (RFO) of each domain was estimated using R software. AMSTAR-2 tool was used to appraise the quality of reporting, and the Confidence in the Evidence from Reviews of Qualitative research approach developed by Grading of Recommendations Assessment, Development and Evaluation was used to analyse the certainty of evidence among included studies. The 12 794 screened reviews generated 132 eligible records for assessment. Top-ranked RFO identifiers showed associations of DHT with the enhancement of health-care workers’ performance (10·9% [95% CI 5·3–22·5]), improvement of clinical practice and management (9·8% [3·9–24·2]), and improvement of care delivery and access to care (9·2% [4·1–20·9]). Our overview found that DHTs positively influence the daily practice of health-care workers in various medical specialties. However, poor reporting in crucial domains is widely prevalent in reviews of DHT, hindering our findings’ generalisability and interpretation. Likewise, most of the included reviews reported substantially more data from high-income countries. Improving the reporting of future studies and focusing on low-income and middle-income countries might elucidate and answer current knowledge gaps.

Introduction

Multiple digital health technologies (DHTs) have revolutionised health care by providing an integrated, efficient, and decentralised environment for health-care delivery. DHT has improved patient outcomes and access to health services and professional knowledge,1 and include any information technology targeted towards improving care delivery and enhancing the end-user experience.2, 3 New digital solutions have been introduced annually, including remote sensors and wearables, telemedicine and health information, data analytics and artificial intelligence approaches, evidence-based medical posts on social media, and clinical decision support systems.4, 5, 6, 7 Multiple health-related institutions and stakeholders, including WHO, are promoting the adoption and scale-up of DHT innovations worldwide.8, 9 These promotional initiatives aim to translate scientific research into action and enhance knowledge through scientific engagement assessing and linking geographical needs with innovation pipelines, and implementing practical approaches that balance benefits and risks of DHTs.

Numerous DHTs have outperformed standard care delivery methods, suggesting an improvement in clinically relevant variables and psychological, social, and financial outcomes.10, 11, 12, 13, 14 Nevertheless, endeavours from top-ranked scientific institutions worldwide are not only focused on the effect of DHTs on patients’ perspectives, but also those servicing them.15, 16 Published investigations suggest that these technologies have gained broader acceptance among healthcare workers due to readiness to access information or support from senior specialists, increased communication, lower operational costs, provision of data-driven patient feedback, and increased agility and productivity.17, 18, 19 However, these findings are still controversial, and the conclusive contribution made by DHTs to health-care workers is still undefined.20, 21

Because of the fast-paced creation and implementation of such interventions, several primary studies and systematic reviews have evaluated the cost-effectiveness and the effect of DHTs on health-care workers’ performance and safety, and the perceived usefulness of DHTs. However, even among evidence-based analyses, these reviews show inconsistent effectiveness or cost-related benefits.22, 23, 24 To the best of our knowledge, no previous overview of systematic reviews has been conducted to understand and interpret the effect of DHTs on health-care workers’ competencies and performance metrics, nor to assess the available evidence of DHTs effects on practice and workplace. Therefore, we aimed to collate, analyse, and verify the current evidence of the effect of these digital solutions on health-care workers’ performance and professional competencies and streamlining provider–provider communication, delivery of care, knowledge acquisition, and profitability for all levels of care.

Methods

This project (protocol registered on PROSPERO [CRD42022304372, appendix pp 13–23) followed well known guidelines and methodologies used in evidence-based medicine, including the Cochrane Handbook, PRISMA, the Synthesis Without Meta-analysis, and the Quality of Reporting of Meta-analyses document.25, 26, 27, 28

Data extraction and quality assessment

After shortlisting eligible records, four investigators (IJBdN, HMA, LTV, and LØ) independently extracted information regarding study identification, objective, publication year, journal name, number of included databases, number of included studies, targeted health-care workers, number of workers enrolled, preliminary study designs, technology under investigation, the setting, the disease or condition considered, and our primary outcome—effects of DHTs on health-care workers’ performance and competencies and on health-care workplaces. Instead of tying the methods to a particular epistemological or theoretical perspective, the findings were summarised using a thematic analysis associated with discourse analysis, a method that relies upon identifying patterns or themes within qualitative data that also considers sociocultural and political contexts.29, 30 In this method, major line-by-line codes are generated based on sentences, words, lexical patterns, and similarities of in-text presentation. Subsequently, generated codes are framed in analytical themes (higher order domains), a direct translation of concepts identified in each study that contain key messages from a group of collected results. Concomitantly, the codes are reviewed to consider implications and then discussed in a group to allow for abstract messages and themes to emerge beyond the content in the original materials. Therefore, in our study, during the early data preparation phase, the same four investigators summarised the effect of distinct DHTs on health-care workers’ performance aspects, satisfaction with and acceptance of DHTs, and their influence on multiple variables of the integrated framework defined during study implementation for specific health conditions. We also recorded results of the effects of DHTs on health-care-related processes or workplaces. In the late data preparation phase, four other collaborators independently evaluated the summarised data, appraised the overall content, and, if necessary, restructured the findings, complementing the extracted information. Any disagreements were resolved by discussion. After data preparation, we clustered all conclusions (vote counting based on the direction of effect) of each included review according to their similarity, arrangement, and writing into 25 stratified core outcome domains (all higher level domains). The rationale for grouping findings into core domains created by our group involved principles of discursive resonance, discursive approach to paraphrastic processes, and evaluation of the repetition of lexical items (appendix p 24).31

Concurrently, all studies were quality-appraised independently by IJBdN, HMA, and LTV applying 16 core methodological quality criteria based on the Assessing the Methodological Quality of Systematic Reviews 2 protocol.32 After assessing each domain, we used the AMSTAR 2 platform's algorithm to provide a final rating. The appendix (pp 25–651) includes a detailed judgment for each review.

Statistical analyses and summary of the evidence

We evaluated the occurrence and relevance of the identified core outcome domains using a standard aggregate data meta-analysis for proportions. We obtained the 95% CIs of the relative frequencies of occurrence (RFO) under each domain (25 core domains) using the metaprop function within the R package version 4.1.1. After intervals estimation, we presented the results graphically. Homogeneity among the DHTs was tested using Cochrane's test. The certainty of the evidence was appraised for the top ten most-frequently observed core domains using a modified version of the Confidence in the Evidence from Reviews of Qualitative research approach developed by Grading of Recommendations Assessment, Development and Evaluation. This approach considers the methodological limitations, coherence, data adequacy, and findings’ relevance.33

Results

Search results

Our Jan 21, 2022, search retrieved 9912 records, of which 139 underwent full-text review (figure 1). Based on the inclusion and exclusion criteria, 52 studies were included that directly assessed the effect of DHTs on health-care workers’ routines, and seven ongoing studies were identified. On March 1, 2023, 4150 new publications were identified through an additional database search (figure 1). Of those, 150 studies were shortlisted for full-text assessment, and 80 reviews were added to our umbrella review. Two additional ongoing studies or protocols were identified. In total, this overview of systematic reviews included 132 primary systematic reviews and nine ongoing studies (figure 1). One study required external translation (from Chinese to English). The reference list of included studies is available upon request. The appendix (pp 689–693) includes a list of all full-text excluded studies (n=148) with the justification for exclusion.

Figure 1.

Figure 1

Open in a new tab

PRISMA flow diagram

Reason 1=wrong intervention or platform was unclear. Reason 2=the study did not provide any relevant outcome influencing health-care workers. Reason 3=targeted population was not health-care workers. Reason 4=study design used did not match our inclusion criteria. *Studies can be accessed upon request.

Characteristics of included reviews

Most records included quantitative (eg, randomised clinical trials, observational studies, and economic analysis), qualitative (eg, case study research, public online surveys, and ethnographic research), or mixed method studies. Studies were published between 1998 (n=1) and 2023 (n=15), predominantly in the Journal of Medical Informatic Research (n=12), the Journal of the American Medical Informatics Association (n=6), Cochrane Database of Systematic Reviews (n=6), Implementation Science (n=6), Telemedicine and eHealth (n=5), and BMJ Open (n=5). Overall, approximately 4000 primary publications were considered about health-care workers, including physicians, frontline and community health-care workers, nurses, pharmacists, dentists, and physiotherapists, from a variety of medical specialties.

The appendix (pp 26–57) summarises the characteristics of included studies by intervention type and primary disease or condition evaluated. Sytematic reviews and scoping reviews focused on mental health disorders and syndromes, maternal and gynaecological health, infectious diseases, cardiovascular pathologies, and oral diseases. Most systematic reviews described, calculated, and displayed the occupations of health-care workers’ enrolled in each primary study, totalling approximately 298 000 individuals. Geographically, reviews involved professionals located in Africa (n=32), Europe (n=80), North America (n=80), Latin America (n=37), Asia (n=71), and Australia and New Zealand (n=56).

Quality appraisal

The appendix (pp 50–56) shows the reliability of included reviews based on the AMSTAR-2 criteria and the methodological quality summary. Five reviews (4·0%) achieved the maximum quality of reporting, 17 (12·8%) were rated as low-quality of reporting, and 108 (81·8%) reviews were rated as critically low quality of reporting. Population, intervention, comparison, and outcome components related to the research question and inclusion criteria were adequately stated in all reviews, which was a benchmark prioritised in our inclusion criteria. Only 35 reviews (28·4%) explicitly stated that the review methods were established before the commencement of the review or detailed any protocol deviation (critical item 2). Similarly, predominant inappropriate reporting of one non-critical item (item 10), and two critical domains (items 7 and 13) was observed among included reviews. None of the critical domains had a predominantly positive observation among included reviews. Conversely, five non-critical domains were usually correctly stated among included reviews, such as review authors explanation of their selection of the study designs for inclusion in the review (item 3), dual study selection (item 5) and extraction (item 6), sufficient description of included studies (item 8), and disclosure of conflicts of interest (item 16). A meta-analysis, used as the primary statistical combination technique, was executed only in eight (6·5%) reviews. All meta-analyses satisfactorily explained heterogeneity observed in the results. In-depth justification for each quality assessment judgement is available in the appendix (pp 59–681).

Summary of effects of digital health technologies on health-care workers’ associated characteristics and metrics and on health workplace

We extracted, classified, and evaluated data from several different DHTs, as shown in figure 2. Considering all categorised modalities of DHTs, the enhancement of health-care workers’ technical performances was the most common effect described among included reviews (10·9% [95% CI 5·3–22·5], moderate confidence). This enhancement of health-care professionals’ technical performance was reported as increased accuracy and efficacy during decision-making processes commonly faced in clinical practice; reduced time needed to execute tasks; improved worker productivity; faster and more reliable real-time access to data; increased knowledge acquisition; or more timely technical and specialised reports, comprising activities, progressions, and remedies. In addition, improved clinical practice and management (9·8% [95% CI 3·9–24·2], moderate confidence); improved care delivery and access to care (9·2% [4·1–20·9], moderate confidence); high satisfaction perceived among health-care workers using such technologies (9·0% [4·2–19·4], low confidence); and worker training, recruitment, and continuous education (7·1% [3·8–13·4], moderate confidence) made up the ranking of the five most-frequently reported effects associated with the use of DHTs. Enhanced interprofessional communication (decrease in communication gaps, enhanced shared information quality, improved information and knowledge exchange, strengthened quality and efficiency of communication, enhanced accessibility and inter-team relationships, and increased professional networks) had an RFO of 6·2% ([2·1–18·3], moderate confidence). Less frequently reported were domains associated with improved diagnosis and screening processes (4·4% [1·8–10·7], moderate confidence); improved health-care workers’ skills, motivation, empowerment, self-confidence, self-accuracy, and improved personal competencies (4·8% [2·0–11·4], moderate evidence); decreased indirect or direct costs of processes (eg, travel-related costs; 4·6% [1·6–13·3], very low confidence); and improved longitudinal follow-up of patients and better patient monitoring (3·3% [1·3–8·2], moderate confidence). The appendix (pp 57–58) shows the summary of findings from the ten most-reported domains with respective RFOs and overall quality of evidence. Based on the heterogeneity assessment across the generated domains, we found significant heterogeneity (p<0·01) for seven outcomes.

Figure 2.

Figure 2

Open in a new tab

Relative frequency of occurrence of higher level domains, by major modality of digital health technologies

AI=artificial intelligence. CDSS=clinical decision support system. EMR=electronic medical record.

Telemedicine, telehealth, telemonitoring, and remote monitoring technologies (n=46)

46 (34·8%) reviews provided data associating the use of telemedicine, telehealth, telemonitoring, and remote monitoring technologies with outcomes related to health-care workers. Numerous reviews suggested that the use of these technologies in several medical specialties improves decision-making processes (30·3% [95% CI 21·0–41·0]), enhances the delivery of care (27·0% [18·1–37·4]), and is associated with high satisfaction rates among health-care workers (29·2% [20·1–39·8]). In addition, these technologies enable faster identification of new clinical issues (11·2% [5·5–19·7]). Reviews also evidenced that telemedicine might further career advancement, expand knowledge, and enhance efficiency and performance metrics (23·6% [15·2–33·8]); be used in therapeutic interventions (11·2% [5·5–19·7]); and facilitate health-care workers’ training and recruitment (15·7% [8·9–25·0]). We also recognised that telemedicine might increase health-care worker burden and burnout (2·2% [0·3–7·9]). According to Gagnon and colleagues,34 telemedicine and remote monitoring technologies might also contribute to occupational therapists’ willingness to practise in rural or remote regions. For cost-effectiveness, 14 (31·8%) reviews evidenced beneficial effects in reducing overall public health expenditures and financing basic health services (16·1% [9·1–25·5]). Interprofessional communication and collaboration between different health-care professions improved when using telemedicine (27·0% [18·1–37·4]). In one review, instant messaging applications appeared to be an adjunctive tool for telemedicine due to portability, speed, simplicity, and the ability to provide constant and easy updates.

Mobile health technologies (n=34)

The effect of mobile health technologies on health-care workers’ environments was evaluated in 34 reviews (25·7%). Figure 2 highlights the relative occurrence in 25 core domains observed among the 34 reviews. The highest RFO was 29·3% (95% CI 19·4–41·0), which suggested a relationship between mobile health technologies and strengthened interprofessional communication, information and knowledge exchange, and enhanced accessibility and inter-team relationships. Furthermore, mobile health technologies were associated with improved care delivery, access to care, and quality of care (24·0% [14·9–35·3]), and enhanced technical performance, accuracy, and efficacy of workers (28·0% [18·2–39·6]). Mobile technologies were associated with high satisfaction rates among general users (RFO of 20·0% [11·6–30·8]), and with improved clinical management and practice, more rapid responses to health-related issues or emergencies, and facilitation of decision making (22·7% [13·8–33·8]). Other core domains showed relationships between mobile health technologies and improved worker skills, self-confidence, and personal competencies; improved data collection and acquisition (14·7% [7·6–24·7]); improved adherence to treatment regimens and protocols and enhanced information provision about treatment (13·3% [6·6–23·2]); and better utilisation of emergency referral systems (13·3% [6·6–23·2]). Figure 2 shows the full frequency of occurrence of each registered domain.

Electronic medical records or electronic health systems (n=23)

23 reviews (17·4%) provided data on the effect of electronic medical records or any documentation system on health-care workers’ practice. Electronic health systems and clinical information systems have been shown to positively enhance multiple metrics related to health-care workers, including clinical management and practice (16·9% [95% CI 9·0–27·7]), data collection and documentation, expedited registration of patient-relevant data, decreased documentation errors, improved access to quality information (12·7% [6·0–22·7]), and professional performance at work (14·1% [7·0–24·4]). Health-care workers identified that interprofessional communication also improved following the implementation of electronic documenting systems (9·9% [4·1–19·3]) during and between shifts due to a better coordinated system in which all workers can be easily reached. Overall, the acceptance and satisfaction of electronic documentation systems were elevated, with high satisfaction among health-care workers (12·7% [6·0–22·7]), mostly because of reduced cognitive load, improved quality of care, and enhanced patient safety. Remarkably, mental health improved when electronic medical records were connected to workflow redesign and quality improvement initiatives, addressing physicians’ insecurities and concerns by reducing documentation and task time and expanding care teams. Despite this, electronic health systems were also associated with increased workload and documentation time (9·9% [4·1–19·3]). One review suggested that usability scores depend on the specialty—general internal medicine physicians tend to rate electronic medical records’ usability higher than family medicine physicians, radiologists, and surgeons. Additionally, Fiander and colleagues35 reported that when practitioners are trained on systems, use increases.

Clinical decision support systems (n=20)

Clinical decision support systems were analysed in 20 reviews in clinical and non-clinical activities. Similar to other technologies, computerised clinical decision support systems considerably improved and optimised clinical behaviour, performance, and professional knowledge and beliefs, along with improved care delivery (15·5% [95% CI 8·0–26·0]). Moreover, integrating clinical decision support systems with health-care processes empowered workers when making decisions (16·9% [9·0–27·7]) and increased the adherence to guidelines and clinical protocols (15·5% [8·0–26·0]). According to Adepoju and colleagues,36 although evidence indicates a substantial improvement in performance, existing evidence does not support their ability to improve the quality of care or clinical outcomes in sub-Saharan Africa. Additionally, conflicts might occur in clinical decision making when the expert knowledge of health workers conflicts with the recommendations of the system. The cost-effectiveness of clinical decision support systems was considered positive in four reviews (12·7% [6·0–22·7]), due to reduced costs in managing diseases and laboratory and radiology assessments. Nevertheless, cost-effectiveness was deemed inconclusive in two reviews and geographically dependent in two other reviews. Included reviews catalogued the core applicability of clinical decision support systems in the routines of health-care workers, enhancing the rationing involved in emergency referrals (1·4% [0·0–7·6]) and diagnosis (9·9% [4·1–19·3]), predicting patients’ prognosis (2·8% [0·3–9·8]), and supporting research initiatives (2·8% [0·3–9·8]).

Artificial intelligence systems (n=8)

The effect of artificial intelligence technologies on health-care workers was assessed in eight reviews. These reviews showed the effect of a spectrum of artificial intelligence technologies, such as digital phenotyping, chatbots, natural language processing and deep learning, and machine learning prediction models. The most prominent outcomes were improved care delivery and enhanced clinical practice through better history taking (RFO of 8·8% [95% CI 3·3–18·2]), improved professional communication (8·8% [3·3–18·2]), and high acceptance and satisfaction rates of such technology (10·3% [4·2–20·1]). Multiple objective and subjective measures were reported as indicators of such enhancements, such as clinical indicators, interviews, surveys, focus groups, and consensus-building approaches. Čartolovni and colleagues,37 for example, depicted a graphical representation of the interplay of ethical, legal, and social considerations that showed various health-care workers’ responses to surveys. Another study reported the increase of disease control and prevention (1·5% [0·0–7·9]), where technology serves as an effective infosurveillance and infoepidemiology tool. Moreover, Boonstra and colleagues38 reviewed the artificial intelligence-based clinical decision-making tools in emergency health-care settings and reported potential positive improvement of clinical practice and reduction of workload.

Miscellaneous

Although some DHT modalities were emphasised less, findings also showed their effectiveness and impact. For instance, clinical reminders and alert systems integrated into health-care workers’ practices were evaluated in four reviews. In five core domains, these systems were mentioned in some aspect (1·7% [95% CI 0·0–8·9]) and showed their potential effectiveness (improved diagnosis, delivery of care, prescription-related outcomes, health-care workers’ performance, and adherence to guidelines). Regarding laboratory reporting systems, a general evaluation showed that the method has been accepted and widely used by its intended clinical clientele. Social media platforms were evaluated in eight reviews. Social media improved health-care workers’ performance (6·2% [95% CI 1·7–15·2]) and interprofessional communication (4·7% [1·0–13·1]), and facilitated training and recruitment of workers (9·4% [3·5–19·3]). These technologies improved clinical practice and management; enhanced health-care workers’ mental health (4·7% [1·0–13·1]); and improved workers’ skills, motivation, and empowerment (1·6% [0·0–8·4]).

Discussion

Findings from this overview and lexical-based and sentence-based meta-analysis show that DHTs are associated with improving health-care workers’ performance benchmarks, including delivery of care, diagnosis, clinical management, prescription-related practice, patient follow-up, and data management. Moreover, these technologies enhance interprofessional communication, compliance with standard clinical protocols, health-care workers’ skills and personal competencies, and health-care workers mental health outcomes. Although we included many systematic reviews, their unsatisfactory reporting hinders the extrapolation of findings from particular settings to broader professional environments.

Health technologies reportedly increase the performance of health-care workers, yet defining and quantifying work performance is still challenging. In our included studies, performance was commonly correlated to optimised communication skills; reliable and rapid access to data; development of professional expertise and skills; increased productivity, efficacy, and accuracy; improvement in quality of care; decreased time commitment to professional activities; and advanced knowledge acquisition. However, evaluating and establishing work performance metrics and tools to measure them are still difficult in health care. The absence of standardised performance evaluation scores and nomenclature limits the understanding of the effectiveness of DHTs and makes the clustering and classification of findings less straightforward, less transparent, and based on smaller amounts of high-quality homogenous data. Moreover, DHTs are often introduced in health-care facilities without pre-analysis, intra-analysis, and post-analysis of workers’ satisfaction; consideration of corporate culture; thorough testing for effectiveness or cost-effectiveness; or solid strategic planning. Consequently, substantial amounts of resources are invested and lost without properly evaluating their positive effect on the health-care workers and evaluating patients’ outcomes and efficiency gains.39 Therefore, health employees, medical and health companies, policy makers, and stakeholders involved in assessing success and productivity should mobilise efforts to create a standardised tool to measure job performance, weighting work quality and quantity, efficiency, and organisational-level metrics. Furthermore, balancing the job satisfaction and motivation of the health-care worker might prove crucial for increasing the accuracy of this scale.

Our overview did not catalogue and analyse barriers to and facilitators for the dissemination, creation, implementation, and follow-up of multiple DHTs. Nevertheless, the literature suggests that several components affect the success or failure of a particular intervention.34, 40, 41 Infrastructure and technical barriers; personal and psychological issues; legal and ethical variables; and cultural, social, and political factors need to be addressed to promote outreach to health-care workers to encourage them to use health technologies in their professional routines.34, 40, 41 In contrast, involving health-care workers in the development and implementation of DHTs, the availability of government and multisectoral incentives, and willingness and perceived use of DHTs by health workers, seem to actively enable consistent and positive use of these technologies.34, 40, 42 The availability or lack of training and updated educational programmes either facilitates the adoption of digital solutions or reduces wide use and perception of usefulness.43, 44, 45, 46, 47

Although most readers might perceive our study as a representation of the effects of DHTs on health-care workers from a global perspective, attention should be paid when evaluating our results. We did not identify the original setting from each primary study indirectly included in our study (this is not the goal of an umbrella review), so we could not properly classify the most frequently evaluated countries and regions associated with our research question. However, our team noted that most included reviews reported substantially more data from high-income countries. Therefore, effectiveness, safety, and feasibility data are insufficient for most regions worldwide, especially from low-income and middle-income countries, which frequently have staff shortages, high prevalence of infectious and non-communicable diseases, and restricted access to medical facilities.48, 49, 50, 51 In addition, Braa and colleagues52, 53 observed challenges faced by low-income and middle-income countries (eg, fragmentation of health-care systems and associated processes, multiple data sources, and lack of standards for data labelling and processing), which should be considered when preparing strategies to improve health information systems and information structures.52 Likewise, considering national diversity within these nations (based on differences in health services and basic infrastructure) is highly relevant.52, 53 However, when these idiosyncrasies are addressed or alleviated, as evidenced by some reviews, the effect of DHTs is impressive, particularly in remote and rural regions, emphasising the value of future research in these regions.36, 43, 44, 48, 54, 55

Although not explicitly discussed in the included reviews, our results might be affected by publication bias, as observation of negative effects linked to the use of health technologies were not reported. Typical reasons for observing publication bias include fewer published primary studies (leading to non-relevant intervention effects), unreliability, and subjectivity during the peer-reviewing process.56, 57 Additionally, reporting bias might have influenced our findings along with external funding and commercial interests. Thus, although overviews of systematic reviews outrank all study designs in terms of certainty of the evidence, our findings should be interpreted deliberatively by health-care workers and decision makers.

Harmful or unintended consequences caused by emerging health technologies were vaguely identified in the included reviews. However, considering negative outcomes through the design, implementation, and longitudinal technology assessment is essential for the digital health scientific progress and its self-correcting nature. At the organisational level, disruption of workflows, changing process dynamics, changes in the power structure, and hierarchical relationships (other than strategic repositioning) are central factors requiring the attention of developers and supporters of DHTs.58, 59, 60 Regarding personal and cognitive-related negative consequences, the potential dematerialisation and depersonalisation of relationships and human interactions and escalated stress-related emotions during implementation phases because of inexperience and scepticism play a substantial role in the rate of adoption of DHTs. Clinical and professional factors and economic, social, legal, and ethical consequences should also be addressed by stakeholders dealing with telemedicine, electronic medical records, or any other DHT, as they affect not only the overall job performance and satisfaction but also the quality of care and patient contentment.

Our results clearly showed unequal evidence regarding multiple types of DHT. Electronic medical records, telemedicine and correlated technologies, and mobile health technologies have historically been used in a disproportionate number of publications compared with clinical alert reminders, alert systems, laboratory reporting platforms, and social media. Thus, the complete, holistic, and integrated evaluation of the effects of these less prevalent health technologies designed for health workers is yet to be seen, while the consolidation of the most common DHTs perpetuates their reputations as reliable and effective mechanisms for improvement of health workers-related performance and overall enhancement of public health-care conditions. However, two actions should be initiated following on from these observations. First, we urgently need high-quality studies analysing the effect, effectiveness, and safety of these neglected technologies. As the global economy and public health status plunge deeper into a financial crisis and health-care programmes deteriorate, innovative research decisions must cover lesser investigated technologies to discover opportunities and improve existing systems and professional frameworks. Likewise, innovative and unprecedented study designs involving the use of telemedicine, telehealth, electronic medical records, and mobile technologies could also facilitate continued progress in digital medicine, labour medicine, and evidence-based medicine. Therefore, a call to action is required to maintain the progressive and functional evolution of DHTs’ effects on health workers’ lives.

In this Review, we applied a well known summarising technique from linguistics and discourse analysis, which presents relevant strengths.61, 62 Indeed, synthesising qualitative data is challenging, but by using a thematic-based analysis we believe that we successfully examined the perspectives of researchers across the globe, highlighted similarities and differences in reporting, and generated unanticipated insights.61 Considering our large set of included reviews, we strictly adhered to our methodological criteria, which consisted of data identification, analysis, organisation, description, and reporting performed by four independent researchers. However, some limitations exist. Subject bias (a confounding component originated by individuals responsible for the research) may have surfaced during the data summarisation process, but this was minimised as we formed a research team with multidisciplinary skills, different linguistic and sociocultural backgrounds, and from contrasting geopolitical settings. Secondly, thematic analysis studies are affected by their inherent theoretical freedom (allowing a range of analytic options), which might initially impede researchers deciding what aspects of data to focus on and might generate a risk of reporting inconsistency. However, we systematically applied our approach to concept identification, in shared decision making. Moreover, we weighted and limited the generalisability of results within the primary studies included in our review. Therefore, future studies that focus on a strategic set of countries might be required for better triangulation of findings. Lastly, overlapping themes can occur during the code creation process in thematic analysis, hindering the coherence and consistency of results. Nevertheless, our study focused on identifying main ideas or concepts, taking care not to misrepresent generated higher-level domains as an overarching theme.

In our evaluation, our assessment showed significant heterogeneity (p<0·01) among some reported domains. Explanations might include the aggregation of data from different types of health-care workers, the publication year, the geographical location of the assessed sample or the manuscript writer and their understanding of sentence structures, or even how the technologies were tailored or implemented. However, further studies might investigate these characteristics to draw proper inferences and conclusions.

Our overview integrates and evidences the best research evidence available regarding the effectiveness, feasibility, and acceptance of DHTs for health-care workers. We showed that even though included reviews evidenced the beneficial effect of multiple DHTs, data from low-income and middle-income countries are mostly absent. A further limitation worth mentioning is that some DHTs might require long intervals of time post implementation before the health-care institution can measure a benefit. Therefore, the effect of a DHT on an outcome might vary if comparing a shorter assessment period with a longer assessment period. Since several systematic reviews included in our study might have chosen to assign equal weighting across primary studies with multiple measurements periods, this could preclude the ability to distinguish between short-term and long-term effects on health workers’ metrics. Additionally, even for high-income nations, high-quality scientific data are still difficult to interpret, hampered by low-quality methodological reporting and vague measurements of professional performance. Technology use in the health-care sector tends to evolve intensely due to rapid technological advancements in software development, operating systems, and improvement of professional competencies. Therefore, it is important to understand how emerging, highly advanced systems will synergistically be incorporated into workflows, and how to assess their applicability for a particular health condition or disease. This process of understanding the emerging technologies should be assessed through high-quality epidemiological, clinical, and socioeconomic research, combining data from diverse predefined settings that use standardised nomenclature, definitions, and measures for primary and secondary outcomes.

Conclusion

We have found moderate evidence that DHTs positively affect health-care workers’ performance, mental health, clinical management, and features associated with delivery and access to care. Furthermore, a very low certainty of evidence exists that DHTs decrease direct and indirect costs, which consequently might reduce government or private sector-related spending. Based on included reviews, studies sharing concepts between medicine, information technology, and DHTs currently do not have essential methodological components, substantially decreasing the certainty of the evidence of our findings. Therefore, well structured studies evaluating health-care workers and clinically related outcomes are still needed.

Search strategy and selection criteria

Initially, we searched the Cochrane Database of Systematic Reviews, Embase, MEDLINE, Epistemonikos, and Scopus for publications published from database inception to Jan 21, 2022, and hand-searched references across the shortlisted reviews. In addition, we searched PROSPERO for ongoing research associated with digital health technologies’ effect on the healthcare worker environment. However, considering the prodigious scientific output in digital medicine commencing in 2022, we re-executed our search on March 1, 2023. Our search term definitions and thesaurus (appendix pp 5–12) were based on recent studies.29, 30, 31, 32 Unpublished data were requested, when pertinent, by emailing the corresponding author or through ResearchGate messages. An expert librarian helped develop our search strategy.

We followed the published definition of systematic reviews: a review study adhering to evidence-based medicine-related guidelines, which searches at least two databases for its systematic analysis.27, 33 Moreover, the study should describe the search strategy and present transparent methods for study selection and data extraction. We considered including scoping reviews, depending on the existence and report of adequate methodological characteristics. Therefore, two investigators (IJBdN and HMA) independently included studies that precisely analysed the literature associated with digital health technology use by health-care workers, evaluating the effects of these technologies in the health-care workplace (our primary outcome as reported in our protocol), regardless of the outcome assessed and the publication design of the primary studies. Therefore, based on Cochrane's population, intervention, control, and outcomes framework, studies were deemed eligible when enrolling health-care providers from multiple levels of care (participants), using several digital health technologies (interventions), in the existence of a control group (comparator), and outcome studies were shortlisted in the standard Cochrane methodology (abstract and title screening followed by full text assessment). However, the general effects we report were part of a careful, group-based decision analysis to guarantee holistic and integrative reporting of the existing literature. Inclusion was not restricted because of the existence of a comparator or control group from included studies (ie, studies without a control group were included), nor published language, medical field, or publication date. Furthermore, eligible studies needed to exclusively include formal health-care workers in their analyses. We excluded preprints, unpublished data, narrative and literature reviews, and reviews targeting informal health-care workers or individuals in training (eg, students).

In our registered protocol, we planned to search the first 100 records from Google Scholar as a supplement.34 However, results differed based on the user's geographical location and previous searching patterns, so we decided not to analyse Google Scholar, as reproducibility would be hindered. No other protocol deviation was executed.

Declaration of interests

DN-O, NA-M, and TZ are staff members of WHO. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the decisions, policy, or views of WHO. All other authors declare no competing interests.

Acknowledgments

Acknowledgments

We are highly grateful for the research collaborators involved in the data extraction phase of this project, including Kusum Singal, Anirudha Agnihoty, Muhammad Ayyan, and Atiya Karim. We thank Anneliese Arno for granting complimentary access to Covidence. We thank Érika Amâncio for supporting and providing insights on the thematic analysis assessment.

Contributors

IJBdN and DN-O designed, conceptualised, and defined the methods in the study. IJBdN, HMA, LTV, and LØ did data curation, visualisation, and data validation. EZM and MLZ did formal data analysis and performed software-based assessments. IJBdN, EZM, MLZ, HMA, LTV, NA-M, TZ, and DN-O drafted the manuscript and its final version. IJBdN and DN-O supervised the study. All authors contributed to the article and approved the submitted version. All authors had access to all the data. IJBdN, HMA, LØ, and LTV have verified the data.

Supplementary Material

Supplementary appendix
mmc1.pdf (5.5MB, pdf)

References

  • 1.Ronquillo Y, Meyers A, Korvek SJ. StatPearls Publishing; Treasure Island, FL: 2022. Digital health. [PubMed] [Google Scholar]
  • 2.Sheikh A, Anderson M, Albala S, et al. Health information technology and digital innovation for national learning health and care systems. Lancet Digit Health. 2021;3:e383–e396. doi: 10.1016/S2589-7500(21)00005-4. [DOI] [PubMed] [Google Scholar]
  • 3.Fatehi F, Samadbeik M, Kazemi A. What is digital health? Review of definitions. Stud Health Technol Inform. 2020;275:67–71. doi: 10.3233/SHTI200696. [DOI] [PubMed] [Google Scholar]
  • 4.Lustig TA. In: The role of telehealth in an evolving health care environment. Lustig TA, editor. National Academies Press; Washington, DC: 2012. The evolution of telehealth: where have we been and where are we going? [PubMed] [Google Scholar]
  • 5.Darkins AW, Cary MA. Free Association Books; London: 2000. Telemedicine and telehealth: principles, policies, performance, and pitfalls. [Google Scholar]
  • 6.Tissington P, Senior C. Use of social media to enable medical teams to work virtually (and better) Lancet. 2013;381:e11. doi: 10.1016/S0140-6736(13)60935-6. [DOI] [PubMed] [Google Scholar]
  • 7.Palmer KR, Tanner M, Davies-Tuck M, et al. Widespread implementation of a low-cost telehealth service in the delivery of antenatal care during the COVID-19 pandemic: an interrupted time-series analysis. Lancet. 2021;398:41–52. doi: 10.1016/S0140-6736(21)00668-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Center for Devices and Radiological Health What is digital health? 2020. https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health
  • 9.WHO Digital health. 2019. https://www.who.int/health-topics/digital-health#tab=tab_2
  • 10.Chan RJ, Crichton M, Crawford-Williams F, et al. The efficacy, challenges, and facilitators of telemedicine in post-treatment cancer survivorship care: an overview of systematic reviews. Ann Oncol. 2021;32:1552–1570. doi: 10.1016/j.annonc.2021.09.001. [DOI] [PubMed] [Google Scholar]
  • 11.van der Kolk NM, de Vries NM, Kessels RPC, et al. Effectiveness of home-based and remotely supervised aerobic exercise in Parkinson's disease: a double-blind, randomised controlled trial. Lancet Neurol. 2019;18:998–1008. doi: 10.1016/S1474-4422(19)30285-6. [DOI] [PubMed] [Google Scholar]
  • 12.Salisbury C, O'Cathain A, Edwards L, et al. Effectiveness of an integrated telehealth service for patients with depression: a pragmatic randomised controlled trial of a complex intervention. Lancet Psychiatry. 2016;3:515–525. doi: 10.1016/S2215-0366(16)00083-3. [DOI] [PubMed] [Google Scholar]
  • 13.Ong MK, Romano PS, Edgington S, et al. Effectiveness of remote patient monitoring after discharge of hospitalized patients with heart failure: the better effectiveness after transition—heart failure (BEAT-HF) randomized clinical trial. JAMA Intern Med. 2016;176:310–318. doi: 10.1001/jamainternmed.2015.7712. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Tsou C, Robinson S, Boyd J, et al. effectiveness of telehealth in rural and remote emergency departments: systematic review. J Med Internet Res. 2021;23 doi: 10.2196/30632. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Jarva E, Oikarinen A, Andersson J, et al. Healthcare professionals' perceptions of digital health competence: a qualitative descriptive study. Nurs Open. 2022;9:1379–1393. doi: 10.1002/nop2.1184. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Socha-Dietrich K. Empowering the health workforce to make the most of the digital revolution. OECD Health Working Papers. 2021 doi: 10.1787/18152015. (preprint). [DOI] [Google Scholar]
  • 17.Jandoo T. WHO guidance for digital health: what it means for researchers. Digit Health. 2020;6 doi: 10.1177/2055207619898984. 2055207619898984. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Odone A, Buttigieg S, Ricciardi W, Azzopardi-Muscat N, Staines A. Public health digitalization in Europe. Eur J Public Health. 2019;29(suppl 3):28–35. doi: 10.1093/eurpub/ckz161. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Aapro M, Bossi P, Dasari A, et al. Digital health for optimal supportive care in oncology: benefits, limits, and future perspectives. Support Care Cancer. 2020;28:4589–4612. doi: 10.1007/s00520-020-05539-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Gogia SB, Maeder A, Mars M, Hartvigsen G, Basu A, Abbott P. Unintended consequences of tele health and their possible solutions. Yearb Med Inform. 2016;10:41–46. doi: 10.15265/IY-2016-012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Safi S, Thiessen T, Schmailzl KJ. Acceptance and resistance of new digital technologies in medicine: qualitative study. JMIR Res Protoc. 2018;7 doi: 10.2196/11072. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Pawloski PA, Brooks GA, Nielsen ME, Olson-Bullis BA. A systematic review of clinical decision support systems for clinical oncology practice. J Natl Compr Canc Netw. 2019;17:331–338. doi: 10.6004/jnccn.2018.7104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Thong HK, Wong DKC, Gendeh HS, Saim L, Athar PPBSH, Saim A. Perception of telemedicine among medical practitioners in Malaysia during COVID-19. J Med Life. 2021;14:468–480. doi: 10.25122/jml-2020-0119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Orlando JF, Beard M, Kumar S. Systematic review of patient and caregivers' satisfaction with telehealth videoconferencing as a mode of service delivery in managing patients' health. PLoS One. 2019;14 doi: 10.1371/journal.pone.0221848. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi: 10.1136/bmj.n71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900. doi: 10.1016/s0140-6736(99)04149-5. [DOI] [PubMed] [Google Scholar]
  • 27.Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions. 2022. https://training.cochrane.org/handbook
  • 28.Campbell M, McKenzie JE, Sowden A, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368 doi: 10.1136/bmj.l6890. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Yazdannik A, Yousefy A, Mohammadi S. Discourse analysis: a useful methodology for health-care system researches. J Educ Health Promot. 2017;6:111. doi: 10.4103/jehp.jehp_124_15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. doi: 10.1186/1471-2288-8-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Serrani S. Editora Pontes; Campinas, SP: 2005. Discurso e Cultura na Aula de Língua.https://www.unicamp.br/~serrani/publics/livros_3.htm [Google Scholar]
  • 32.Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358 doi: 10.1136/bmj.j4008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Lewin S, Booth A, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(suppl 1):2. doi: 10.1186/s13012-017-0688-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Gagnon M-P, Ngangue P, Payne-Gagnon J, Desmartis M. m-Health adoption by healthcare professionals: a systematic review. J Am Med Inform Assoc. 2016;23:212–220. doi: 10.1093/jamia/ocv052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Fiander M, McGowan J, Grad R, et al. Interventions to increase the use of electronic health information by healthcare practitioners to improve clinical practice and patient outcomes. Cochrane Database Syst Rev. 2015;2 doi: 10.1002/14651858.CD004749.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Adepoju IOO, Albersen BJA, De Brouwere V, van Roosmalen J, Zweekhorst M. mHealth for clinical decision-making in sub-Saharan Africa: a scoping review. JMIR Mhealth Uhealth. 2017;5:e38. doi: 10.2196/mhealth.7185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Čartolovni A, Tomičić A, Lazić Mosler E. Ethical, legal, and social considerations of AI-based medical decision-support tools: a scoping review. Int J Med Inform. 2022;161 doi: 10.1016/j.ijmedinf.2022.104738. [DOI] [PubMed] [Google Scholar]
  • 38.Boonstra A, Laven M. Influence of artificial intelligence on the work design of emergency department clinicians a systematic literature review. BMC Health Serv Res. 2022;22:669. doi: 10.1186/s12913-022-08070-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.George J. Significance of technology management and project management aligned with portfolio management is vital for any project based organisation. 2008. https://www.pmi.org/learning/library/technology-project-management-aligned-7144
  • 40.Brewster L, Mountain G, Wessels B, Kelly C, Hawley M. Factors affecting front line staff acceptance of telehealth technologies: a mixed-method systematic review. J Adv Nurs. 2014;70:21–33. doi: 10.1111/jan.12196. [DOI] [PubMed] [Google Scholar]
  • 41.de Grood C, Raissi A, Kwon Y, Santana MJ. Adoption of e-health technology by physicians: a scoping review. J Multidiscip Healthc. 2016;9:335–344. doi: 10.2147/JMDH.S103881. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Nguyen OT, Jenkins NJ, Khanna N, et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc. 2021;28:974–984. doi: 10.1093/jamia/ocaa339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Agarwal S, Perry HB, Long LA, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health. 2015;20:1003–1014. doi: 10.1111/tmi.12525. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Bervell B, Al-Samarraie H. A comparative review of mobile health and electronic health utilization in sub-Saharan African countries. Soc Sci Med. 2019;232:1–16. doi: 10.1016/j.socscimed.2019.04.024. [DOI] [PubMed] [Google Scholar]
  • 45.Early J, Gonzalez C, Gordon-Dseagu V, Robles-Calderon L. Use of mobile health (mHealth) technologies and interventions among community health workers globally: a scoping review. Health Promot Pract. 2019;20:805–817. doi: 10.1177/1524839919855391. [DOI] [PubMed] [Google Scholar]
  • 46.Odendaal WA, Anstey Watkins J, Leon N, et al. Health workers' perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2020;3 doi: 10.1002/14651858.CD011942.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Gagnon MP, Desmartis M, Labrecque M, et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst. 2012;36:241–277. doi: 10.1007/s10916-010-9473-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Braun R, Catalani C, Wimbush J, Israelski D. Community health workers and mobile technology: a systematic review of the literature. PLoS One. 2013;8 doi: 10.1371/journal.pone.0065772. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Xyrichis A, Iliopoulou K, Mackintosh NJ, et al. Healthcare stakeholders' perceptions and experiences of factors affecting the implementation of critical care telemedicine (CCT): qualitative evidence synthesis. Cochrane Database Syst Rev. 2021;2 doi: 10.1002/14651858.CD012876.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Williams F, Boren SA. The role of electronic medical record in care delivery in developing countries. Int J Inf Manage. 2008;28:503–507. doi: 10.1016/j.ijinfomgt.2008.01.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Wright D, Androuchko L. Telemedicine and developing countries. J Telemed Telecare. 1996;2:63–70. doi: 10.1177/1357633X9600200201. [DOI] [PubMed] [Google Scholar]
  • 52.Braa J, Hanseth O, Heywood A, Mohammed W, Shaw V. Developing health information systems in developing countries: the flexible standards strategy. Manage Inf Syst Q. 2007;31:381–402. [Google Scholar]
  • 53.Braa J, Monteiro E, Sahay S. Networks of action: sustainable health information systems across developing countries. Manage Inf Syst Q. 2004;28:337–362. [Google Scholar]
  • 54.Amoakoh-Coleman M, Borgstein ABJ, Sondaal SFV, et al. Effectiveness of mHealth interventions targeting health care workers to improve pregnancy outcomes in low- and middle-income countries: a systematic review. J Med Internet Res. 2016;18:e226. doi: 10.2196/jmir.5533. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Brown J, Pope N, Bosco AM, Mason J, Morgan A. Issues affecting nurses' capability to use digital technology at work: an integrative review. J Clin Nurs. 2020;29:2801–2819. doi: 10.1111/jocn.15321. [DOI] [PubMed] [Google Scholar]
  • 56.Godlee F, Dickersin K. In: Peer review in health sciences. 2nd ed. Godlee F, Jefferson T, editors. BMJ Books; London: 2003. Bias, subjectivity, chance, and conflict of interest in editorial decisions; pp. 91–117. [Google Scholar]
  • 57.Peters DP, Ceci SJ. Peer-review practices of psychological journals: the fate of published articles, submitted again. Behav Brain Sci. 1982;5:187–195. [Google Scholar]
  • 58.Menachemi N, Collum TH. Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy. 2011;4:47–55. doi: 10.2147/RMHP.S12985. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Jacob C, Sanchez-Vazquez A, Ivory C. Social, organizational, and technological factors impacting clinicians' adoption of mobile health tools: systematic literature review. JMIR Mhealth Uhealth. 2020;8 doi: 10.2196/15935. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Alami H, Gagnon M-P, Fortin J-P. Some multidimensional unintended consequences of telehealth utilization: a multi-project evaluation synthesis. Int J Health Policy Manag. 2019;8:337–352. doi: 10.15171/ijhpm.2019.12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods. 2017;16 1609406917733847. [Google Scholar]
  • 62.Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE guide no. 131. Med Teach. 2020;42:846–854. doi: 10.1080/0142159X.2020.1755030. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary appendix
mmc1.pdf (5.5MB, pdf)

RESOURCES