Abstract
ABSTRACT
Objective
The aim of this study was to assess how different groups of health professionals evaluated the usability of a new electronic health record (EHR) and to investigate the association between the usability and burnout, insomnia and turnover intention.
Methods
This cross-sectional study included 1424 health professionals who worked at a Norwegian University Hospital. The usability was measured with the System Usability Scale (SUS) 6 months after the previous electronic record was replaced with a more comprehensive, sector-wide, patient-centred EHR in 2022.
Results
The median SUS score was 25 (IQR 12.5–37.5) out of 100 and ranged from 15 (IQR 7.5–25.0) among medical doctors to 40 (IQR 27.6–55.0) among laboratory technicians. Nurses reported a score of 25 (IQR 12.5–40.0). In clinical contexts, the median SUS score ranged from 15 (IQR 10.0–30.0) within radiology to 27.5 (IQR 15.0–42.5) within internal medicine, whereas laboratory medicine reported a score of 37.5 (IQR 27.5–55.0). In multivariable analyses using health professionals in the highest quarter of the SUS as the reference, those in the lowest quarter were more likely to report burnout (OR 3.05, 95% CI 1.86 to 5.00), insomnia (OR 1.72, 95% CI 1.18 to 2.50) and turnover intention (OR 2.35, 95% CI 1.53 to 3.64).
Conclusion
Most health professionals across all occupational groups and clinical contexts reported low usability of a new EHR 6 months after go-live. Those who reported the lowest usability were more likely to report burnout, insomnia and turnover intention.
Keywords: Electronic Health Records, Health Information Systems, Health Care Sector, Occupational Medicine
WHAT IS ALREADY KNOWN ON THIS TOPIC
Although electronic health records (EHRs) intend to enhance the quality and efficiency of healthcare delivery, they may create substantial challenges for the health professionals such as increased workload, job dissatisfaction and symptoms of burnout.
Previous studies have focused on the usability of EHRs among medical doctors and nurses only, and no study has investigated the association between the usability and burnout, insomnia and turnover intention shortly after go-live.
WHAT THIS STUDY ADDS
This is the first study to assess and quantify perceived usability among all groups of health professionals when transitioning from a simple record to a more comprehensive EHR in a hospital setting.
Although this study found some differences in usability between occupational groups and clinical contexts, all reported an unacceptable low usability level 6 months after go-live.
Burnout, insomnia and intention to seek a job elsewhere were more prevalent among health professionals who reported the lowest usability.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
These findings highlight the importance of ensuring acceptable usability of a new EHR to support clinical work and reduce the potential burden for health professionals in hospitals.
Our findings are relevant for policymakers and hospitals that are planning the implementation of comprehensive, sector-wide, patient-centred EHRs.
Introduction
Over the last decade, vast investments in health information technology, such as electronic health records (EHRs), have been made globally.1,3 These investments have been built on the assumption that EHRs will improve patient care across healthcare settings.2 4 To support and facilitate clinical work, health professionals depend on the usability of an EHR to enter, view and extract health information quickly and accurately. To reach an acceptable usability level in hospital, it is important to have a well-designed EHR architecture.5,7 Usability, which is defined as the extent to which technology can be used to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use, has therefore emerged as a critical issue in healthcare.8
Replacing a hospital facility-centred electronic patient record (EPR) with a more comprehensive, sector-wide, patient-centred EHR that interacts with other hospitals, municipalities and primary care entails a comprehensive sociotechnical change. This transforms clinical work tasks, workflows and quality of patient care.2 5 9 Such transition is particularly challenging because of the complexity of medical data, security and collaboration across healthcare settings.10 11 Previous studies have reported overall poor EHR usability in hospitals post implementation.12,16 However, these studies have primarily focused on medical doctors and nurses12 13 15 or one clinical context (eg, emergency medicine or mental health setting).14 16 Since hospitals comprise different types of health professionals who are exposed to different work demands and specific work tasks while using the EHR, it is important to evaluate the usability across all occupational groups and clinical contexts. This knowledge will be relevant for hospitals planning to implement comprehensive, sector-wide, patient-centred EHRs.
Health professionals who report poor EHR usability are more likely to report job dissatisfaction,17 turnover intention17 and burnout.12 13 15 A plausible explanation is that poor usability leads to increased cognitive workload by introducing new tasks and more complex reporting.18 However, there is a lack of studies that address the link between these variables after go-live of a new EHR. Moreover, an unexplored factor in the context of usability is sleep. It is well documented that cognitive and emotional work demands are linked to insomnia symptoms,19 and that insomnia among health professionals is associated with reduced work performance, sick leave and accidents.20 Nevertheless, no study has assessed whether insomnia is more prevalent among health professionals who report low EHR usability.
The aim of this cross-sectional study was twofold. First, to describe how health professionals within different occupational groups and clinical contexts evaluate the usability of a sector-wide, patient-centred EHR 6 months after it replaced a hospital facility-centred EPR. Second, to examine the association between the usability and burnout, insomnia and turnover intention.
Methods
Study design, participants and ethics
This study used cross-sectional data from the study of new technology and health among hospital employees (STUNTH, www.stunth.no) at St Olavs University Hospital, Norway. The hospital provides both outpatient and inpatient healthcare and has ~10 500 employees on its record (nurses, n=3949; medical doctors, n=1411; laboratory technicians, n=459), where ~7100 employees are working at a minimum of 70% of full-time capacity. In the current study, we invited 3766 participants (36%) who were already enrolled in the STUNTH study. Of these, 2115 participated (56%) in May 2022. We selected 1670 participants who reported patient-oriented work and and use of the new EHR. Of these, 159 participants had missing data on usability and 87 participants had missing data on health-related variables. Thus, our final sample comprises 1424 participants.
All participants provided written informed consent prior to participation. This study is reported according to the Strengthening the Reporting of Observational Studies in Epidemiology and Statement on Reporting of Evaluation studies in Health Informatics guidelines.21 22
Study context
The new sector-wide, patient-centred EHR9 ‘Helseplattformen’ (English: ‘The Health Platform’)23 replaced the Norwegian-designed hospital facility-centred EPR9 ‘Doculive’ along with several other programmes in a university hospital in November 2022 (figure 1). The EHR is based on software from an American supplier (Epic, www.epic.com). The vision is to obtain health information using standardised information models for each patient across municipalities, hospitals, general practitioners and contract specialists.9 23 The laboratory version of the EHR was implemented at the clinic of laboratory medicine already in 2020.
Figure 1. Timeline of the successive implementation of two different electronic health records at the hospital. aDefinitions according to the International Organization for Standardization.8.
Variables and data sources
Usability of the new EHR
Information about the usability was assessed using the System Usability Scale (SUS).24 The SUS is a 10-item, standardised, poststudy questionnaire developed to assess the perceived usability of computer systems (online supplemental table 1). It consists of five positive and five negative statements. Each statement includes a 5-point Likert scale ranging from ‘Strongly agree’ to ‘Strongly disagree’. The scale provides an internationally recognised, context-independent, overall summative score from 0 to 100 (higher scores indicate better usability). According to the ‘Acceptability/adjective’ interpretation, a score of 71.1–100 is categorised as ‘Acceptable’, 51.7–71 as ‘Marginally acceptable’, 25–51.6 as ‘Not acceptable/poor’ and 0–24.9 as ‘Not acceptable/worst imaginable’25 (online supplemental table 2).
Burnout
The Burnout Assessment Tool (BAT-12) was used to measure burnout.26 The BAT-12 consists of 12 items comprising four dimensions (exhaustion, mental distancing, emotional impairment and cognitive impairment). Each statement is rated on a 5-point Likert scale ranging from ‘Never’ to ‘Always’. If the average score across all items was ≥2.54, participants were considered to have burnout. Moreover, the following cut-off levels were used for the four dimensions: ≥3.17 (exhaustion), ≥2.17 (mental distancing), ≥2.17 (emotional impairment) and ≥2.83 (cognitive impairment).
Insomnia
Insomnia symptoms were assessed using the Insomnia Severity Index,27 a seven-item questionnaire assessing symptoms such as difficulty falling or staying asleep, satisfaction with sleep and degree of impairment with daytime functioning during the last month. Each item was rated from ‘No problem’ (0 points) to ‘Very severe problem’ (4 points), yielding a total score ranging from 0 to 28. Participants were defined as having insomnia if the score was ≥12.
Turnover intention
The following single question from the third version of the Copenhagen Psychosocial Questionnaire III was used to measure turnover intention28: ‘How often do you consider looking for work elsewhere?’ Response options include ‘Always’, ‘Often’, ‘Sometimes’, ‘Seldom’ and ‘Never/hardly ever’. Participants who answered ‘Always’ or ‘Often’ were considered to have ‘high turnover intention’.
Other variables
Detailed data on occupation, clinical context, age, sex and other employment information were obtained from the hospital administrative system. Occupational groups were categorised according to the WHO classification.29
Statistical analysis
Descriptive statistics of the study sample are reported as means with standard deviations (SDs) and frequencies. The median SUS score was calculated with interquartile range (IQR). Boxplots were used to present the median SUS score across occupational groups and clinical contexts with the IQR, minimum and maximum values and outliers. Due to the non-normality of the SUS score, a multiple quantile regression model was used to estimate median differences in the SUS score between occupational groups and clinical contexts. The precision of the estimates was assessed by 95% confidence intervals (CIs) using robust variance estimates. All the occupational groups were compared with the reference category of nurses. The internal medicine was used as a reference group when we analysed differences between the clinical contexts. The analyses were adjusted for employment fraction (continuous), age (continuous) and sex (female, male).
The associations between SUS score and burnout, insomnia and turnover intention were estimated using a logistic regression model. The precision of the odds ratios (ORs) was assessed by 95% CIs. In these analyses, health professionals in the three lowest quarters were compared with the reference group in the highest quarter. All associations were adjusted for age (continuous), sex (female, male), employment fraction (continuous), occupational group and clinical context. We used postestimation commands to obtain prevalence differences (PDs) with 95% CIs, using those with the highest SUS score as the reference category. Restricted cubic splines were used to model possible non-linear relationships between a continuous SUS score and burnout, insomnia and turnover intention. All the statistical analyses were performed in Stata/MP V.18.0 (StataCorp, College Station, Texas).
Sensitivity analyses
We conducted sensitivity analyses to test the robustness of the results. First, we repeated the main analyses on nurses only, since they represent the largest group of workers. Second, we compared the SUS score among medical doctors within different clinical contexts due to the large variety of use of the system. Third, to identify health professionals who were likely to use the EHR on a weekly basis, we repeated the analyses excluding participants working less than 70% of full-time capacity.
Results
Participant characteristics
Table 1 presents the characteristics of the study sample comprising a total of 1424 participants. The sample comprised mostly nurses (47.2%) and medical doctors (16.1%).
Table 1. Participant characteristics (n=1424).
| Age, mean (SD), year | 45.5 (11.5) |
| Sex,n(%) | |
| Female | 1199 (84.2) |
| Male | 225 (15.8) |
| Year of completed education,n(%) | |
| <2000 | 432 (30.3) |
| 2000–2009 | 423 (29.7) |
| 2010–2023 | 522 (36.7) |
| Missing | 47 (3.3) |
| Type of employment,n(%) | |
| Permanent | 1230 (86.4) |
| Temporary | 194 (13.6) |
| Employment fraction,n(%) | |
| 0–69% | 152 (11.7) |
| 70–100% | 1272 (89.3) |
| Setting,n(%) | |
| Inpatient (only) | 764 (53.7) |
| Outpatient (only) | 254 (17.8) |
| Combination of inpatient and outpatient | 406 (28.5) |
| Occupational group,n(%) | |
| Nurse* | 672 (47.2) |
| Medical doctor | 229 (16.1) |
| Manager† | 110 (7.2) |
| Laboratory technician‡ | 87 (6.1) |
| Psychologist | 65 (4.6) |
| Medical secretary§ | 54 (3.8) |
| Therapist¶ | 50 (3.5) |
| Imaging technicianundefined | 47 (3.3) |
| Healthcare assistant | 40 (2.8) |
| Other groupsundefined | 67 (4.7) |
| Clinical context,n(%) | |
| Internal medicineundefined | 329 (23.1) |
| Anaesthesiologyundefined | 243 (17.1) |
| Psychiatryundefined | 214 (15.0) |
| Surgeryundefined | 192 (13.5) |
| Laboratory medicine | 86 (6.0) |
| Orthopaedics | 72 (5.1) |
| Radiology | 69 (4.9) |
| Oncology | 67 (4.7) |
| Other contexts | 152 (10.7) |
| Variables | |
| SUS score median (IQR) | 25 (12.5–37.5) |
| Burnout, n (%) | 200 (14.0) |
| Exhaustion | 289 (20.3) |
| Mental distance | 412 (28.9) |
| Emotional impairment | 155 (20.9) |
| Cognitive impairment | 113 (8.0) |
| Insomnia, n (%) | 333 (23.4) |
| Turnover intention, n (%) | 241 (16.9) |
Including specialist nurses and midwives.
Including administrators.
Including medical and pathology laboratory technicians and scientists, biomedical engineers, and life science technicians.
Including secretaries without health authorizsation but do patient-related work.
Occupational and physiotherapists.
Medical imaging and therapeutic equipment technicians.
Including pedagogues, pharmacists, porters, ambulance workers, and social workers.
Including cardiology, internal medicine (haematology, nephrology and more), neurology, paediatrics, rheumatology, occupational and rehabilitation medicine.
Including intensive care and emergency services.
Adult and child psychiatry, including intoxication, and addiction medicine.
Including abdominal, gynaecological and obstetric surgery, neurosurgery, urology, and other surgery.
IQRInterquartile RangeSDStandard DeviationSUS, System Usability Scale
SUS score according to occupational group and clinical context
Figure 2 presents the median SUS score for participants in different occupational groups and clinical contexts. The median SUS score among the 1424 participants was 25 (IQR 12.5–37.5) out of 100, corresponding to a ‘Not acceptable/poor’ level (online supplemental table 2). The median SUS score for different occupational groups ranged from 15 (IQR 7.5–25.0) among medical doctors to 40 (IQR 27.5–55.0) among laboratory technicians. Compared with nurses who had a median SUS score of 25, medical doctors had a median difference of −8.3 (95% CI −11.0 to −5.6), whereas the median difference was 7.1 (95% CI −1.0 to 15.2) for laboratory technicians (online supplemental table 3). The median SUS score in clinical contexts ranged from 15 (IQR 10.0–30.0) within radiology to 37.5 (IQR 27.5–55.0) within laboratory medicine. Compared with internal medicine (27.5 (IQR 15.0–42.5)), radiology and laboratory medicine had median differences of −9.2 (IQR −5.5 to 2.9) and 3.2 (IQR −5.3 to 11.8), respectively (online supplemental table 3).
Figure 2. Boxplot showing the median System Usability Scale (SUS) score within occupational groups (A) and clinical contexts (B). The line through the box represents the median. The lower edge of the box represents quartile 1, whereas the upper edge of the box represents quartile 3. Whiskers represent 1.5 IQR for the 25% upper and lower scores. Dots represent outliers.
Burnout, insomnia and turnover intention
Table 2 shows the associations between quarters of the SUS score and burnout, insomnia and turnover intention. The prevalence of burnout increased with decreasing SUS score. Participants in the highest and lowest quarters of the SUS score had prevalence estimates of burnout of 8.5% and 20.5%, respectively, corresponding to an adjusted PD of 13% (95% CI 7% to 18%) and an adjusted OR of 3.01 (95% CI 1.86 to 5.00). Moreover, the same trends were found for the four subdimensions (online supplemental table 4). Specifically, the prevalence estimates for the subdimension ‘mental distance’ were 21.3% and 39.1% for participants in the highest and lowest quarters, respectively. This corresponds to an adjusted PD of 20% (95% CI 14% to 27%) and an adjusted OR of 2.83 (95% CI 1.98 to 4.06).
Table 2. Associations of System Usability Scale (SUS) quarters with burnout, insomnia and turnover intention.
| Quarters of the SUS score | n | Casesn (%) | Unadjusted OR(95% CI) | Adjusted OR*(95% CI) | Adjusted PD†(95% CI) |
| Burnout symptoms | |||||
| 4. Quarter (40–100) | 353 | 30 (8.5) | 1.00 | Reference | Reference |
| 3. Quarter (26–39) | 303 | 34 (11.2) | 1.36 (0.81 to 2.28) | 1.41 (0.83 to 2.37) | 0.03 (−0.02 to 0.73) |
| 2. Quarter (15–25) | 364 | 53 (14.6) | 1.83 (1.14 to 2.95) | 2.01 (1.22 to 3.31) | 0.07 (0.02 to 0.11) |
| 1. Quarter (0–14) | 404 | 83 (20.5) | 2.78 (1.78 to 4.35) | 3.05 (1.86 to 5.00) | 0.13 (0.07 to 0.18) |
| Insomnia symptoms | |||||
| 4. Quarter (40–100) | 353 | 65 (18.4) | 1.00 | Reference | Reference |
| 3. Quarter (26–39) | 303 | 58 (19.1) | 1.05 (0.71 to 1.55) | 1.00 (0.67 to 1.50) | 0.00 (−0.06 to 0.06) |
| 2. Quarter (15–25) | 364 | 100 (27.5) | 1.68 (1.18 to 2.39) | 1.79 (1.23 to 2.60) | 0.09 (0.04 to 0.16) |
| 1. Quarter (0–14) | 404 | 119 (27.2) | 1.66 (1.17 to 2.35) | 1.72 (1.18 to 2.50) | 0.09 (0.03 to 0.15) |
| Turnover intention | |||||
| 4. Quarter (40–100) | 353 | 30 (8.5) | 1.00 | Reference | Reference |
| 3. Quarter (26–39) | 303 | 33 (10.9) | 0.84 (0.52 to 1.35) | 0.93 (0.57 to 1.53) | −0.01 (−0.06 to 0.04) |
| 2. Quarter (15–25) | 364 | 71 (19.5) | 1.66 (1.11 to 2.49) | 1.86 (1.21 to 2.86) | 0.08 (0.03 to 0.13) |
| 1. Quarter (0–14) | 404 | 92 (22.8) | 2.02 (1.37 to 2.98) | 2.35 (1.53 to 3.64) | 0.11 (0.06 to 0.17) |
Adjusted for occupational group, clinical context, employment fraction (continuous, 0–100), age (continuous, 24–78), and sex (female, male).
Prevalence difference (mean predicted probability) adjusted for occupational group, clinical context, employment fraction (continuous, 0–100), age (continuous, 24–78), and sex (female, male).
The same trends were found for insomnia and turnover intention. Specifically, the prevalence estimates for insomnia were 18.4% and 27.2% for those in the highest and lowest quarters of the SUS score, respectively, corresponding to an adjusted PD of 9% (95% CI 3% to 5%) and an adjusted OR of 1.72 (95% CI 1.18 to 2.50). The prevalence estimates for turnover intention were 12.8% and 22.8%, respectively, corresponding to an adjusted PD of 11% (95% CI 6% to 17%) and an adjusted OR of 2.23 (95% CI 1.53 to 3.64). Figure 3 depicts the estimated prevalence of burnout, insomnia and turnover intention, which decreases with increasing SUS score.
Figure 3. Estimated prevalence of burnout, insomnia and turnover intention with 95% CIs by System Usability Scale (SUS) score among 1424 participants. All models are adjusted for occupational group, clinical context, employment fraction, sex and age. The solid line represents the predicted prevalence of burnout, insomnia and turnover intention according to the SUS score, respectively. The shaded area represents the 95% CI. Restricted cubic splines were used to model non-linear relationships using three knots (20, 40, 60) for the SUS score.
Sensitivity analyses
When we restricted our analyses to nurses, we found somewhat stronger associations between SUS score and burnout and insomnia (online supplemental table 5). When we compared the SUS scores between clinical contexts, we observed that both nurses and medical doctors reported higher usability within internal medicine and lower usability within oncology and psychiatry (online supplemental tables 6A and 6B). Medical doctors at laboratory medicine reported the highest SUS score. Excluding participants who worked less than 70% of full-time capacity had negligible influence on the results (online supplemental tables 7 and 8).
Discussion
This cross-sectional study showed that health professionals across all occupational groups and clinical contexts in hospital reported low usability of a sector-wide, patient-centred EHR 6 months after go-live. The overall median SUS score was 25 out of 100, indicating a not acceptable usability level.25 Health professionals who reported the lowest usability were more likely to report burnout, insomnia and turnover intention.
Usability among different occupational groups and clinical contexts
The median SUS score in the current study was very low compared with previous studies.12 13 16 These studies have primarily described usability among nurses and medical doctors12 13 within one clinical context16 or without indication of time period for implementation.12 13 Some studies using the same software supplier as in the current study have shown poor usability in the years after go-live, with some variability across different health professionals.3 16 A study that analysed the implementation of EHRs in Denmark and Finland found extensive user dissatisfaction in both countries several years after go-live.3 In Finland, they scored the usability (eg, logical functions, understandable terminology, easy access to patient information) much lower after implementation of the new system, and medical doctors were less satisfied than nurses and social workers.3 Our study builds on these previous evaluations of usability by studying the usability among all occupational groups and clinical contexts in a hospital 6 months after a patient-centred EHR replaced a facility-centred EPR. Despite the overall low usability in the current study, we found some statistically significant differences between occupational groups and clinical contexts. Specifically, medical doctors reported the lowest usability, and laboratory technicians reported the highest. Some variability is expected considering that different groups of health professionals rely differently on the EHR as a tool to perform their clinical work tasks. The heterogeneity of patients and clinical work in different clinical contexts means that there is a large variety in documentation time, complexity of medical data, acuteness of work and consequences of mistakes. The variety of work tasks and responsibility for patient treatment and medication prescriptions could explain why medical doctors reported the lowest usability. In a sensitivity analysis where we restricted our analysis to medical doctors, we found that medical doctors in oncology and psychiatry reported lower usability than those within internal medicine and laboratory medicine. However, these results should be interpreted with caution due to low participation in some of the clinical contexts. Moreover, our findings showing the highest usability among laboratory technicians could be explained by more standardised work tasks and the fact that they had been using the system for a longer period due to an earlier point of go-live.
Notably, the present study evaluated a comprehensive patient-centred EHR that replaced a hospital facility-centred EPR. Compared with the old EPR, the new EHR has a more detailed interface and requires health professionals to perform more standardised work to deliver semantic interoperable health data to support clinical decision-making and information exchange across primary, secondary and tertiary care.2 11 The overall low usability in this study could therefore be related to poor adaptation to the hospital work processes. For instance, it is important that the design of the EHR fits the health professional’s specific needs and that it has been properly tested regarding learnability, efficiency in use and risk of errors.6 Previous research highlights the importance of adequate training and real-time support, especially if work is planned to be performed differently.30 However, after 6 months with full operation and frequent use of the system, most health professionals should have gained experience using the system. This suggests that the system has a hard-to-learn interface that does not support specified users in a specified context of use.6
Usability and its association with burnout, insomnia and turnover intention
Despite the overall low SUS score, we found that burnout, insomnia and turnover intention were more prevalent among health professionals who reported the lowest usability. Our results are in line with previous cross-sectional studies showing that low usability is associated with higher prevalence of burnout.12 13 15 The current study expands on these findings by showing that the prevalence of all the interdependent dimensions of burnout increased with decreasing usability scores. However, it should be noted that the prevalence of the dimension that captures enthusiasm, aversion and cynicism at work was especially high among those who reported the lowest usability. Thus, this finding, together with our finding that turnover intention is more prevalent among those with low usability, suggests that usability is related to employees’ motivation to work and intentions to stay in their job.
We are not aware of any previous study that has investigated the association between usability and insomnia symptoms. Since cognitive intrusion caused by stressful work exposure may contribute to poor sleep quality,19 it is plausible that a high cognitive workload due to low usability leads to insomnia symptoms, possibly due to worrying thoughts (eg, fear of mistakes, unfinished work). Indeed, it is possible that health professionals who suffer from insomnia or burnout have reduced abilities to learn and adapt to digital transformation.
Implications and future directions
Our findings showing very low usability after 6 months with full operation and use of extra resources are relevant for hospitals that are planning the implementation of comprehensive, sector-wide, patient-centred EHRs. In times when the healthcare system is under pressure, it is important that EHRs are designed to support clinical work for the health professionals. An EHR that is intuitive and easy to use will maintain daily operations in all phases of the EHR transition and facilitate quick onboarding of new staff. Policymakers and stakeholders must ensure that the usability architecture is acceptable before implementation and continuously test the usability after go-live to further optimise the system. Future research should explore the role of specific usability issues (eg, ease of use, information quality, collaboration) and focus on usability testing across different occupations within different clinical contexts. Moreover, additional research is needed to determine causal relations between usability and health outcomes in health professionals.
Strengths and limitations
The strengths of this study include the evaluation of the usability across several occupational groups and clinical contexts and the use of standardised questionnaires. Some limitations should be considered when interpreting the results. First, the SUS questionnaire is designed to evaluate the usability of specific computerised tasks and not the overall usability of an EHR system. Second, there was a lot of debate in the media prior to this survey, and we cannot exclude the possibility that workers with negative attitudes towards the new EHR were more likely to participate. However, it is also possible that the workers with more positive attitudes towards the new EHR and with sufficient time were more likely to participate, or that workers preoccupied with high workload due to the EHR did not have the time or desire to participate. Third, when we examined the associations between usability and burnout, insomnia and turnover intention, we could not stratify our analyses by occupations or clinical contexts due to the sample size. Finally, our cross-sectional study was not designed to investigate any causal relations between the usability and burnout, insomnia and turnover intention.
Conclusions
In a university hospital, we found very low usability across all occupational groups and clinical contexts 6 months after a sector-wide, patient-centred EHR replaced a hospital facility-centred EPR. Health professionals who reported the lowest usability were more likely to report burnout, insomnia and turnover intention. Our study suggests that it is important to ensure acceptable usability to support clinical work for the health professionals.
supplementary material
Acknowledgements
We acknowledge St Olavs Hospital HF. We especially thank analyst Mads Herdahl for his help with preparing data from the hospital database. We also thank the Nordic eHealth Research Network (NeRN) for relevant meetings and discussions. We acknowledge the help of the language assistant Curie (www.aje.com/curie).
Footnotes
Funding: This work was supported by a grant to Signe Lohmann-Lafrenz from the Liaison Committee between the Central Norway Regional Health Authority (RHA) and the Norwegian University of Science and Technology (grant number: 32969).
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Not applicable.
Ethics approval: This study involves human participants and was approved by the Regional Committee for Ethics in Medical Research, Central Norway (228249). Participants gave informed consent to participate in the study before taking part.
Data availability free text: Due to participant confidentiality, participant data are not publicly available.
Data availability statement
No data are available.
References
- 1.Flyvbjerg B, Budzier A, Lee JS, et al. The Empirical Reality of IT Project Cost Overruns: Discovering A Power-Law Distribution. JMIS. 2022;39:607–39. doi: 10.1080/07421222.2022.2096544. [DOI] [Google Scholar]
- 2.Tsai CH, Eghdam A, Davoody N, et al. Effects of Electronic Health Record Implementation and Barriers to Adoption and Use: A Scoping Review and Qualitative Analysis of the Content. Life (Basel) 2020;10:327. doi: 10.3390/life10120327. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hertzum M, Ellingsen G, Cajander Å. Implementing Large-Scale Electronic Health Records: Experiences from implementations of Epic in Denmark and Finland. Int J Med Inform. 2022;167:104868. doi: 10.1016/j.ijmedinf.2022.104868. [DOI] [PubMed] [Google Scholar]
- 4.Subbe CP, Tellier G, Barach P. Impact of electronic health records on predefined safety outcomes in patients admitted to hospital: a scoping review. BMJ Open. 2021;11:e047446. doi: 10.1136/bmjopen-2020-047446. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sittig DF, Singh H. Cognitive informatics for biomedicine: human computer interaction in healthcare. 2015. A new socio-technical model for studying health information technology in complex adaptive healthcare systems; pp. 59–80. [Google Scholar]
- 6.Nielsen J. Usability engineering. Academic Press; 1993. [Google Scholar]
- 7.Kutney-Lee A, Sloane DM, Bowles KH, et al. Electronic Health Record Adoption and Nurse Reports of Usability and Quality of Care: The Role of Work Environment. Appl Clin Inform. 2019;10:129–39. doi: 10.1055/s-0039-1678551. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Standardization IOf Ergonomics of human-system interaction International Organization for Standardization. ISO 9241-11:2018 (en) 2018. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en Available.
- 9.Standardization IOf Health informatics. Capacity-based eHealth architecture roadmap. Part 1: Overview of national eHealth initiatives. ISO/TR 14639-1:2012. 2012. https://www.iso.org/obp/ui/en/#iso:std:iso:tr:14639:-1:ed-1:v1:en Available.
- 10.Holmes JH, Beinlich J, Boland MR, et al. Why Is the Electronic Health Record So Challenging for Research and Clinical Care? Methods Inf Med. 2021;60:32–48. doi: 10.1055/s-0041-1731784. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Gamal A, Barakat S, Rezk A. Standardized electronic health record data modeling and persistence: A comparative review. J Biomed Inform. 2021;114:103670. doi: 10.1016/j.jbi.2020.103670. [DOI] [PubMed] [Google Scholar]
- 12.Melnick ER, Dyrbye LN, Sinsky CA, et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc. 2020;95:476–87. doi: 10.1016/j.mayocp.2019.09.024. [DOI] [PubMed] [Google Scholar]
- 13.Melnick ER, West CP, Nath B, et al. The association between perceived electronic health record usability and professional burnout among US nurses. J Am Med Inform Assoc. 2021;28:1632–41. doi: 10.1093/jamia/ocab059. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Buivydaite R, Reen G, Kovalevica T, et al. Improving usability of Electronic Health Records in a UK Mental Health setting: a feasibility study. J Med Syst. 2022;46:50. doi: 10.1007/s10916-022-01832-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Heponiemi T, Kujala S, Vainiomäki S, et al. Usability Factors Associated With Physicians’ Distress and Information System-Related Stress: Cross-Sectional Survey. JMIR Med Inform. 2019;7:e13466. doi: 10.2196/13466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bloom BM, Pott J, Thomas S, et al. Usability of electronic health record systems in UK EDs. Emerg Med J. 2021;38:410–5. doi: 10.5061/dryad.7h44j0zsq. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Sinsky CA, Dyrbye LN, West CP, et al. Professional Satisfaction and the Career Plans of US Physicians. Mayo Clin Proc. 2017;92:1625–35. doi: 10.1016/j.mayocp.2017.08.017. [DOI] [PubMed] [Google Scholar]
- 18.Dunn Lopez K, Chin C-L, Leitão Azevedo RF, et al. Electronic health record usability and workload changes over time for provider and nursing staff following transition to new EHR. Appl Ergon. 2021;93:103359. doi: 10.1016/j.apergo.2021.103359. [DOI] [PubMed] [Google Scholar]
- 19.Skarpsno ES, Nilsen TIL, Sand T, et al. Work-Related Mental Fatigue, Physical Activity and Risk of Insomnia Symptoms: Longitudinal Data from the Norwegian HUNT Study. Behav Sleep Med. 2020;18:488–99. doi: 10.1080/15402002.2019.1614927. [DOI] [PubMed] [Google Scholar]
- 20.Mao Y, Raju G, Zabidi MA. Association Between Occupational Stress and Sleep Quality: A Systematic Review. Nat Sci Sleep. 2023;15:931–47. doi: 10.2147/NSS.S431442. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Lesko CR, Fox MP, Edwards JK. A Framework for Descriptive Epidemiology. Am J Epidemiol. 2022;191:2063–70. doi: 10.1093/aje/kwac115. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Talmon J, Ammenwerth E, Brender J, et al. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform. 2009;78:1–9. doi: 10.1016/j.ijmedinf.2008.09.002. [DOI] [PubMed] [Google Scholar]
- 23.Helseplattformen Helseplattformen. 2023. https://www.helseplattformen.no/ Available.
- 24.Brooke J. Usability evaluation in industry. 1996. SUS: a “quick and dirty" usability scale; pp. 189–94. [Google Scholar]
- 25.Sauro J. 5 ways to interpret a SUS score. 2018. https://measuringu.com/interpret-sus-score Available.
- 26.Schaufeli W, De Witte H, Desart S, et al. Manual. Burnout assesment tool. 2020. https://burnoutassessmenttool.be/wp-content/uploads/2020/08/Test-Manual-BAT-English-version-2.0-1.pdf Available. [DOI] [PMC free article] [PubMed]
- 27.Bastien CH, Vallières A, Morin CM. Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Med. 2001;2:297–307. doi: 10.1016/s1389-9457(00)00065-4. [DOI] [PubMed] [Google Scholar]
- 28.COPSOQ International Network COPSOQ III: licence, guidelines & questionnaire. 2020. https://www.copsoq-network.org/licence-guidelines-and-questionnaire Available.
- 29.WHO Classifying health workers. 2019. https://www.who.int/publications/m/item/classifying-health-workers Available.
- 30.Rinne ST, Brunner J, Mohr DC, et al. Practices Supporting Electronic Health Record Transitions: Lessons from Four US Healthcare Systems. J Gen Intern Med. 2023;38:1015–22. doi: 10.1007/s11606-023-08279-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
No data are available.



