Using automated methods to detect safety problems with health information technology: a scoping review

Abstract

Objective

To summarize the research literature evaluating automated methods for early detection of safety problems with health information technology (HIT).

Materials and Methods

We searched bibliographic databases including MEDLINE, ACM Digital, Embase, CINAHL Complete, PsycINFO, and Web of Science from January 2010 to June 2021 for studies evaluating the performance of automated methods to detect HIT problems. HIT problems were reviewed using an existing classification for safety concerns. Automated methods were categorized into rule-based, statistical, and machine learning methods, and their performance in detecting HIT problems was assessed. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Reviews statement.

Results

Of the 45 studies identified, the majority (n = 27, 60%) focused on detecting use errors involving electronic health records and order entry systems. Machine learning (n = 22) and statistical modeling (n = 17) were the most common methods. Unsupervised learning was used to detect use errors in laboratory test results, prescriptions, and patient records while supervised learning was used to detect technical errors arising from hardware or software issues. Statistical modeling was used to detect use errors, unauthorized access, and clinical decision support system malfunctions while rule-based methods primarily focused on use errors.

Conclusions

A wide variety of rule-based, statistical, and machine learning methods have been applied to automate the detection of safety problems with HIT. Many opportunities remain to systematically study their application and effectiveness in real-world settings.

BACKGROUND AND SIGNIFICANCE

Health information technology (HIT) can play an important role in supporting care delivery and improving patient safety.^1–6 Problems with HIT however can introduce new, often unforeseen,⁷ modes of failure that reduce the safety and quality of clinical care and may lead to patient harm and death.⁸ Risks to patients can arise from problems with HIT design or manufacture, the way they are implemented, or how they are used.⁹ Unlike most other risks to patient safety, HIT problems can—because of their scale and scope—harm many patients in a single event.^10,11 Current approaches to detecting HIT problems are primarily based on clinicians’ reports, which often appear well after the fact and so are not directly actionable.^8,12 Many HIT problems are thus either undetected or are detected after patients have been harmed.¹³

One approach to minimize the impact of HIT problems is to proactively monitor clinical data to detect risks in real-time before care is disrupted or patients are harmed. Traditional methods for IT system surveillance are focused on the IT infrastructure and software applications (Figure 1), and have limited ability to detect complex problems that emerge from interactions among separate software and hardware components.¹⁴ Yet it is these interactional failures that characterize HIT system problems, which often comprise multiple software and hardware components. Monitoring at the data level can overcome these limitations. By prospectively monitoring at the clinical data level, HIT problems can be detected early before they become more widespread, thus minimizing disruption to care delivery and risks to patients.^15,16

Figure 1.

Automated methods can be used to monitor HIT at the clinical data level to proactively detect risks. HIT: health information technology.

While previous reviews have examined challenges with HIT and its effects on care delivery and patient outcomes,^11,17 the use of automated methods to monitor clinical data and detect the different kinds of HIT problems has not been reviewed. To develop an overview of the heterogeneous literature on automated methods for detecting problems with HIT, we conducted a scoping review. The goal was to better understand the different automated methods for detecting problems with HIT, the types of HIT systems for which these methods were developed, and method performance.

MATERIALS AND METHODS

This scoping review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Reviews (PRISMA-ScR).¹⁸ We followed the process outlined by Arksey and O’Malley with 4 stages: (1) identifying relevant studies; (2) study selection; (3) data extraction; and (4) summarizing and reporting findings.¹⁹ As the goal of a scoping review is to map a heterogenous field rather than assess the evidence for a specific question, the quality of studies is not typically assessed.²⁰

Identifying relevant studies

A search of the literature was performed in June 2020 and updated in July 2021. We searched MEDLINE, ACM Digital (The ACM Guide to Computing Literature), Embase, CINAHL Complete, PsycINFO, and Web of Science Core Collection (the search strategy is available in Supplementary Material). A total of 5492 studies were retrieved from the search. To be included, articles needed to report studies about automated methods for detecting HIT problems, use data from healthcare settings, and evaluate performance. Only English-language studies published between January 2010 and June 2021 were included. Gray literature (including dissertations, theses, and conference abstracts), errata, and duplicates were excluded (Figure 2). The retrieval set was limited to articles published after 2010 when there was a marked increase in studies about HIT problems,⁸ especially after publication of the IOM report in 2012.¹²

Figure 2.

Study identification and selection.

Study selection

Articles were screened using title and abstract by a single reviewer (initial search: DS, updated search: YW). After screening, 5429 studies were excluded, leaving 63 studies for full-text screening. Full-length articles were retrieved and independently assessed by 2 reviewers against the inclusion criteria and identified 32 eligible studies. We also performed a hand search of the reference lists of included studies, identifying a further 14 studies that met the inclusion criteria. An extension of a previous study by the same authors reporting the same methods and problems was removed,²¹ leaving 45 studies for data extraction. Articles that did not meet any of the inclusion criteria were excluded and any disagreements were resolved by discussion and consensus.

Data extraction, summarizing, and reporting findings

For each included study, we extracted information about the study design (simulated or real-world and, retrospective or real-time), data sources utilized, HIT problems, methods for detecting HIT problems, and their performance (Table 3). Each identified HIT problem was categorized by 2 investigators (DS and YW) using an existing classification for HIT problems developed by the authors.¹⁰ HIT problems were first divided into types involving human factors (ie, use errors) or technical problems, and then categorized into 1 or more subclasses. Use errors related to interaction of humans with HIT, for example, wrong entry of patient information in an electronic health record (EHR). For technical errors, a range of hardware and software issues were included.

Table 3.

Summary of the included studies (n = 45)

Authors (year). Country	Detection method	Data source	HIT problem	HIT problem category 10	Performance measures
Machine learning methods
Chen and Malin (2011).62 United States	Combination of k-NN and PCA	EHR access log	Unauthorized data access	2.10 Computer virusb	AUC-ROC
Chen et al (2011).63 United States	Similarity-based method on network graph	EHR access log	Unauthorized data access	2.10 Computer virusb	AUC-ROC
Boxwala et ala (2011).30 United States	SVM	EHR access log	Unauthorized data access	2.10 Computer virusb	AUC-ROC, sensitivity
Chen et al (2012).64 United States	Nearest-neighbor-based method	EHR access log	Unauthorized data access	2.10 Computer virusb	AUC-ROC
Menon et al (2014).31 United States	Collaborative filtering method	EHR access log	Unauthorized data access	2.10 Computer virusb	AUC-ROC, RMSE, AUPRC, G-mean
Hussain and Qamar (2016).32 Pakistan	Similarity-based method (cosine similarity)	Electronic medical record	Wrong-drug errors in medical records	1.1.1 Wrong entryc	Precision
D’hondt et al (2016).33 France	Long short-term memory	Patient notes (foetopathology)	OCR errors in French clinical reports	1.2.1 Wrong outputd	Accuracy
Zhang et al (2016).74 China	Topical PMAD based on LDA	EHR records	Medication errors	1.1.1 Wrong entryc	Precision
Fivez et al (2017).34 Belgium	Word embeddings and text classification	MIMIC III clinical notes	Errors in clinical text	1.1.1 Wrong entryc	Accuracy
Estiri et al (2019).35 United States	Hybrid hierarchical k-means algorithm	EHR records	Errors in laboratory test results	1.1.1 Wrong entryc	Sensitivity, specificity
Estiri and Murphy (2019).36 United States	Autoencoder (neural network)	EHR records	Errors in laboratory test results	1.1.1 Wrong entryc	Sensitivity, specificity
dos Santos et al (2019).37 Brazil	Graph centrality-based method	CPOE	Medication errors in CPOE	1.1.1 Wrong entryc	Precision, sensitivity, F measure
Zech et al (2019).66 United States	Neural network (seq2seq) model	Head CT and chest radiograph reports	Errors in clinical text	1.1.1 Wrong entryc	Sensitivity, specificity, precision
Aaron et al (2019).38 United States	Naïve Bayes classifier	CDSS	CDSS alerting problems	2.6.2 System configuratione	AUC-ROC, precision

Authors (year). Country	Detection method	Data source	HIT problem	HIT category 10	Performance measures
Zhao et al (2019).57 China	Multi-layer neural networks with Continuous Bag of Words	EHR records	Medication errors	1.1.1 Wrong entryc	Accuracy
Mitani et al (2020).65 Japan	A gradient-boosting-decision-tree	Blood test results	Laboratory result mix-up	1.1.1 Wrong entryc	AUC-ROC, precision
Yazdani et al (2020).39 Iran	Damerau–Levenshtein distance	Breast ultrasonography reports, head and neck ultrasonography reports, abdominal and pelvic ultrasound reports	Errors in clinical text	1.1.1 Wrong entryc	Precision, sensitivity, F measure, accuracy
Watson and Al Moubayed (2020).70 United Kingdom	Neural networks with explainable methods: SHAP-MLP: a simple multi-layer perceptron (SHAP-MLP) SHAP-Conv: a CNN Autoencoders (SHAP-AE) + SVM Variational autoencoders (SHAP-VAE) + SVM	EHR records Chest X-ray images	Adversarial attacks	2.10 Computer virusb	Accuracy
Balabaeva et al (2020).58 Russia	Word embedding with Word2Vec and FastText (Skip-Gram and Continuous Bag of Words)	EHR records	Spelling errors in clinical text	1.1.1 Wrong entryc	Precision
Khaleghi et al (2020).59 United States	HCHAC	Surgery records	Spelling errors in clinical text	1.1.1 Wrong entryc	Accuracy

Table 3.

Continued

Authors (year). Country	Detection method	Data source	HIT problem	HIT category 10	Performance measures
Tong et al (2020).71 Canada	Lipschitz anomaly discriminator (LDA) with Wasserstein-distance	EHR lab data	Recording error	1.1.1 Wrong entryc	AUC-ROC
Kim et al (2021).60 Korea	Similarity-based spelling correction using word embedding	EHR clinical reports	Errors in clinical text	1.1.1 Wrong entryc	Accuracy, precision, recall and F1 score
Statistical modeling methods
Wong and Glance (2011).40 Australia	Statistical semantic analysis		Errors in clinical text	1.1.1 Wrong entryc	Accuracy, detection time
Kim et al (2011).41 United States	Logistic regression	EHR access log	Unauthorised data access	2.10 Computer virusb	AUC-ROC, false negative rates
Boxwala et ala (2011).30 United States	Logistic regression	EHR access log	Unauthorised data access	2.10 Computer virusb	AUC-ROC, sensitivity
Fabbri and LeFevre (2012).67 United States	Access/treatment probability, responsibility, reasons for access-based method	Audit log data from EHR database	Unauthorised data access	2.10 Computer virusb	Precision, sensitivity, F measure
Yamashita et al (2013).68 Japan	Weighted cumulative delta index	LIS	Laboratory result mix-up	1.1.1 Wrong entryc	AUC-ROC, sensitivity
Ong et al (2013).69 Australia	Shewhart statistical control process chart	LIS	Erroneous/duplication data Data loss at record and field levels	1.2.1 Wrong outputd 1.1.1 Wrong entryc 1.2.3 No outputf	Sensitivity, specificity, detection time
Lai et al (2015).42 United States	Shannon’s noisy channel model	EHR clinical notes	Errors in clinical text	1.1.1 Wrong entryc	Precision, sensitivity, F measure, accuracy
Siklósi et al (2016).43 Hungary	Statistical machine translation	Clinical notes (ophthalmology)	Errors in clinical text	1.1.1 Wrong entryc	Precision, sensitivity, F measure, accuracy

Authors (year). Country	Detection method	Data source	HIT problem	HIT category 10	Performance measures
Ray and Wright. (2016).44 United States	Poisson changepoint, Autoregressive Integrated Moving Average, Seasonal Hybrid Extreme Studentized Deviate, E-Divisive with Median	CDSS	CDSS alerting problems	2.6.2 System configuratione	Yes/no detection, detection time
Goodloe et al (2017).45 United States	A change ratio distribution over a certain time interval method	EHR	Problems of transcription and recording error	1.1.1 Wrong entryc	Yes/no detection
Liu et al (2017).73 United States	Multi-Process Dynamic Linear model	CDSS	CDSS alerting problems	2.6.2 System configuratione	AUC-AMOC, detection time
Kassakian et al (2017).46 United States	Statistical process control c-charts	CDSS	CDSS alerting problems	2.6.2 System configuratione	Precision, sensitivity, F measure
Liu et al (2018).47 United States	Seasonal-trend decomposition and likelihood ratio statistics	CDSS	CDSS alerting problems	2.6.2 System configuratione	AUC-AMOC, detection time
Ray et al (2018).48 United States	Poisson changepoint, Autoregressive Integrated Moving Average, Seasonal Hybrid Extreme Studentized Deviate, E-Divisive with Median, Hierarchical Divisive Changepoint, Bayesian Changepoint	CDSS	CDSS alerting problems	2.6.2 System configuratione	Yes/no detection, detection time
Gewald et al (2018).49 Germany	Scoring-based method (date, time, department distance, job title)	Log files from the logging system of the information system	Unauthorised data access	2.10 Computer virusb	Yes/no detection
Boddy et al (2019).50 United Kingdom	Local Outlier Factor algorithm	EHR access log	Unauthorised data access	2.10 Computer virusb	Yes/no detection
Phan et al (2020).61 United Kingdom	Combination of standard deviation values and linear regression	EHR records	Recording error	1.1.1 Wrong entryc	Sensitivity, PPV

Table 3.

Continued

Authors (year). Country	Detection method	Data source	HIT problem	HIT category 10	Performance measures
Rule-based methods
Patrick et al (2010).51 Australia	Rules based on edit distance, phonetic, whitespace, and concatenated words	Clinical notes (Emergency Department)	Errors in clinical text	1.1.1 Wrong entryc	Accuracy
Adelman et al (2013).52 United States	Rules based on retract-and-order	CPOE	Wrong-patient errors in CPOE	1.1.1 Wrong entryc	Precision, detection time
Uddin and Dalianis (2014).53 Sweden	Rules based on lexical lookup and exact matching technique	Clinical notes (Neurology, Orthopaedia, Infection, Dental Surgery and Nutrition)	Errors in clinical text	1.1.1 Wrong entryc	Yes/no detection
Minn et al (2015).54 United States	Rules using regular expression	Radiology reports	Errors in clinical text	1.1.1 Wrong entryc	Precision
Rash-Foanio et al (2017).55 United States	Rules based on 3 conditions and BI-SIM similarity	CPOE	Medication errors in CPOE	1.1.1 Wrong entryc	Accuracy
Lambert et al (2019).56 United States	Rules based on 2 conditions and LASA similarity	EHR records	Medication errors in EHR	1.1.1 Wrong entryc	Precision
Vaishnavi and Sethukarasi (2020).72 India	Rule-based BTD algorithm	IoT based smart health data	Network attack	2.10 Computer virusb	Detection time Detection rate, False positive rate

AUC-AMOC: area under curve (activity monitoring operating characteristic); AUC-ROC: area under curve (receiver operator characteristic); AUPRC: area under precision–recall curve; BTD: Blue Tits detection; CDSS: clinical decision support systems; CNN: convolutional neural network; CPOE: computerized physician order entry; G-mean: geometric mean; HCHAC: Heuristic Clustering algorithm of Hierarchical Agglomerative Clustering; HIT: health information technology; HER: electronic health record; LDA: Latent Dirichlet Allocation; LIS: laboratory information systems; RMSE: root mean squared errors; SVM: support vector machine.

^aThis study used both SVM (machine learning method) and logistic regression (statistical modeling method) as their proposed methods.

^b2. Software and hardware problems > 2.10 Computer virus.

^c1. Information input/output errors > 1.1 Use errors > 1.1.1 Wrong entry.

^d1. Information input/output errors > 1.2 Technical error > 1.2.1 Wrong output.

^e2. Software and hardware problems > 2.6 Software issue > 2.6.2 System configuration.

^f1. Information input/output errors > 1.2 Technical error > 1.2.3 No output.

Based on the approach to computational reasoning,¹⁵ methods for automated detection were categorized into rule-based, statistical modeling, and machine learning methods by the same 2 investigators^22–24 (Table 1), and measures used to evaluate model performance were extracted. A narrative synthesis then integrated findings into descriptive summaries.

Table 1.

Categories of automated methods for detecting HIT problems

Category	Characteristics and example
Rule-based method	Applies rules developed by human experts with domain-specific knowledge to model data,25,26 for example, rules to detect prescribing errors using medication dose ranges based on clinical practice guidelines.
Statistical modeling method	Uses mathematical models and statistical analyses to identify data patterns for making predictions,27,28 for example, the Poisson distribution used to model number of clinical alerts within a given time interval.
Machine learning method	Uses algorithms to model data by learning patterns from sample/training data,22,27,29 for example, an SVM model trained to detect suspicious activity based on patterns of appropriate access in training data.

HIT: health information technology; SVM: support vector machine.

RESULTS

Descriptive analysis of studies

We identified 45 studies reporting different automated methods for detecting HIT problems (Tables 2 and 3, Supplementary Table S1). Most studies (n = 23) were conducted in the United States and focused on a range of HIT systems including EHRs, computerized physician order entry (CPOE), clinical decision support systems (CDSS), and laboratory information systems (LIS) (Table 2).

Table 2.

Characteristics of the 45 studies reporting automated methods for detecting HIT problems

Characteristics	Studies, n (%)
Setting
Inpatient52,69,74	3 (6.7)
Outpatient45	1 (2.2)
Inpatient and outpatient30,38,42,55,56,65,67,68	8 (17.8)
Not reported31–37,39–41,43,44,46–51,53,54,57–64,66,70–73	33 (73.3)
Country
United States30,31,35,36,38,41,42,44–48,52,54–56,59,62–64,66,67,73	23 (51.1)
Australia40,51,69	3 (6.7)
United Kingdom50,61,70	3 (6.7)
Other32–34,37,39,43,49,53,57,58,60,65,68,71,72,74	16 (35.6)
HIT system type
EHR30–32,34–36,39–42,45,50,51,53,56–58,61–64,66,67,70,71,74	26 (57.8)
Clinical data32,34–36,39,40,42,45,51,53,56–58,61,66,70,71,74	18 (40.0)
EHR access log data30,31,41,50,62–64,67	8 (17.8)
Undefined hospital information logging system49	1 (2.2)
LIS68,69	2 (4.4)
CPOE37,52,55	3 (6.7)
CDSS38,44,46–48,73	6 (13.3)
Healthcare sensor cloud72	1 (2.2)
Not reported33,43,54,59,60,65	6 (13.3)
Detection methodsa
Machine learning method30–39,57–60,62–66,70,71,74	22 (48.9)
Unsupervised learning31,32,34,35,37,39,57–60,62–64,66,71,74	16 (35.6)
Supervised learning30,33,38,65,70	5 (11.1)
Semi-supervised learning36,70	2 (4.4)
Statistical modeling method30,40–50,61,67–69,73	17 (37.8)
Rule-based methods51–56,72	7 (15.6)
Data types (model development, performance evaluation)
Real-world data, real-world errors30–61	32 (71.1)
Real-world data, simulated errors62–72	11 (24.4)
Real-world data, real-world + simulated errors73,74	2 (4.4)
Performance measures
Precision32,37–39,42,43,46,52,54,56,58,60,61,65–67,74	17 (37.8)
Sensitivity30,35–37,39,42,43,46,60,61,66–69	14 (31.1)
Accuracy33,34,39,40,42,43,51,55,57,59,60,70	12 (26.7)
AUC-ROC30,31,38,41,62–65,68,71	10 (22.2)
F measure37,39,42,43,46,60,67	7 (15.6)
Timeliness of detection44,47,48,52,69,72,73	7 (15.6)
Yes/no detection44,45,48–50,53	6 (13.3)
Specificity35,36,66,69	4 (8.9)
AUC-AMOC47,73	2 (4.4)
RMSE31	1 (2.2)
AUPRC31	1 (2.2)
G-mean31	1 (2.2)

^aStudy Boxwala et al³⁰ used both machine learning and statistical modeling methods and study Watson and Moubayed⁷⁰ used both supervised and semi-supervised machine learning methods.

AUC-AMOC: area under curve (activity monitoring operating characteristic); AUC-ROC: area under curve (receiver operator characteristic); AUPRC: area under precision–recall curve; G-mean: geometric mean; HIT: health information technology; RMSE: root mean squared errors; SVM: support vector machine.

Of the 45 studies reviewed, 18 focused on detecting technical errors arising from hardware or software systems (eg, EHRs, CPOE, CDSS, and LIS). These technical errors included the configuration of software systems, computer viruses, and errors in patient information (no output, wrong output, or wrong entry; Table 3). The remaining 26 studies focused on errors in the use of EHRs and CPOE that gave rise to patient identification errors, prescribing errors, and spelling errors. Only 1 study investigated both technical and use errors.

The included studies identified 52 different HIT detection methods including 24 machine learning, 21 statistical methods, and 7 rule-based approaches (Table 3). All the studies used real-world data to develop models. To evaluate method performance, 32 studies used real-world data,^30–61 11 studies simulated HIT errors,^62–72 and 2 used a combination of real-world data and simulated errors.^73,74 A summary of these studies is given in the following sections.

Machine learning methods

Of the 22 studies that used machine learning, the majority (n = 16, 72.7%) used unsupervised learning methods that do not require labeled data.^{31,32,34,35,37,39,57–60,62–64,66,71,74} Only 5 studies examined supervised learning methods.^{30,33,38,65,70} In the following sections, we summarize these studies.

Unsupervised learning methods

Use errors were commonly detected using unsupervised learning methods (n = 12), such as extended Latent Dirichlet Allocation,⁷⁴ similarity-based clustering methods,⁶⁴ and autoencoder neural networks.³⁶ Examples of such errors included implausible laboratory test results,^35,71 prescribing errors,^37,57,74 and errors in the clinical text (misspelling, insertion, substitution, and deletion errors).^{32,34,39,58–60,66} Overall, these studies achieved a high accuracy of detection and used a variety of metrics to evaluate model performance. For instance, an accuracy of 100% (precision, recall, and F1 score: 97%) was achieved in 1 study using pre-learned word embedding from a clinical dataset, to identify spelling errors in bacterial culture and antimicrobial susceptibility reports.⁶⁰ Another study reported high sensitivity (>0.85) and specificity (>0.997) in detecting implausible clinical observations in medical records (laboratory tests and vital signs) using a hybrid hierarchical k-means clustering algorithm.³⁵

Unsupervised methods were also shown to be effective in detecting unauthorized access to EHRs in 4 studies.^31,62–64 Unlike the diverse performance measures reported above, AUC-ROC was commonly used to examine the performance of unsupervised methods. In 1 study, collaborative filtering demonstrated a high AUC-ROC of 0.996,³¹ while a nearest-neighbor-based method achieved an AUC-ROC of 0.92 in another.⁶⁴ A third study, which used a dynamic social network reported an AUC-ROC of 0.83.⁶³

Supervised learning methods

We identified 5 studies (23.8%) in this category examining technical errors (n = 4) as well as use errors (n = 1).⁶⁵ With use errors, an AUC-ROC of 0.9984 was achieved by a gradient-boosting-decision-tree method to detect mix-ups in blood samples.⁶⁵ Studies detecting technical errors were focused on software issues,³⁸ and computer viruses.^30,70 For example, orthography errors from Optical Character Recognition process were detected by the neural network (accuracy on random noises: 0.73),³³ while software issues that caused CDSS malfunctions were detected by the Naïve Bayes classifier (AUC: 0.7384).³⁸ In another study, neural networks were used to detect adversarial attacks, achieving an accuracy of 100% with chest x-ray images.⁷⁰

Semi-supervised learning methods

Only 2 studies (9.5%) used semi-supervised machine learning methods to detect use errors³⁶ and computer viruses.⁷⁰ Neural networks based autoencoders were used in both studies but evaluated with different performance measures. For detecting use errors in laboratory results, the autoencoder was shown to be highly effective, reporting a Youden’s Index (ie, sensitivity + specificity – 1) of 0.9999.³⁶ For computer viruses, a variational autoencoder was shown to achieve an accuracy of 85% in detecting adversarial attacks.⁷⁰

Statistical modeling methods

Statistical modeling methods were used in 17 studies, where use errors were most commonly investigated (n = 7, 41.2%),^{40,42,43,45,61,68,69} followed by various technical errors, including computer viruses (n = 5, 29.4%),^{30,41,49,50,67} software issues (n = 5, 29.4%),^{44,46–48,73} and data errors (n = 1, 5.9%).⁶⁹ One study investigated both use and technical errors (n = 1, 2.2%)⁶⁹; here, the traditional Shewhart statistical control process chart was adopted, achieving a sensitivity of 100% at 5% error rates when detecting missing laboratory test results, erroneous values, and duplicated records.⁶⁹

Examples of use errors included errors in the clinical text (misspelling),^40,42,43 errors in laboratory test results,^68,69 and incorrect measurements in EHRs.^45,61 A misspelling detection study examined the Shannon’s noisy channel model, which showed a high performance in 2 types of clinical notes: allergy entries (precision: 96.2%, recall: 92.7%, F measure: 94.4%, accuracy: 88.2%) and medication orders (precision: 90.0%, recall: 91.5%, F measure: 90.8%, accuracy: 81.5%).⁴² Another study using a combination of standard deviation values and linear regression reported high sensitivity values for detecting errors in height (97.91%) and weight measurements (90.87%).⁶¹

With technical issues, several studies demonstrated good performance of statistical modeling in detecting unauthorized access. For example, Boxwala et al predefined 26 features (eg, care visit match or patient ID match) and reported a sensitivity of 75.8% with a logistic regression model.³⁰ Then they adopted an integrated filtering method by leveraging previous features based on symbolic clustering and signature detection, achieving an AUC-ROC of 0.998.⁴¹ Another study that used a detection method based on access probability, treatment probability, responsibility, and reasons for access reported a precision of 99.7% (recall: 22.9%, F measure: 37.2%).⁶⁷

Statistical modeling methods have also been used to examine the detection of software issues such as CDSS malfunctions.^{44,46–48,73} One study used statistical process control c-charts to retrospectively examine errors in the firing of 226 CDSS rules over a 5-year period achieving a sensitivity of 95% (precision: 29%, F measure: 44%) in detecting knowledge management errors.⁴⁶ Another statistical modeling method called the Seasonal-Trend (STL) decomposition was studied to identify CDSS at Brigham and Women’s Hospital.⁴⁷ It outperformed the benchmark method of Multi-Process Dynamic Linear Model (AU-AMOC: 0.19), achieving an Area Under Curve (Activity Monitoring Operating Characteristic), AU-AMOC, of 0.32.⁷³

Rule-based methods

Seven studies examined rule-based methods. Of these, 6 were aimed at detecting a variety of use errors including errors in clinical notes, medication, and wrong-patient errors.^51–56 One academic medical center demonstrated the feasibility of monitoring clinician use of CPOE in real-time to detect and correct wrong-patient errors.¹⁴ Their retract-and-reorder measurement tool effectively identified 170 of 223 events as wrong-patient electronic orders (Positive Predictive Value, PPV, or precision = 76.2%). The study evaluated 2 interventions: an ID-verify alert (single-click confirmation of patient identity) which reduced wrong-patient electronic orders by 16%, while an active ID-reentry function (requiring active reentry of identifiers) achieved a 41% reduction. Another study focused on detecting prescribing errors due to the look-alike/sound-alike drug names.⁶² Rules to identify mismatches between a drug’s known indications and a patient’s active diagnoses were shown to be effective in detecting 69% of instances in which cycloserine was ordered or documented erroneously over a 7-year period. Another study demonstrated the effectiveness of rule-based approaches to detect and correct spelling errors in the emergency department (accuracy: 93.54%) and intensive care records (accuracy: 81.83%).⁵¹ With radiology reports, rule-based approaches were shown to be effective in detecting dictation errors about gender but not laterality errors (wrong side or site; precision 93% vs 26%).⁵⁴ Rule-based approaches were also shown to be effective in detecting network attacks on health sensor data (detection rate = 99.7%; detection time = 0.11ms).⁷²

DISCUSSION

Automated methods have the potential to enable more timely detection of emerging problems with HIT. However, the current literature is largely confined to studies demonstrating proof of concept of a variety of methods with limited evidence about their clinical usefulness. While sensitivity was commonly reported, the timeliness of detection was not examined in most studies making it difficult to assess the clinical usefulness of methods in real-world settings. The generalizability of methods is also not known as most studies were based on datasets from a single setting.

Applying automated methods to detect HIT safety problems

Rule-based methods appeared to be highly effective in detecting well-structured HIT problems such as use errors (eg, wrong-patient⁵² and prescribing errors⁵⁵). Statistical modeling tended to be applied to more complex technical errors, such as CDSS malfunctions,^46–48,73 based on an understanding of expected local data distributions. More recently, machine learning methods have been applied to detect a wide variety of use and technical errors with most studies focusing on unsupervised learning. When the application of supervised approaches is limited by the availability of large amounts of labeled data, transfer learning offers a potential solution. Here pre-trained models from other fields can be used as their parameters can be refined with smaller sets of labeled data.^75,76 For example, pre-trained models have been shown to be effective in detecting protected health information in EHRs.⁷⁵ Ultimately, a combination of automated methods may be needed to effectively detect HIT problems. Further studies are required to examine the efficacy, clinical usefulness, and timeliness of the different types of automated methods for the early detection of HIT problems. One possible strategy is to target the most safety-critical HIT problems such as those that have the potential to impact care delivery processes and patients on a large scale.¹⁰

Evaluating the performance of automated methods

Only a few studies examined the timeliness of detection which is a critical indicator for assessing real-world performance.^{44,47,48,52,69,72,73} Many studies were undertaken retrospectively using static data collected from HIT and did not dynamically simulate the onset of HIT problems. None of the machine learning methods reported time-response but 4 of the 6 statistical modeling studies examined the timeliness of detection for CDSS alerting issues. As HIT systems are dynamic and networked, future studies should examine the timeliness of detection to ensure automated methods are effective in real-world clinical settings.

We observed a high degree of heterogeneity in the datasets used to evaluate performance. Most studies were focused on detecting HIT problems involving an isolated HIT system at a single site. While a few studies were undertaken across multiple sites,^{31,35,42,51,66} none involved multiple HIT systems that more closely represented real-world use (eg, a combination of EMR, CPOE, and CDSS). One possible way to facilitate evaluation across multiple sites is via the use of a common data model (CDM). The Observational Medical Outcomes Partnership (OMOP) CDM from the Observational Health Data Sciences and Informatics is an example of a well-accepted CDM that focuses on transforming datasets in various formats into a common representation, that is, using the same terminologies, vocabularies, coding schemes.⁷⁷ The OMOP CDM is currently developing real-time analytic capabilities to detect safety problems with medical products that can potentially be extended to detecting HIT problems.^78,79 A CDM that facilitates collaboration and sharing between organizations with disparate IT implementations can support effective application of automated detection methods in real-world settings.

Limitations

This review has several limitations. We performed an extensive literature search of 6 different databases, all eligible studies over an 11.5-year period were included, and we also performed a hand search. However, we did not include the gray literature. It is thus possible that the automated methods we identified are not exhaustive. A second limitation relates to the heterogeneity in HIT systems, HIT problems, and performance measures that prevented quantitative examination of the efficacy, clinical usefulness, and timeliness of the different automated methods.

CONCLUSIONS

The use of automated methods to detect HIT problems is still emerging and evidence about their effectiveness in real-world settings is limited. There remain many opportunities to systematically apply and evaluate rule-based, statistical modeling and machine learning methods for real-time detection of the different problems with HIT that disrupt care delivery and pose risks to patient safety.

FUNDING

This research is supported by the Australian National Health and Medical Research Council Centre for Research Excellence in Digital Health (APP1134919). The funding source did not play any role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication.

AUTHOR CONTRIBUTIONS

FM conceptualized the study. DS, YW, and FM undertook the literature search. DS and YW performed data analysis and drafted the article. All authors participated in writing and revising the article. All aspects of the study (including design; collection, analysis, and interpretation of data; writing of the report; and decision to publish) were led by the authors.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

DATA AVAILABILITY

All data relevant to the study are included in the article or uploaded as online supplementary material. All data relevant to the analysis are reported in the article.