Skip to main content
NCBI home page
BMJ Open Quality logoLink to BMJ Open Quality
. 2023 Jan 31;12(1):e002143. doi: 10.1136/bmjoq-2022-002143

Impact of computerised provider order entry on the quality and quantity of clinical information included with investigation requests: an interrupted time series analysis

Daniel Weiand 1,✉, Caroline Cullerton 1, Robert Oxley 1, Chris J Plummer 2
PMCID: PMC9890764  PMID: 36720495

Abstract

Introduction

Relevant clinical information is vital to inform the analytical and interpretative phases of most investigations. The aim of this study is to evaluate the impact of implementation of computerised provider order entry (CPOE), featuring order-specific electronic order entry forms (eOEFs), on the quality and quantity of clinical information included with investigation requests.

Methods

The CPOE module of a commercially available electronic health record (Cerner Millennium) was implemented at a large, tertiary care centre. The laboratory information management system was interrogated to collect data on specimens sent for microbiological culture 1 year before implementation of CPOE (2018), immediately post implementation (2019) and 6 months post implementation (2020). An interrupted time series analysis was performed, using text mining, to evaluate the quality and quantity of free-text clinical information.

Results

In total, 39 919 specimens were collected from 16 458 patients. eOEFs were used to place 10 071 out of 13 735 orders in 2019 (73.3%), and 9155 out of 12 229 orders in 2020 (74.9%). No clinical details were included with 653 out of 39 919 specimens (1.6%), of which 22 (3.4%) were ordered using eOEFs. The median character count increased from 14 in 2018, to 41 in 2019, and 38 in 2020. An anti-infective agent was specified in 581 out of 13 955 requests (4.2%) in 2018; 5545 out of 13 735 requests (40.4%) in 2019; and 5215 out of 12 229 requests (42.6%) in 2020. Ciprofloxacin or piperacillin-tazobactam (Tazocin) were mentioned in the clinical details included with 421 out of 15 335 urine culture requests (2.7%), of which 406 (96.3%) were ordered using eOEFs. Subsequent detection of in vitro non-susceptibility led to a change in anti-infective therapy for five patients.

Conclusions

Implementation of CPOE, featuring order-specific eOEFs, significantly and sustainably improves the quality and quantity of clinical information included with investigation requests, resulting in changes to patient management that would not otherwise have occurred.

Keywords: Data Accuracy, Diagnostic Services, Electronic Health Records, Information technology, Laboratory medicine

WHAT IS ALREADY KNOWN ON THIS TOPIC

  • Relevant clinical information is vital to inform the analytical and interpretative phases of most investigations.

  • Differences in the design of computerised provider order entry (CPOE) systems, which are used to submit electronic investigation requests, may affect end user perceptions around system accessibility, flexibility and usability.

WHAT THIS STUDY ADDS

  • Our results show that CPOE featuring order-specific electronic order entry forms significantly and sustainable improves both the quality and quantity of clinical information included with investigation requests, resulting in changes to patient management that would not otherwise have occurred.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

  • More research is needed to understand how CPOE designs can be optimised to: benefit patient outcomes; leverage the potential of clinical decision support rules in guiding investigation utilisation; and safely expand the role of text mining and machine learning in the analysis, interpretation and repurposing of free-text clinical data.

Introduction

Electronic health record (EHR) systems are being implemented in many high-income and middle-income countries with the aim of improving the quality, safety and efficiency of healthcare.1 2 Commercially available EHRs can be cheaper to procure and maintain than locally developed electronic systems, which is likely to drive their ongoing adoption.2

EHRs support multiple healthcare functions, including creation and storage of patient notes, managing electronic prescriptions and patient-facing web portals.3 Widening access to EHRs is also driving greater adoption of computerised provider order entry (CPOE) systems, which are used by healthcare workers (HCWs) to submit diverse electronic requests, ranging from laboratory investigations,4–14 to imaging studies and radiological examinations,15–22 to health education videos,3 to drugs and medications.1 2 23–25 The proportion of hospitals using CPOE is not known.5 By leveraging their sophisticated search modules, CPOE systems can be set up to request practically all imaginable investigations.13

Clinical details included with orders inform the analytical and interpretative phases of most investigations. The provision of accurate and timely patient information is critical to investigation choice, result interpretation and follow-up.20 26 As a result of CPOE, traditional paper-based systems for submitting investigation requests are being superseded by electronic order entry forms (eOEFs), also known as ‘ask at order entry’ questions. When completing eOEFs, HCWs are presented with order-specific questions, requiring a response on an optional or mandatory basis. Individual eOEFs can be bundled into order sets, which colocate similar and related orderables. This change represents a significant opportunity for quality improvement, particularly given the many practical difficulties intrinsic to managing and maintaining paper-based ordering systems at scale.3

Concerns have been raised about the role of CPOE in the proliferation of investigation requests. Differences in CPOE choice architectures, including order set design, can have a dramatic impact on investigation ordering patterns. HCWs with EHR access order more investigations than their non-EHR counterparts27 and may become over-reliant on order sets, accepting bundled orders without making appropriate modifications.5 14

Aim

As the volume of investigation requests grows, there is increased scrutiny over the appropriateness of testing.10 Repetitive testing without explicit justification is not clinically appropriate.8 9 14 28

Knowledge of the impact of CPOE on different care processes is limited,2 and high-quality evidence of benefit is lacking.25 The ability of CPOE to improve communication of important clinical information has yet to be widely demonstrated.26

The aim of this study is to evaluate the impact of implementation of CPOE, featuring order-specific eOEFs, on the quality and quantity of clinical information included with laboratory investigation requests, concentrating on specimens sent for microbiological culture.

Materials and methods

Population and setting

In autumn 2019, the CPOE module of a commercially available EHR system, Cerner Millennium,29 was implemented at a hospital Trust accounting for 1.72 million patient contacts each year and including more than 1500 beds, offering the widest range of tertiary specialist care in the UK.

CPOE design and implementation

To prepare for implementation of CPOE, order-specific eOEFs were codesigned by a multidisciplinary team of professionals, including Cerner representatives, information technology specialists, clinicians and scientists, with targeted input from end users. During electronic order entry, free-text fields highlighted in yellow required a response, whereas free-text fields in white were optional (figure 1A).

Figure 1.

Figure 1

Open in a new tab

(A) Example of an order-specific eOEF for urine culture requests. (B) Example of a paper-based request form used for placing microbiology, virology and serology requests.

In order to avoid ‘over-alerting’ and its product, ‘alert fatigue’, pop-up boxes were avoided altogether in the design of eOEFs.2 5 8 30–33 Mandatory pull-down menus and pick lists were avoided to improve completion rates of free-text fields.

Investigations could be ordered using eOEFs in both inpatient and outpatient settings. Where appropriate, individual eOEFs were bundled together into order sets. Orderables were only preselected by default where this was approved by a multidisciplinary panel of subject matter experts. Clinical decision support (CDS) rules were introduced to facilitate duplicate checking of electronic orders.

Paper-based request forms continued to be accepted post implementation of CPOE, with all included clinical details being routinely transcribed to the laboratory information management system (LIMS) (figure 1B). To reject requests for microbiological culture solely on the basis of requesting method would almost certainly be considered unreasonable by both patients and HCWs. For example, electronic requesting of laboratory investigations requires a certain amount of infrastructure. Practically speaking, stickers need to be printed and applied to specimens. However, the EHR occasionally requires maintenance and sticker printers can malfunction. Completion of electronic request forms also requires easy access to a computer terminal, which may be considered impractical in certain clinical settings, like in theatre. Finally, electronic requesting is less well supported by Cerner Millennium in specific clinical settings, like the dialysis unit, which falls in the grey area between inpatient care and community care. In summary, although electronic requesting of investigations has certainly become very popular and comes with many advantages, precluding the use of paper request forms altogether has its own downsides.

Prior to implementation of CPOE, HCWs were invited to mandatory educational sessions where they were shown how to place electronic orders. HCWs were also instructed to remove paper-based request forms from clinical areas in order to encourage use of CPOE in the immediate post implementation period.

Data collection

The LIMS was interrogated to retrospectively collect data on specimens collected between 13 August 2018 and 15 October 2018 (1 year before implementation of CPOE); between 3 August 2019 and 15 October 2019 (immediately post implementation of CPOE); and between 10 January 2020 and 15 March 2020 (6 months post implementation of CPOE).

Data analysis

An interrupted time series analysis was performed to evaluate the impact of implementation of CPOE featuring eOEFs on the quality and quantity of clinical information included with requests for culture of blood, urine, wound swabs, cerebrospinal fluid (CSF) and drain fluid. Data analysis and tests for statistical significance were conducted in R,34 with figures produced using the ggplot235 or NHSRplotthedots36 packages. The Anhøj rules were used to analyse data over time for non-random variation.37 To analyse free-text clinical information included with investigation requests, text mining (text parsing and vectorisation) was performed using the tidytext package.38 This package was used to split free-text information into tokens, thereby facilitating further evaluation. For example, non-sensical free-text data were detected by means of comparison with common stop words,38 the 10 000 most common English words39 and words included in the Manual of Clinical Microbiology.40 Any remaining free-text data were manually reviewed to finally classify the data as non-sensical.

As there are no established standards available to assess the quality of clinical details included with investigation requests,18 41 text mining concentrated on the following quality metrics.

First, text mining was used to analyse all data for specific mention of anti-infectives for systemic use, which could be mentioned by brand name, generic name or drug class, as defined by WHO Anatomical Therapeutic Chemical/Defined Daily Dose (ATC/DDD) Index: Group J.42 Fuzzy matching of search terms was facilitated using the stringr package.43 Excluding patients under the care of urology, antimicrobial susceptibility testing (AST) for ‘second-line’ anti-infectives is only performed in limited circumstances, including specific mention of these agents in the clinical details included with investigation requests. Furthermore, to support antimicrobial stewardship and combat antimicrobial resistance, AST results for ‘second-line’ anti-infectives are only released from the LIMS to the EHR in limited circumstances, including specific mention of these agents in the clinical details included with investigation requests. Where specific anti-infectives were mentioned, LIMS and EHR data were retrospectively analysed to evaluate the impact on laboratory processing of relevant specimens and subsequent patient management.

Second, data included with all urine culture requests were analysed for specific mention of symptoms or clinical signs suggestive of cystitis, pyelonephritis or urethritis, as defined by Scottish Intercollegiate Guidelines Network (SIGN) guideline 8844 and the National Health Service.45

Exclusion criteria

This study excluded mortuary specimens and investigations requested by community-based general practitioners and other hospitals, as Cerner Millennium29 is not presently used in these clinical settings.

Patient and public involvement

In the course of this research, patient and public involvement was not actively sought as it was not considered feasible or appropriate.

Results

Patient demographics

Over the course of the study, 39 919 specimens were collected from 16 458 distinct patients, of which 8557 (52%) were female. The median age of patients was 55 years (IQR 27–73 years). No information was available on patients’ main comorbidities, ethnic grouping or socioeconomic status.

Specimen ordering method

Of the 39 919 distinct specimens, 13 955 (35%) were collected in 2018, 13 735 (34.4%) in 2019 and 12 229 (30.6%) in 2020. CPOE featuring order-specific eOEFs was implemented in 2019. In total, 20 693 out of 39 919 (51.8%) orders were placed using order-specific eOEFs. As mentioned, paper-based request forms continued to be accepted post implementation of CPOE. Immediately post implementation of CPOE in 2019, there were 3664 out of 13 735 specimens (26.7%) received with a paper-based request form, which remained stable 6 months later in 2020, when 3074 out of 12 229 specimens (25.1%) were received with a paper-based request form.

Specimen collection location

In total, specimens were collected in 107 locations. Of the 39 919 specimens, inpatient areas accounted for 35 899 (89.9%) and the top 10 locations accounted for 14 438 (36.2%). The emergency department made the highest number of requests, accounting for 3575 out of 39 919 (8.96%) specimens.

Specimen type information

Of the five different specimen types included in this study, blood cultures were the most common, accounting for 15 600 out of 39 919 specimens (39.1%), followed by urine specimens (15 335, 38.4%), wound swabs (7835, 19.6%), CSF (834, 2.1%) and drain fluid specimens (315, 0.8%).

Quantity of information included with specimen request forms

Implementation of CPOE had a significant impact on the total character count per request, and particularly the proportion of specimens received with few clinical details (defined as 1–9 characters). In 2018, 1 year before implementation of CPOE, 4384 out of 13 955 specimens (32.1%) included between 1 and 9 characters, compared with 272 out of 13 530 specimens (2%) in 2019, immediately post implementation, and 183 out of 12 084 specimens (1.5%) in 2020, 6 months post implementation (figure 2)

Figure 2.

Figure 2

Open in a new tab

Bar chart illustrating the character count of clinical details included with specimens, stratified by study year.

In total, 653 out of 39 919 specimens (1.6%) were received with no clinical details. Of these, 303 (46.4%) were requests submitted prior to implementation of CPOE, 328 (50.2%) were paper-based request forms submitted post implementation of CPOE and 22 (3.4%) were order-specific eOEFs.

For the remaining 39 266 non-blank request forms, the median character count of clinical details included with specimens was 31 (IQR 31). One year before implementation of CPOE, in 2018, the median character count was 14 (max 148, IQR 18). This increased significantly to 41 (max 171, IQR 31, p<0.05) in 2019, immediately post implementation, and was maintained at 38 (max 196, IQR 26, p<0.05) in 2020, 6 months post implementation. The mean character count of clinical details, stratified by study year, is shown in figure 3.

Figure 3.

Figure 3

Open in a new tab

Statistical process control chart illustrating the mean character count of clinical details included with specimens collected per 72-hour period, stratified by study year.

One year before implementation of CPOE, in 2018, the mean character count of clinical details included with request forms was 19.5 (95% CI 19.3 to 19.8). The mean character count increased significantly post implementation of CPOE, in 2019 and 2020, to 46.9 (95% CI 46.6 to 47.2, p<0.05). When paper-based request forms were used, post implementation of CPOE, in 2019 and 2020, the mean character count increased to 34.4 (95% CI 33.9 to 35.0, p<0.05). This may be explained by HCWs reflecting on the questions included with order-specific eOEFs, and developing a better understanding of the specimen type-specific clinical details that are of interest to medical laboratories (figure 4).

Figure 4.

Figure 4

Open in a new tab

Column chart illustrating the mean character count of clinical details included with specimen request forms, stratified by study year and specimen ordering method. Error bars illustrate the 95% CIs of the mean character count. CPOE, computerised provider order entry.

Quality of information included with specimen request forms

Non-sensical clinical details were included with 39 out of 39 266 (0.1%) non-blank investigation requests, of which 32 (82.1%) were pre-CPOE request forms; 4 (10.3%) were post-CPOE paper-based request forms; and 3 (7.6%) were post-CPOE eOEFs.

Free-text clinical details included with specimen requests were further analysed for specific quality metrics. First, at least one antimicrobial agent was mentioned in the clinical details included with 11 341 out of 39 919 specimens (28.4%). When stratified by study year, 581 out of 13 955 specimens (4.2%) collected in 2018 specifically mentioned an anti-infective. Post implementation of CPOE, request forms were significantly more likely to specifically mention an anti-infective, with 5545 out of 13 735 specimens (40.4%) collected in 2019, and 5215 out of 12 229 specimens (42.6%) collected in 2020 mentioning this clinical information (p<0.05).

Second, of the 15 335 specimens collected for urine culture, 7149 (46.6%) were accompanied by a request form specifically mentioning symptoms or clinical signs suggestive of cystitis, pyelonephritis or urethritis. When stratified by study year, 1664 out of 5414 specimens (30.7%) collected in 2018 were accompanied by a request form mentioning symptoms or signs of urinary tract infection. This increased significantly in 2019 and 2020, with requests mentioning symptoms or signs of urinary tract infection in 2975 out of 5436 specimens (54.7%) and 2510 out of 4485 specimens (56.0%), respectively (p<0.05). The impact of CPOE was similarly pronounced when results were stratified by specimen ordering method (figure 5).

Figure 5.

Figure 5

Open in a new tab

Column chart illustrating the percentage of urine culture requests specifically mentioning symptoms and clinical signs of urinary tract infection (UTI) in the free-text clinical details, stratified by study year and specimen ordering method. Error bars illustrate the 95% CIs of the percentage. CPOE, computerised provider order entry; ns, not significant.

Finally, clinical details included with urine culture requests were analysed for specific mention of so-called ‘second-line’ anti-infectives, including ciprofloxacin and piperacillin-tazobactam (Tazocin, Pfizer). Implementation of CPOE significantly increased the frequency with which HCWs mentioned current and/or intended use of specific ‘second-line’ anti-infective agents.

Ciprofloxacin was mentioned in 93 out of 15 335 (0.6%) urine culture requests, of which 5 (5.4%) were paper-based requests and 88 (94.6%) were post-CPOE eOEFs. Culture led to isolation of a urinary tract pathogen from 14 out of 82 (17.1%) non-urology urine specimens. AST was appropriately performed in 13 out of 14 (92.95%) cases. In vitro non-susceptibility to ciprofloxacin led to a change in patient management in 2 out of 13 (15%) cases.

Tazocin was mentioned in the clinical details included with 328 out of 15 335 (2.1%) urine culture requests, of which 10 (3%) were paper-based requests and 318 (97%) were post-CPOE eOEFs. Culture led to isolation of a urinary tract pathogen from 58 out of 305 (81%) non-urology urine specimens. AST was appropriately performed in 57 out of 58 (98.3%) cases. In vitro non-susceptibility to Tazocin led to a change in patient management in 3 out of 57 (5.3%) cases.

Discussion

The results show that implementation of CPOE had a significant and sustained impact on both the quality and quantity of clinical information included with investigation requests. Furthermore, the additional clinical information gathered by means of eOEFs had a positive impact on both the analytical and interpretative phases of specimen processing, directly resulting in changes to patient management that would not otherwise have occurred.

Related research

Concerns have been raised in the literature around the potential for misuse of free-text data entry fields in EHRs.31 For example, Dekarske et al evaluated the appropriateness of 22 HCWs’ over-ride reasons for 1829 electronic alerts. They reported that 13.9% of alert over-ride reasons included non-sensical free-text responses (eg, ‘asdf’ or ‘—’). Of note, 83% of the non-sensical responses were entered by 2 of the 22 subjects.31 However, this study shows that implementation of CPOE significantly improved the richness of clinical information included with investigation requests. Investigations ordered using eOEFs were significantly less likely to be blank or include non-sensical information. This reassuring finding adds to the existing evidence that, compared with pull-down menus and pick lists, free-text data entry fields can facilitate more detailed communication and help mitigate the risk of selection errors.2 7 31 Indeed, CPOE systems designed with mandatory pull-down menus and pick lists have been shown to reduce completion rates of free-text fields.6 Grisson et al maintain that virtually all complex order entry systems must tolerate some level of free-text data entry to be clinically efficient.13 Given the large scale of this study, it is unlikely that the results were disproportionately affected by contributions from only a few individuals.

Delivering high-quality, safe and reliable patient care involves influencing the decisions that HCWs make. Traditional interventions, including new evidence, guidelines and education, can be relatively ineffective at influencing HCW behaviour.8 25 41 Timelines for realising the benefits of new technologies are also often greatly underestimated.2 This study demonstrates that implementation of CPOE, featuring order-specific eOEFs, helps nudge the bedside clinical practice of HCWs, leading to significant improvements in both the quality and quantity of clinical information included with investigation requests.

Few published studies have evaluated the real-world impact of implementation of CPOE.7 Understandably, many of the earlier studies assessed the impact of locally developed systems, rather than modern, commercially available EHRs with CPOE functionality. Electronic ordering can save staff time in trying to decipher handwriting; facilitate faster transfer of information; reduce duplication; and improve accessibility of information.2 There is also evidence that requests submitted by CPOE are significantly more likely than paper-based requests to contain useful information, including clinical questions and information on prior diagnoses.4 15 17 26 41 Furthermore, computerised ordering systems have been shown to reduce healthcare expenditure; decrease the number of tests per admission46; and improve test turnaround time.26

Limitations of this study

As with any observational before-and-after design, the results may have been influenced by other unmeasured factors.25 Due to differences in clinical practice and EHR configuration, this study’s findings may not be generalisable to other hospitals. Furthermore, this study focused on laboratory investigation requests, concentrating on specimens sent for microbiological culture. However, CPOE systems are also used to submit electronic requests for other types of orderables, including imaging studies, drugs and medications. This study’s findings may not be generalisable to these other types of orderables.

Depending on CPOE design, use of eOEFs may require more time than completion of paper-based request forms, which was not evaluated in this study. However, other time savings inherent to EHRs may offset the time spent on CPOE, including less time spent looking for paper-based forms and charts.5

Other studies have reported on the impact of CPOE on the number, cost and appropriateness of investigations requested,2 46 as well as the costs of staffing for EHR and CPOE development, implementation, maintenance and troubleshooting.5 However, an evaluation of these variables was not within the scope of the present study.

This study did not evaluate the effect of different CPOE designs on HCWs’ clinical and laboratory practice. The implementation of new technologies risks the inadvertent introduction of new, often unanticipated clinical hazards. For example, this study did not evaluate the role of CPOE in facilitating the ordering of investigations on the wrong patient, or on electronic medical records propagating copy and paste errors.47

Nor did this study evaluate the effect of different CPOE designs on perceptions around system accessibility, flexibility and usability.1 2 Lack of system flexibility can result from systems attempting to improve safety by making certain tasks compulsory or sequential.2 Mandatory documentation can lead to unintended behaviours, including proliferation of local adaptations and workarounds to usual working practices.2 31 Compulsory tasks and documentation, which are perceived as clinically irrelevant or unnecessarily repetitive, can cause frustration among end users.31 Concerns have also been raised about the impact of CPOE on HCWs’ autonomy in decision-making.5

Further research opportunities

More research is needed to understand how EHR design impacts on patient outcomes,20 and how CPOE choice architectures can be further optimised.2 For example, defaults are one of the most widely employed tools in CPOE design, bearing considerable influence on decision-making.48 When all laboratory tests included in an order set are preselected, HCWs order significantly more investigations, thereby increasing the cost of admission.49 Other CPOE functions that are known to impact on ordering practices include order sets; electronic alerts and warnings; and publicising laboratory test charges.28 31 46

Further research is also required to explore the wider potential of more advanced CDS rules in guiding investigation utilisation.30 For example, alerts and warnings could be triggered on the basis of clinical information included with eOEFs,31 perhaps assisting in the detection of incorrectly placed investigation requests.50 Some CPOE systems already have the ability to integrate orders with pharmacy and laboratory data, to make helpful recommendations and alert requestors of significant issues.8

Beyond CDS, additional research is required to safely expand the role of text mining and machine learning (ML) in the analysis, interpretation and repurposing of free-text clinical data. For example, specific eOEF fields could be automatically prepopulated with clinical information documented elsewhere in the EHR.20 Insights gained through ML algorithms could also be used to optimise healthcare workflow and reduce costs.18 Although not currently in routine use, association-based recommender systems have been described, which automatically analyse inpatient data to inform CDS rules and electronic order set contents.51

Conclusions

This study shows that implementation of CPOE, featuring order-specific eOEFs, significantly and sustainably improves both the quality and quantity of clinical information included with investigation requests. This additional information directly impacts on healthcare workflow, resulting in changes to patient management that would not otherwise have occurred.

Acknowledgments

The authors wish to acknowledge the contributions of the following colleagues employed by the Newcastle upon Tyne Hospitals NHS Foundation Trust: Mr John Clinton, Ms Deborah Osborne and Ms Sarah Evans. The authors have not had any writing assistance.

Footnotes

Twitter: @send2dan

Contributors: DW: conceptualisation, methodology, software, formal analysis, investigation, resources, data curation, writing—original draft, writing—review and editing, visualisation, project administration, guarantor. CC: writing—review and editing. RO: methodology, investigation, resources, writing—review and editing. CJP: resources, writing—review and editing, supervision. All authors read and approved the final manuscript.

Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: None declared.

Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review: Not commissioned; externally peer reviewed.

Author note: DW: clinical informatics lead for the Integrated Laboratory Medicine (ILM) Directorate at Newcastle upon Tyne Hospitals NHS Foundation Trust. CJP: chief clinical information officer (CCIO) at Newcastle upon Tyne Hospitals NHS Foundation Trust.

Data availability statement

Data are available upon reasonable request. Data are not publicly available. Certain data may be available upon reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The privacy rights of all subjects have been observed throughout. This report is a summary of a quality and service improvement (QSI) that used anonymous patient data, which are routinely collected at Newcastle upon Tyne Hospitals NHS Foundation Trust, and no further ethical approval or patient consent was required.

References

  • 1.Chan J, Shojania KG, Easty AC, et al. Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual Saf 2011;20:932–40. 10.1136/bmjqs.2010.050021 [DOI] [PubMed] [Google Scholar]
  • 2.Cresswell KM, Bates DW, Williams R, et al. Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two “early adopter” hospitals. J Am Med Inform Assoc 2014;21:e194–202. 10.1136/amiajnl-2013-002252 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Wojcik L. Developing an interface to order and document health education videos in the electronic health record. Med Ref Serv Q 2015;34:359–69. 10.1080/02763869.2015.1052700 [DOI] [PubMed] [Google Scholar]
  • 4.Cuda J, Seigh L, Clark K, et al. Utilizing computerized provider order entry (CPOE) to reduce the garbage in garbage out effect in the cytology laboratory. J Am Soc Cytopathol 2016;5:S85. 10.1016/j.jasc.2016.07.195 [DOI] [Google Scholar]
  • 5.Baron JM, Dighe AS. Computerized provider order entry in the clinical laboratory. J Pathol Inform 2011;2:35. 10.4103/2153-3539.83740 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Georgiou A, Prgomet M, Toouli G, et al. What do physicians tell laboratories when requesting tests? A multi-method examination of information supplied to the microbiology laboratory before and after the introduction of electronic ordering. Int J Med Inform 2011;80:646–54. 10.1016/j.ijmedinf.2011.06.003 [DOI] [PubMed] [Google Scholar]
  • 7.Georgiou A, Williamson M, Westbrook JI, et al. The impact of computerised physician order entry systems on pathology services: a systematic review. Int J Med Inform 2007;76:514–29. 10.1016/j.ijmedinf.2006.02.004 [DOI] [PubMed] [Google Scholar]
  • 8.Henricks WH. Electronic health records and their implications and opportunities for laboratories. In: Clinical \textlessh\textgreaterLaboratory\textless/h\textgreater Management. 2nd ed. Washington, DC, USA: ASM Press, 2014: 918–32. [Google Scholar]
  • 9.Munigala S, Jackups RR, Poirier RF, et al. Impact of order set design on urine culturing practices at an academic medical centre emergency department. BMJ Qual Saf 2018;27:587–92. 10.1136/bmjqs-2017-006899 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Rubinstein M, Hirsch R, Bandyopadhyay K, et al. Effectiveness of practices to support appropriate laboratory test utilization. Am J Clin Pathol 2018;149:197–221. 10.1093/ajcp/aqx147 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Schreiber R, Sittig DF, Ash J, et al. Orders on file but no labs drawn: investigation of machine and human errors caused by an interface idiosyncrasy. J Am Med Inform Assoc 2017;24:958–63. 10.1093/jamia/ocw188 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Forest SK, Shirazi M, Wu-Gall C, et al. The impact of an electronic ordering system on blood bank specimen rejection rates. Am J Clin Pathol 2017;147:105–9. 10.1093/ajcp/aqw204 [DOI] [PubMed] [Google Scholar]
  • 13.Grisson R, Kim JY, Brodsky V, et al. A novel class of laboratory middleware. promoting information flow and improving computerized provider order entry. Am J Clin Pathol 2010;133:860–9. 10.1309/AJCPCVT30YEMRKRY [DOI] [PubMed] [Google Scholar]
  • 14.Leis B, Frost A, Bryce R, et al. Altering standard admission order sets to promote clinical laboratory stewardship: a cohort quality improvement study. BMJ Qual Saf 2019;28:846–52. 10.1136/bmjqs-2018-008995 [DOI] [PubMed] [Google Scholar]
  • 15.Pevnick JM, Herzik AJ, Li X, et al. Effect of computerized physician order entry on imaging study indication. J Am Coll Radiol 2015;12:70–4. 10.1016/j.jacr.2014.07.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Georgiou A, Prgomet M, Markewycz A, et al. The impact of computerized provider order entry systems on medical-imaging services: a systematic review. J Am Med Inform Assoc 2011;18:335–40. 10.1136/amiajnl-2010-000043 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Alkasab TK, Alkasab JR, Abujudeh HH. Effects of a computerized provider order entry system on clinical histories provided in emergency department radiology requisitions. J Am Coll Radiol 2009;6:194–200. 10.1016/j.jacr.2008.11.013 [DOI] [PubMed] [Google Scholar]
  • 18.Assad WA, Topaz M, Tu J, et al. The application of machine learning to evaluate the adequacy of information in radiology orders. 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); Kansas City, MO. 10.1109/BIBM.2017.8217668 [DOI] [Google Scholar]
  • 19.Bhayana R, Wang CD, Menezes RJ, et al. Optimising after-hours workflow of computed tomography orders in the emergency department. BMJ Open Qual 2020;9:e000969. 10.1136/bmjoq-2020-000969 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Lacson R, Laroya R, Wang A, et al. Integrity of clinical information in computerized order requisitions for diagnostic imaging. J Am Med Inform Assoc 2018;25:1651–6. 10.1093/jamia/ocy133 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Trinh TW, Ip I, Healey MJ, et al. An electronic health record order entry-enabled educational intervention is not effective in reducing STAT inpatient radiology orders. J Am Coll Radiol 2019;16:1018–26. 10.1016/j.jacr.2019.01.033 [DOI] [PubMed] [Google Scholar]
  • 22.Kruger JF, Chen AH, Rybkin A, et al. Displaying radiation exposure and cost information at order entry for outpatient diagnostic imaging: a strategy to inform clinician ordering. BMJ Qual Saf 2016;25:977–85. 10.1136/bmjqs-2015-004242 [DOI] [PubMed] [Google Scholar]
  • 23.Georgiou A, Ampt A, Creswick N, et al. Computerized provider order entry -- what are health professionals concerned about? A qualitative study in an Australian hospital. Int J Med Inform 2009;78:60–70. 10.1016/j.ijmedinf.2008.09.007 [DOI] [PubMed] [Google Scholar]
  • 24.Georgiou A, Prgomet M, Paoloni R, et al. The effect of computerized provider order entry systems on clinical care and work processes in emergency departments: a systematic review of the quantitative literature. Ann Emerg Med 2013;61:644–53. 10.1016/j.annemergmed.2013.01.028 [DOI] [PubMed] [Google Scholar]
  • 25.Bourdeaux CP, Davies KJ, Thomas MJC, et al. Using “nudge” principles for order set design: a before and after evaluation of an electronic prescribing template in critical care. BMJ Qual Saf 2014;23:382–8. 10.1136/bmjqs-2013-002395 [DOI] [PubMed] [Google Scholar]
  • 26.Georgiou A, Lang S, Rosenfeld D, et al. The use of computerized provider order entry to improve the effectiveness and efficiency of coagulation testing. Arch Pathol Lab Med 2011;135:495–8. 10.5858/2010-0286-SO.1 [DOI] [PubMed] [Google Scholar]
  • 27.Hakim I, Hathi S, Nair A, et al. Electronic health records and the frequency of diagnostic test orders. Am J Manag Care 2017;23:e16–23. 10.1111/maq.12375 [DOI] [PubMed] [Google Scholar]
  • 28.Krasowski MD, Chudzik D, Dolezal A, et al. Promoting improved utilization of laboratory testing through changes in an electronic medical record: experience at an academic medical center. BMC Med Inform Decis Mak 2015;15:11. 10.1186/s12911-015-0137-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Cerner . Cerner products. 2021. Available: www.cerner.com/Products
  • 30.Szlosek D, Haydar SA, Williams RJ, et al. The impact of an electronic best practice advisory on brain computed tomography ordering in an academic emergency department. Am J Emerg Med 2017;35:1776–7. 10.1016/j.ajem.2017.05.023 [DOI] [PubMed] [Google Scholar]
  • 31.Dekarske BM, Zimmerman CR, Chang R, et al. Increased appropriateness of customized alert acknowledgement reasons for overridden medication alerts in a computerized provider order entry system. Int J Med Inform 2015;84:1085–93. 10.1016/j.ijmedinf.2015.09.001 [DOI] [PubMed] [Google Scholar]
  • 32.Ojeleye O, Avery A, Gupta V, et al. The evidence for the effectiveness of safety alerts in electronic patient medication record systems at the point of pharmacy order entry: a systematic review. BMC Med Inform Decis Mak 2013;13:69. 10.1186/1472-6947-13-69 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Barry C, Kaufman S, Feinstein D, et al. Optimization of the order menu in the electronic health record facilitates test patterns consistent with recommendations in the choosing wisely initiative. Am J Clin Pathol 2020;153:94–8. 10.1093/ajcp/aqz134 [DOI] [PubMed] [Google Scholar]
  • 34.R Core Team . R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2021. Available: https://www.R-project.org/ [Google Scholar]
  • 35.Wickham H. ggplot2: elegant graphics for data analysis. Springer-Verlag New York; 2016. Available: https://ggplot2.tidyverse.org [Google Scholar]
  • 36.Reading C, Wellesley-Miller S, Turner Z, et al. NHSRplotthedots: draw XmR charts for NHSE/i making data count programme. 2021. Available: https://nhs-r-community.github.io/NHSRplotthedots/
  • 37.Anhøj J, Olesen AV. Run charts revisited: a simulation study of run chart rules for detection of non-random variation in health care processes. PLoS One 2014;9:e113825. 10.1371/journal.pone.0113825 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Silge J, Robinson D. Tidytext: text mining and analysis using tidy data principles in R. JOSS 2016;1:37. 10.21105/joss.00037 [DOI] [Google Scholar]
  • 39.first20hours . 10,000 most common English words. Available: https://github.com/first20hours/google-10000-english [Accessed 10 Apr 2022].
  • 40.Versalovic J, Carroll KC, Funke G, et al. Manual of clinical microbiology. 10th ed. American Society for Microbiology Press, 2011. [Google Scholar]
  • 41.Davies S, George A, Macallister A, et al. “Its’s all in the history”: a service evaluation of the quality of radiological requests in acute imaging. Radiography (Lond) 2018;24:252–6. 10.1016/j.radi.2018.03.005 [DOI] [PubMed] [Google Scholar]
  • 42.WHO . ATC/DDD index: group J - general antiinfectives for systemic use. 2021. Available: https://www.whocc.no/atc_ddd_index/?code=J&showdescription=no [Accessed 22 Dec 2021].
  • 43.Wickham H. Stringr: simple, consistent wrappers for common string operations. 2022. Available: https://CRAN.R-project.org/package=stringr
  • 44.Health Improvement Scotland . SIGN guideline 88: management of suspected bacterial urinary tract infection in adults; 2012.
  • 45.NHS . Cystitis. 2021. Available: https://www.nhs.uk/conditions/cystitis/ [Accessed 22 Dec 2021].
  • 46.Nightingale PG, Peters M, Mutimer D, et al. Effects of a computerised protocol management system on ordering of clinical tests. Qual Health Care 1994;3:23–8. 10.1136/qshc.3.1.23 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Wong BM, Etchells EE. Improving communication of critical laboratory results: know your process. BMJ Qual Saf 2012;21:624–6. 10.1136/bmjqs-2012-001272 [DOI] [PubMed] [Google Scholar]
  • 48.Jachimowicz JM, Duncan S, Weber EU, et al. When and why defaults influence decisions: a meta-analysis of default effects. Behav Public Policy 2019;3:159–86. 10.1017/bpp.2018.43 [DOI] [Google Scholar]
  • 49.Probst CA, Shaffer VA, Chan YR. The effect of defaults in an electronic health record on laboratory test ordering practices for pediatric patients. Health Psychol 2013;32:995–1002. 10.1037/a0032925 [DOI] [PubMed] [Google Scholar]
  • 50.Rough K, Dai AM, Zhang K, et al. Predicting inpatient medication orders from electronic health record data. Clin Pharmacol Ther 2020;108:145–54. 10.1002/cpt.1826 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Chen JH, Podchiyska T, Altman RB. OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records. J Am Med Inform Assoc 2016;23:339–48. 10.1093/jamia/ocv091 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data are available upon reasonable request. Data are not publicly available. Certain data may be available upon reasonable request.

Articles from BMJ Open Quality are provided here courtesy of BMJ Publishing Group

RESOURCES