Retrospective analysis of the impact of electronic medical record alerts on low value care in a pediatric hospital

Abstract

Objectives

Hospital costs continue to rise unsustainably. Up to 20% of care is wasteful including low value care (LVC). This study aimed to understand whether electronic medical record (EMR) alerts are effective at reducing pediatric LVC and measure the impact on hospital costs.

Materials and Methods

Using EMR data over a 76-month period, we evaluated changes in 4 LVC practices following the implementation of EMR alerts, using time series analysis to control for underlying time-based trends, in a large pediatric hospital in Australia. The main outcome measure was the change in rate of each LVC practice. Balancing measures included the rate of alert adherence as a proxy measure for risk of alert fatigue. Hospital costs were calculated by the volume of LVC avoided multiplied by the unit costs. Costs of the intervention were calculated from clinician and analyst time required.

Results

All 4 LVC practices showed a statistically significant reduction following alert implementation. Two LVC practices (blood tests) showed an abrupt change, associated with high rates of alert adherence. The other 2 LVC practices (bronchodilator use in bronchiolitis and electrocardiogram ordering for sleeping bradycardia) showed an accelerated rate of improvement compared to baseline trends with lower rates of alert adherence. Hospital savings were $325 to $180 000 per alert.

Discussion and Conclusion

EMR alerts are effective in reducing pediatric LVC practices and offer a cost-saving opportunity to the hospital. Further efforts to leverage EMR alerts in pediatric settings to reduce LVC are likely to support future sustainable healthcare delivery.

Introduction

Across the globe, healthcare expenditure continues to grow unsustainably, accounting for a higher proportion of spending each year.¹ Despite this, a recent study of Organisation of Economic Co-operation and Development (OECD) countries demonstrated that 15%-20% of healthcare expenditure is wasteful.² This includes low value care (LVC); tests, treatments, and procedures that add little benefit compared to their cost or potential for harm.³

Understanding how to effectively reduce unnecessary testing is critically important given the need for interventions themselves to not further contribute to wasteful spending. A systematic review of 64 studies in pediatric radiology and pathology found greater effect when multifaceted strategies are employed.⁴ Interventions reported included resource-intensive new programs of care, new clinical guidelines and processes, audit and feedback tools, education, and system changes. With frequent junior staff rotations in most hospitals, many of these interventions require sustained effort and resources to be effective. System changes, such as electronic medical record (EMR) alerts, may offer a more sustainable method of impacting ordering behaviors; however, effectiveness of EMR alerts in isolation has not yet been demonstrated in the pediatric literature.

In adult medicine, observational and experimental studies have reported success in using EMR alerts which direct clinicians away from ordering tests and treatments which are considered LVC. Whilst many studies have short follow-up periods (up to 2 years), one study reported the impact of alert removal 3 years after successful behavior change was achieved and saw a rise in inappropriate ordering, suggesting EMR alerts may play an important role in sustainable practice change.⁵

An important consideration in alert implementation is the risk of alert fatigue. Frequently over-ridden alerts can contribute to clinician frustration and a tendency to ignore all alerts, even those which present critical safety information.⁶ Most studies report only the relative reduction of the LVC practice in question without evaluating clinician behavior in response to the alert. Alerts can be designed to require different levels of interaction from clinicians. Educational messages can be bypassed without justification, some alerts require the clinician to take action, and others present a hard stop preventing the clinician from progressing at all. A single pre-post study in the United States of radiology orders across multiple hospitals (n = 98 894) showed users ignored 98.9%, modified 1.1%, and cancelled 0.03% of orders in response to alerts, but this differed by type of alert.⁷ Alerts that required action had a 10-fold higher rate of modification (8.1% vs 0.7%, P < .0001) or cancellation (0.2% vs 0.02%) than those that did not require action. Thus, although an overall reduction in ordering practice was seen, the high rate of alert bypassing suggests that a unidimensional evaluation of outcome such as LVC practice change alone is insufficient. In a 2015 study, a hard stop for duplicate laboratory testing which required clinicians to call the lab for approval was compared to an alert that could be bypassed. The hard stop was significantly more effective (reduction in LVC of 92.3% vs 42.6%, respectively, P < .0001) with greater cost savings ($16.08/alert compared to $US3.52/alert).⁸ However, the impact on clinicians and the costs on their time was not measured. Given a phone call introduces a time-consuming extra step in the workload of busy clinicians, such a hard stop must be balanced against the risk of clinician frustration and burnout. With EMR alerts now an available form of workflow disruption, it is likely more and more EMR alerts will be introduced over time. The makes the evaluation of the effectiveness and impact on clinicians critical to avoid the escalating risk of alert fatigue or clinician burnout.

Other EMR interventions have included simple reconfiguration of EMR order lists such as reduction in “Complete Blood Count (CBC)-differential” ordering simply by placing it lower down the list of possible orders presented to physicians.⁹ This simple reconfiguration was superior to an education campaign in reducing rates of CBC-differential ordering.

Between 2018 and 2020, we introduced 4 EMR alerts into a large Australian pediatric hospital to support reduction of identified LVC practices. This provided an opportunity to evaluate the impact of EMR interventions on pediatric care and costs.

Aims

We aimed to measure the impact of 4 EMR alerts for reducing LVC practices introduced at the Royal Children’s Hospital (RCH), a large pediatric teaching hospital in Melbourne, Australia. Specifically, we aimed to:

1. Measure baseline rates of 4 LVC practices;

2. Co-design (with clinicians and EMR experts) EMR alerts to direct away from each LVC practice, with appropriate inclusion and exclusion criteria, informed by knowledge of clinician workflows;

3. Measure the impact of each EMR alert on the relevant LVC practice using time series analysis to account for underlying time-based trends;

4. Analyze the rate of each EMR alert firing over time; and

5. Calculate the cost implications from the hospital perspective.

Methods

Context

The RCH is the major specialist pediatric hospital in Victoria, Australia’s second most populous state, providing secondary and tertiary care to children from Victoria and neighboring states. In 2018, the RCH received philanthropic funding to identify and address LVC. A governance committee was created with broad hospital representation (clinicians, executive, laboratory, pharmacy, and radiology). Frontline clinicians were encouraged to share LVC practices occurring in the hospital with the committee through departmental meetings, an online web form, and hospital-wide communications. Through this process, 4 LVC practices were identified and prioritized:

1. Full iron studies for investigation of iron deficiency anemia when ferritin alone would provide sufficient information.¹⁰ In this instance, ordering full iron studies requires more blood from the child, costs more and may create confusion in interpreting results, with low serum iron (which reflects iron intake over the past 24 h) mistakenly being interpreted as iron deficiency. Misinterpretation can lead to over treatment with iron supplements which have a high rate of side effects, most commonly constipation and gastrointestinal irritation.

2. Bronchodilator medication for infants with bronchiolitis.^11–14 Bronchiolitis is a viral chest infection in infants and the commonest reason for admission to pediatric hospitals. Bronchodilators are commonly used to treat children with asthma, but large systematic reviews have shown there is no benefit in infants with bronchiolitis. Administering the medication can be difficult in an unwell child as a spacer needs to be held over the infant’s mouth, time-consuming, and lead to unnecessary side effects such as tachycardia and irritability.

3. Full thyroid function tests (TFTs) for patients on thyroxine or for screening when TSH (thyroid stimulating hormone) and T4 (thyroxine) provide sufficient information.¹⁵ Whilst adding on additional tests incurs little cost, the extra results can create confusion and further unnecessary testing.

4. Electrocardiogram (ECG) ordering for children with sleeping bradycardia.^16,17 There is a natural physiological decrease in heart rate when children are relaxed and sleeping. Our hospital charts do not account for arousal status and are designed to detect deterioration in the awake child. Thus, overnight there are many calls to junior staff for bradycardia during sleep with a tendency to order unnecessary ECGs. This disturbs the sleeping child and uses substantial hospital resources in terms of both taking and reporting the ECG. A Clinical Practice Guideline (CPG) was developed which includes new lower limits of normal heart rates appropriate for children during sleep and advises against unnecessary investigations.

Intervention

Small working groups comprised of clinicians and EMR experts were formed for each LVC practice and a project manager was engaged. These were tasked with identification of the drivers of behavior and design of interventions.³ EMR alerts were carefully considered for rules (inclusion and exclusion criteria), timing in the workflow, wording, and action required. Hard stops were considered inappropriate given the lack of significant safety implications however actionable alerts were preferred where possible.

EMR alerts implemented:

1. Iron studies: An alert was introduced in December 2018 to fire when a clinician enters “iron studies” in the order panel (Figure 1A). Instructional text advises against the need for iron studies and an order for ferritin is queued by default. The user can choose to (1) accept the ferritin order or (2) over-ride the alert, decline the ferritin and continue with iron studies. Exclusion criteria prevents the alert firing for users logged in to the renal department or patients with inflammatory bowel disease or cystic fibrosis on their problem list as these conditions impact ferritin interpretation and require the full panel. In addition to the alert, “Iron studies” was removed from departmental preference lists and clinician order sets. The alert was otherwise implemented in isolation without an education or communication campaign.

2. Bronchodilators in bronchiolitis—An alert was introduced on February 4, 2019 to fire for all infants under the age of 12 months, regardless of diagnosis, if a clinician in the emergency department enters an order for a bronchodilator (Figure 1B). The alert includes brief instructional text and defaults to “remove.” If the clinician choses to “keep,” an acknowledge reason is requested. The alert was accompanied by an education campaign, and the roll out of a department-level audit-feedback tool over the first year of implementation.

3. Thyroid function tests: An alert was introduced on August 28, 2020, to fire when a clinician orders TFTs or T3 in a patient without a thyroid-related problem on their problem list (Figure 1C). The alert contains brief instructional text and defaults to removing T3 from the order. The clinician can choose to accept or change the alert to ‘keep’. In addition to the alert, TFTs were removed from order sets and preference lists.

4. ECG for bradycardia. From July 21, 2021, an educational banner replaced the ‘rapid review or medical emergency team response’ alert between 8 pm and 6 am if the child’s heart rate sits above the acceptable limit as per the RCH CPG (Figure 1D). The banner directs clinicians to the CPG and displays both the current heart rate and the acceptable lower limit as per the CPG. If a clinician attempts to order an ECG for a patient with a heart rate between these limits, an alert fires (Figure 1E). The alert defaults to removing the ECG. These EMR changes were pre-dated by the CPG in November 2020 which was broadly promoted to junior medical staff.

Figure 1.

EMR alerts. (A) Iron deficiency alert. (B) Bronchodilator alert. (C) Thyroid function alert. (D) Banner on Criteria Breach. (E) ECG alert. © 2023 Epic Systems Corporation.

Time series analysis

We extracted EMR data between April 30, 2016 (when the EMR went live), and August 30, 2022 (76 months) for each LVC practice. The interventions were introduced in sequential order (Figure 2) due to resource constraints in designing and building the alerts rather than by intentional design. Each alert was designed, introduced, and evaluated independently.

Figure 2.

Timeline.

For each LVC practice the denominator was the total population of interest and the numerator was the LVC measure (Appendix S2).

Descriptive statistics were used to compare the differences in proportions of each LVC practice in the pre-intervention period, compared to the post-intervention period.

A time series analysis was performed on aggregated monthly count data for the LVC practice of interest (numerator), offset by the size of the total target population each month (denominator). Poisson regression analysis was performed to test 3 potential impacts of each intervention: immediate change in behavior (step change), accelerated change of behavior (slope change), and a combination of both (step and slope change). The intervention for each LVC practice was specified by the date of the EMR alert implementation. Models included: a constant, a baseline slope to control for secular trends and terms estimating changes in the level (step) and slope of outcome rates. The following equation was used: Yt = B0 + B1T + B2Xt + B3TXt, where Y = the LVC practice, B0 represents the baseline level at the study start, B1 is the change in outcome each time unit increase (pre-intervention trend), B2 is the level change (step effect), and B3 is the slope change following the intervention.¹⁸ Sensitivity analysis was performed as part of robustness checks to allow for autocorrelation and seasonality, given our junior doctors rotate every 3 months.¹⁸ The model of best fit was chosen using Akaike information criterion and Bayesian information criterion.

Analyses were conducted using STATA statistical software version 16 (TX, United States).

Balancing measures

For each alert, we extracted the total number of times each alert was activated, and the action taken (accepted or over-ridden). If the alert was over-ridden (ie, the ordering clinician choses to “keep” the original LVC practice), or the BPA (best practice advisory) was dismissed/cancelled, this was considered an over-ride. Rates of each alert activation were calculated using the target population as a denominator to account for changing patient populations, particularly in light of the impact of the Cornovirus-19 pandemic on patient presentations over time. The rates of the alert being both activated and actioned were calculated for the first 6 months and compared to the final 6 months of the study period. Descriptive statistics (chi-square) were used to analyze whether there was any statistically significant change over the 2 time periods. A guide as to the effectiveness of the alert was created a priori by the authors and was then used to interpret findings for each alert (Figure 3).

Figure 3.

Interpretation of effectiveness of alert based on impact on LVC practice. Abbreviation: LVC = low value care.

Cost analysis

Costs impacts were calculated, using a hospital perspective, accounting for the direct costs of the LVC practice. Societal costs (such as patient out of pocket costs and caregiver burden) were not included as these were not expected to be significant and would be difficult to accurately capture. No comparison was made between healthcare outcomes. We have taken this approach based on the assumption that the net impact on patient health and quality of life is zero.

Costs were calculated based on the volume of each practice between the time of the intervention until the end of the study period (August 2022) multiplied by the unit cost of each. The difference in costs was calculated by the cost of the observed volumes compared to the counterfactual estimate of volumes. Counterfactual estimates were based on predictions from the model to calculate anticipated volume of the practice without the intervention (see Figure 4). Three of the 4 practices were already improving (bronchodilator use, iron studies, and TFT) therefore the counterfactual rate accounts for the pre-existing rates of improvement to avoid over-estimating the cost-savings. In the case of ECG ordering, the rate of ordering for sleeping bradycardia had been demonstrating a rise over time following the introduction of new state-wide observation charts which meant previously normal heart rates were now in a “medical review” zone.

Figure 4.

Change in ordering practices. Solid line = actual ordering practice and dashed line = counterfactual (anticipated).

Costs were obtained from the RCH Decision Support Unit (DSU) referencing the Power Cost Performance system FY2021-2022, and account for clinician time in addition to consumables. Sensitivity analysis used costs derived from the Australian Pharmaceutical Benefits Schedule (PBS) (for bronchodilators) and Medicare Benefits Schedule (MBS) (for blood tests and ECG). The PBS and MBS respectively provide the government funding for medications and investigations.

Costs of implementing the intervention were calculated based on the physician and analyst time involved in designing, building, and testing the EMR alert using hospital salaries as of 2021-2022. This varied from half an hour of physician time and 2 h of build time for the TFT alert to 5 h of clinician time and 5 h of analyst time for the ECG alert.

Savings were calculated by subtracting the costs of the EMR alert implementation from the costs saved.

Ethical considerations

Ethics approval for quality assurance was obtained from the RCH Ethics Department (QA/83710/RCHM-2022) and noted by the University of New South Wales Ethics Executive.

Results

Using data extracted between April 30, 2016 and June 30, 2022, 3 LVC practices showed a statistically significant reduction between pre- and post-intervention (Table 1).

Table 1.

Pre-post comparison of rates of LVC practices.

	Population (total number of encounters)	Intervention date (months since intervention)	Preintervention proportion (%)	Postintervention proportion (%)	P-value
Reducing iron studies	74 918	December 2018	56.7%	31.0%	<.001
		(43 months)	(n = 17 481)	(n = 13 929)
Reducing bronchodilators in bronchiolitis	9576	February 2019	6.9%	3.2%	<.001
		(41 months)	(n = 318)	(n = 180)
Reducing T3 in TFT	9512	August 2020	25.1%	9.9%	<.001
		(24 months)	(n = 1542)	(n = 337)
Reducing ECG for sleeping bradycardia	20 014	July 2021	28.8%	28.1%	.305
		(12 months)	(n = 5086)	(n = 661)

Abbreviations: ECG = electrocardiogram; LVC = low value care; TFT = thyroid function tests.

Time series analysis

Time series analyses demonstrated statistically significant improvements in rates for all 4 practices (Table 2, Figure 4). Statistically significant step changes were seen for iron studies and TFTs (abrupt change in ordering practices), whereas bronchodilator and ECG ordering each showed a statistically significant slope change (more gradual impact). Iron studies demonstrated a significant difference for both step and slope change; however, this was not the model of best fit. All analyses remained statistically significant after adjusting for autocorrelation and seasonality (see Appendix S1 for full analysis).

Table 2.

Model analyses for the 4 LVC practices.

	Step change IRR (95% CI)	P-value	Slope change IRR (95% CI)	P-value	Step change and slope change (95% CI)	P-value
Ferritin (replacing iron studies)	0.61 (0.58-0.63)	.000	0.998 (0.996-1.00)	<.001	0.61 (0.58-0.63)	.000
					0.996 (0.994-0.998)	.001
Reducing bronchodilators in bronchiolitis	0.82 (0.57-1.16)	.261	0.97 (0.96-0.99)	.001	0.79 (0.57-1.11)	.172
					0.97 (0.96-0.99)	.000
Reducing T3 in TFT	0.52 (0.44-0.62)	.000	0.97 (0.96-0.98)	.000	0.51 (0.40-0.64)	.000
					1.00 (0.98-1.02)	.732
Reducing ECG for sleeping bradycardia	0.90 (0.82-0.98)	.021	0.98 (0.97-0.99)	.001	1.06 (0.91-1.23)	.436
					0.97 (0.96-0.99)	.007

Statistical significance was tested for all models and the model of best fit determined by the Akaike information criterion/Bayesian information criterion. Model of best fit was used as the final model and is presented in bold.

Results presented are not adjusted for seasonality and autocorrelation.

Abbreviations: CI = confidence interval; ECG = electrocardiogram; IRR = incident rate ratio (ratio of the expected number of LVC for each unit change in time); LVC = low value care; TFT = thyroid function tests.

Alert outcomes

Comparing the volume of alert activations in the initial 6 months following alert implementation to the final 6 months of the study period showed an increase rate of firing for blood tests, a decreased rate of firing for ECGs in bradycardia during sleep and no significant change for bronchodilators in bronchiolitis (Table 3). There was no significant change seen in the proportion of alerts accepted in the initial 6 months compared to the final 6 months for any of the LVC practices.

Table 3.

Alert activations and rates of alert adherence, over time.

	Volume of alert activations			Rate of alert adherence
	Initial 6 months	Final 6 months (January-June 2022)	P-value	Initial 6 months	Final 6 months (January to June 2022)	P-value
Reducing iron studies	766 (5.1%)	539 (8.9%)	<.001	95%	96%	.38
	N = 14 972	N = 6014
	January-June 2019
Bronchodilators in bronchiolitis	59 (5.4%)	52 (3.4%)	.20	14%	10%	.06
	N = 1099	N = 1515
	February-July 2019
Reducing T3 in TFT	300 (15.1%)	356 (21.3%)	<.001	41%	32%	.60
	N = 1982	N = 1669
	August 20-January 21
Reducing ECG for sleeping bradycardia	988 (56.5%)	871 (39.9%)	<.001	24%	21%	.13
	N = 1747	N = 2180
	July-December 2021

Abbreviations: ECG = electrocardiogram; TFT = thyroid function tests.

Impact on practice rates and alert adherence were interpreted according to Figure 3 with the outcome that all alerts were worthy of keeping (Figure 5).

Figure 5.

Interpretation of results.

Cost impact

Cost impact calculated using RCH DSU costings show an total saving of $131 052 (Table 4). These savings reduce on sensitivity analysis using MBS (for investigations)/PBS (for medications) costings only (not accounting for clinical time and non-consumable costs) with a total savings of $160 336 (Appendices). Each intervention produced cost savings with an average return on investment time of 3 days to 16 months with a quicker payback for higher volume, more costly LVC practices.

Table 4.

Cost impact over life of project.

	Costs saved per annum (using decision support unit costings)	Cost of intervention	Time to return on investment
Ferritin (replacing iron studies)	$103 788	$268	3 days
Reducing bronchodilators in bronchiolitis	$2684	$321	5 months
Reducing T3 in TFT	$325	$206	16 months
Reducing ECG for sleeping bradycardia	$24 265	$1456	22 days
Total	$131 052	$2251

Abbreviations: ECG = electrocardiogram; TFT = thyroid function tests.

Discussion

This study demonstrates a positive impact of EMR alerts on 4 different LVC practices in a pediatric hospital with a cost-saving result. This is the first study to our knowledge to examine the sustained impact over time on pediatric LVC practices and the positive impact on hospital costs. LVC reduction is a worthy pursuit based on quality of care alone and campaigns such as Choosing Wisely³ would suggest that LVC practice reduction should be driven by patient experience. However, the effectiveness of the alerts combined with the cost savings, suggest that EMR interventions are a worthwhile investment.

Given the evidence base in clinical medicine is constantly evolving, remaining current with all new recommendations at an individual level is challenging. Evidence suggests that the average time from evidence to implementation is 17 years.¹⁹ EMR interventions may offer a way to support translation from evidence into practice in a timelier manner.

Our results begin to shed light on which LVC practices are more amenable to EMR interventions. Better understanding the expected impact of behavior change can help inform the role of EMR interventions. In particular, we see larger impact with practices that have high baseline rates and minimal visible change to the patient such as is seen with iron studies. Simply altering the blood test request from iron studies to ferritin does not involve discussion with the family and therefore does not risk undermining earlier decision-making. For practices where the recommendation is a modification to an existing test (eg, iron studies and TFT), a high rate of EMR alert acceptance accompanied by an abrupt change in ordering behavior can be seen. In contrast, practices such as bronchodilator ordering where efforts to reduce practice have existed for some time, and early adopters have already changed practice, further incremental change appears more challenging. In the case of bronchodilators and ECG ordering where the clinician was asked to stop a practice without replacing it with an alternative action, we found a much higher rate of the alert being ignored and subsequently a more gradual change in behavior. The incident rate ratio for both these practices is small. This likely highlights the challenges associated with changing clinical decision-making based on a computerized alert once a conversation has already been had with the patient and family. Whereas the blood test changes are barely perceptible to the patient and family, there are many barriers to cancelling a test or medication that we assume the family are expecting to receive, such as fear of undermining trust in the clinician-patient relationship, fear of looking incompetent, and time pressures. The fact that there remains a statistically significant trend toward improvement above and beyond underlying trends suggests the alert in these LVC practices may still be playing an important role in educational messaging and influencing ordering practices for future patients.

Other than stakeholder engagement in the design of each EMR alert, changes to the EMR were the only interventions implemented in all LVC practices except for bronchodilator use. In addition to the alerts for blood tests, the unnecessary tests were also removed from order panels. This may have contributed to the improvements seen, although in our institution orderset and panel use by clinicians is limited. Bronchodilator use was accompanied by an education campaign and audit-feedback following the evidence that multifaceted interventions are superior to single interventions.^4,20 However, compared to other interventions such as audit and feedback, and educational campaigns, which, in the setting of constant junior staff turnover, require continued investment over time, EMR interventions offer a sustained intervention at a system level. In the 3 practices that relied on a stand-alone EMR intervention, a return on investment was realized between 3 days and 16 months, further suggesting that EMR alerts hold substantial promise in LVC reduction.

The impact on clinician burnout and alert fatigue was considered in this study design. Whilst all 4 practices demonstrated a significant improvement over time, if high rates of alert dismissal are seen, the risks of growing alert fatigue would negate the reduction in LVC. We developed a novel a priori interpretation, against which to measure the impact of our EMR alerts. Aside from removing the alert and monitoring the impact to practice change, this is the closest proxy measure we could achieve to attribute causation to the alert and balance the risk of alert fatigue with the benefit achieved.

Strengths and limitations

Time series analysis is a quasi-experimental method, and as such detects association rather than causation. However, accounting for underlying trends allows us to control for many confounding variables as well as underlying secular trends. Seasonal adjustment accounts for changes that might be seen with the rotation of junior medical staff every quarter. Testing against the Bradford-Hill criteria suggests a strong likelihood of the alerts being causally related to the improvements in LVC ordering practice seen given the alerts are temporally related, outcomes are plausible, internal consistency is seen across the 4 alerts, and external consistency exists with similar studies.^21–23 Further evidence would be acquired through removal of the alerts and ongoing evaluation of the impact on LVC practice.

One limitation in using large EMR data sets is that we have used gross measures which capture both warranted and unwarranted variation. Given some variation in care is warranted, we are limited to looking at trends which may under or overestimate the true reduction in unwanted variation.

Our cost analysis has captured the hospital perspective only, with only direct costs included. This means any downstream impacts, positive or negative, are not adequately captured. For example, one of the main issues with ordering iron studies instead of ferritin is that the serum iron, representing iron intake over the past 24 h, is often low and misinterpreted as representing low iron stores leading to over-treatment. The potential costs associated with prescription of iron therapy and side effects will be under-estimated in this analysis. Similarly, an ECG overnight has the potential to detect a spurious result that leads to further cardiology review and investigation. On the contrary, it is possible that this unnecessary investigation may detect something significant and lead to improved healthcare outcomes as a result. Without measuring all downstream implications, this broader impact and the cost implications remain poorly understood. However, tests and treatments performed without a clear indication, as is the case in these LVC practices, are far more likely to result in spurious outcomes, and the savings presented are therefore likely to be an underestimate.²⁴ We were unable to perform a cost-effectiveness analysis comparing costs with outcomes as the net effect on health being equivalent is an assumption only. We simply aimed to calculate the costs recovered from reduction in the direct LVC practice itself compared to the cost of the alert, relying on the evidence base that reduction in these practices would lead to higher value care.

Our analysis assumes that the alert criteria adequately detects a population of children in whom the intervention is always inappropriate. Whilst the evidence base is strong, there are always nuances and exceptions in clinical practice, whereby the use of a medication or investigation may be considered an appropriate decision. Our analysis is based on an absolute interpretation of inappropriate and appropriate use and does not account for these nuances.

In addition, we have been unable to separate the impact of the alert from other more subtle EMR changes such as removal of the unnecessary tests from OrderSets. Given our institution uses OrderSets poorly, the impact of this is likely to be small but should be acknowledged.

Conclusions

EMR alerts are a promising intervention for minimizing LVC in pediatric settings. However, our findings indicate that the efficacy of these alerts varies across different contexts, underlining the need for additional research to explore the complexities involved. As we continue to strive to improve patient outcomes, while constraining unnecessary costs, it will be important to fine-tune EMR alerts to guide clinician decision-making toward tests and treatments with proven benefits. The dual advantages of reducing LVC and achieving cost-savings supports the rationale for further investments in the development and refinement of EMR alerts.

Author contributions

J.L. led the co-design of the EMR alerts, designed the methodology, analyzed the data, and drafted and finalized the article. M.S. and H.H. provided clinical oversight to the alerts, critically reviewed the methodology and critically reviewed, and approved the final article. A.S. and J.R. provided critical input into the methodology and analysis and reviewed and approved the final article.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was supported by the Digital Health Fellowship, Centre for Digital Transformation of Health, University of Melbourne, Australia.

Conflicts of interest

None declared.

Data availability

Deidentified raw and analyzed data available on request to main author.