Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2020 Aug 6;15(8):e0237159. doi: 10.1371/journal.pone.0237159

Evaluation and cost estimation of laboratory test overuse in 43 commonly ordered parameters through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital

Andrea Tamburrano 1,*, Doriana Vallone 1, Cinzia Carrozza 2, Andrea Urbani 2, Maurizio Sanguinetti 3, Nicola Nicolotti 4, Andrea Cambieri 4, Patrizia Laurenti 1
Editor: Pal Bela Szecsi5
PMCID: PMC7410244  PMID: 32760101

Abstract

Background

Computerized Clinical Decision Support Systems (CCDSS) have become increasingly important in ensuring patient safety and supporting all phases of clinical decision making. The aim of this study is to evaluate, through a CCDSS, the rate of the laboratory tests overuse and to estimate the cost of the inappropriate requests in a large university hospital.

Method

In this observational study, hospital physicians submitted the examination requests for the inpatients through a Computerized Physician Order Entry. Violations of the rules in tests requests were intercepted and counted by a CCDSS, over a period of 20 months. Descriptive and inferential statistics (Student’s t-test and ANOVA) were made. Finally, the monthly comprehensive cost of the laboratory tests was calculated.

Results

During the observation period a total of 5,716,370 requests were analyzed and 809,245 violations were counted. The global rate of overuse was 14.2% ± 3.0%.

The most inappropriate exams were Alpha Fetoprotein (85.8% ± 30.5%), Chlamydia trachomatis Nucleic Acid Amplification (48.7% ± 8.8%) and Alkaline Phosphatase (20.3% ± 6.5%). The monthly cost of over-utilization was 56,534€ for basic panel, 14,421€ for coagulation, 4,758€ for microbiology, 432€ for immunology exams. All the exams, generated an estimated avoidable cost of 1,719,337€ (85,967€ per month) for the hospital.

Conclusions

The study confirms the wide variability in over-utilization rates of laboratory tests. For these reasons, the real impact of inappropriateness is difficult to assess, but the generated costs for patients, hospitals and health systems are certainly high and not negligible. It would be desirable for international medical communities to produce a complete panel of prescriptive rules for all the most common laboratory exams that is useful not only to reduce costs, but also to ensure standardization and high-quality care.

Introduction

The demands of laboratory tests have become the highest volume medical act [1], after years of steady increase. In the United States and Europe, the annual increase in the use of laboratory tests has been around 5% in the last decade. Medicare spending on clinical laboratory tests peaked at almost $ 10 billion, or 1.7% of the total health care budget [2]. Even if the costs of laboratory tests represent less than 5% of hospital spending [3], different studies indicated that pathology investigations are involved in 70–80% of all healthcare decisions [4,5].

According to Zhi et al. [6] the inappropriate test can be classified in different forms.

Over-utilization or over-referencing refers to tests ordered but not appropriate, while under-utilization refers to the tests appropriate but not ordered. There are also different types of inadequacy criteria. The objective criteria are clearly defined and independent of the investigation, while the subjective criteria depend on the expert review. Restrictive criteria are required when there is a clear indication for ordering a test, while permissive criteria are required only when there are no contraindications.

A procedure is "appropriate" when it produces more benefits than harm enough to justify its use. Instead, procedures are defined as "equivocal" for which the potential benefits and risks of harm to patients are theoretically equivalent, and "inappropriate" are the procedures for which the risks of harm to the patient clearly outweigh the potential benefits [7].

Computerized Clinical Decision Support Systems (CCDSS) are information technology-based systems that use specific patient characteristics and combine them with a knowledge base using rule-based algorithms [8]. They have become increasingly important in ensuring patient safety and in supporting all phases of clinical decision making. In laboratory medicine, CCDSS are usually used to guide the ordering of tests and diagnostic forecasting by combining informative components and staff skills [9]. By generating reminders or specific patient recommendations that require more appropriate care, CCDSS can also be effective in reducing unnecessary diagnostic tests. In many cases, the assessment is accompanied by an estimate of the savings, often substantial [10–15].

The aim of this study is to evaluate, through a silent CCDSS, the rate of laboratory tests overuse and to estimate the cost of the inappropriate requests in a large university hospital.

Materials and methods

Ethics statement

The study is compliant with the Local Ethical Committee Standards of the Fondazione Policlinico Universitario Agostino Gemelli IRCCS; it was approved and registered (Prot. 45189/19 ID: 2849). The study is in accordance with the Helsinki Declaration and EU Regulation 2016/679 (GDPR) concerning the processing of personal data. For this type of study, Ethical Committee did not foresee the need for participant consent.

Setting

The Fondazione Policlinico Universitario Agostino Gemelli IRCCS is a 1,526-bed high-care-complexity university hospital located in Rome, Italy.

Its laboratory performs about 3.5 million tests every year for inpatients, of about a thousand different types (clinical biochemistry, hematology, coagulation and microbiology). The most frequently requested exams (around 100 types) are performed in the high-automation Corelab, a forefront centralized laboratory, and reported on the same day.

During a period of 20 months (from July 2016 to February 2018), we monitored the requests of the most representative laboratory exams made by all the hospital internal departments (except for Emergency, Intensive Care Units and urgent blood test requests). Physicians made inpatient-exam requests through a Computerized Physician Order Entry (CPOE), accessible to medical staff only, that communicated with the Laboratory Information System (LIS) DNLab (Dedalus SpA). The laboratory processed the samples, analyzed them and, after validation, automatically sent the results to the clinicians through the LIS.

Computerized Clinical Decision Support System (CCDSS)

The Prometeo Appropriatezza Software (ver. 2.1.3, 2016, NoemaLife SpA) intercepted and counted, for each exam, all the laboratory requests and the violations of the rules, in silent mode without blocking or generating pop-ups.

A total of 43 laboratory tests were monitored and 2 different rules have been applied:

  • Biological invariance rules (minimal re-testing intervals): each request is verified for the presence (in the same patient) of a still valid result preceding the date of the request acceptance;

  • Incompatibility rules: each test is associated with a list of incompatible laboratory tests and a list of related exams that must be requested simultaneously.

39 tests for biological invariance, 4 tests for incompatibility rules were monitored.

The rules, principally based on minimal re-testing intervals criteria reported in the most recent international guidelines on the prescription appropriateness for laboratory tests [16,17], were shared and approved, over a period of 1 month, by an expert panel made up of laboratory physicians, hospital department chiefs and hospital management.

Laboratory tests were also grouped into 4 different categories: basic panel, coagulation, immunology, microbiology.

Among basic panel exams, electrolytes, lipid and liver panel, among microbiology exams, culture and antibodies tests were considered as sub-categories, respectively.

All the examined tests have been ordered and counted as single tests, except for Complete Blood Count that included: Red Blood Cells (RBC), Hemoglobin (Hb), Hematocrit (Ht), Mean Corpuscular Volume (MCV), Mean Corpuscular Hemoglobin (MCH), Mean Corpuscular Hemoglobin Concentration (MCHC), Red Cells Dispersion Width (RDW), Platelets (PLTS). This panel of tests was considered and counted in bulk.

Table 1 shows the list of the monitored laboratory tests, their division into categories and sub-categories, and the rules applied.

Table 1. Laboratory tests, categories, sub categories and rules.

Laboratory test Category Sub-category Rule Rule description
Amylase Basic panel - Incompatibility Test cannot be requested with lipase test
Sedimentation Rate Basic panel - Biological invariance Test was already requested in 7 days
White blood cell count and differential Basic panel - Biological invariance Test was already requested in 7 days
Complete Blood Count* Basic panel - Biological invariance Test was already requested in 7 days
Triglycerides Basic panel Lipid panel Biological invariance Test was already requested in 21 days
Protein Electrophoresis Basic panel - Biological invariance Test was already requested in 21 days
Total Cholesterol Basic panel Lipid panel Biological invariance Test was already requested in 21 days
Magnesium Basic panel Electrolytes Biological invariance Test was already requested in 24 hours
Sodium Basic panel Electrolytes Biological invariance Test was already requested in 24 hours
Potassium Basic panel Electrolytes Biological invariance Test was already requested in 24 hours
Chloride Basic panel Electrolytes Biological invariance Test was already requested in 24 hours
Creatinine Basic panel - Biological invariance Test was already requested in 24 hours
Aspartate Aminotransferase Basic panel Liver panel Biological invariance Test was already requested in 24 hours
Alanine Transaminase Basic panel Liver panel Biological invariance Test was already requested in 24 hours
Alkaline Phosphatase Basic panel - Biological invariance Test was already requested in 24 hours
Bilirubin Basic panel Liver panel Biological invariance Test was already requested in 24 hours
Albumin Basic panel - Biological invariance Test was already requested in 24 hours
Total Serum Protein Basic panel - Biological invariance Test was already requested in 24 hours
Gamma-Glutamyl Transferase Basic panel Liver panel Biological invariance Test was already requested in 24 hours
Lactate Dehydrogenase Basic panel - Biological invariance Test was already requested in 24 hours
HDL Cholesterol Basic panel Lipid panel Biological invariance Test was already requested in 21 days
LDL Cholesterol Basic panel Lipid panel Biological invariance Test was already requested in 21 days
HIV1-2 Antibodies Microbiology Antibody Biological invariance Previous test with positive result
Toxoplasma Antibody (IgG) Microbiology Antibody Biological invariance Previous test with positive result
Cytomegalovirus Antiboy (IgG) Microbiology Antibody Biological invariance Previous test with positive result
Epstein-Barr Virus Antibody (IgG) Microbiology Antibody Biological invariance Previous test with positive result
Rubella Antibody (IgG) Microbiology Antibody Biological invariance Previous test with positive result
Chlamydia trachomatis nucleic acid amplification (NAATs) Microbiology Culture Incompatibility Test cannot be requested on vaginal swab/secretion
Blood Culture—Aerobic Microbiology Culture Biological invariance Test was already requested in 24 hours
Blood Culture—Anaerobic Microbiology Culture Biological invariance Test was already requested in 24 hours
Antinuclear Antibody Immunology - Biological invariance Test was already requested in 90 days
Immunoglobulin A Immunology - Biological invariance Test was already requested in 21 days
Immunoglobulin G Immunology - Biological invariance Test was already requested in 21 days
Immunoglobulin M Immunology - Biological invariance Test was already requested in 21 days
Prothrombin time Coagulation - Biological invariance Test was already requested in 24 hours with results in the normal range
Partial Thromboplastin Time Coagulation - Biological invariance Test was already requested in 24 hours with results in the normal range
Fibrinogen Coagulation - Biological invariance Test was already requested in 24 hours with results in the normal range
D-Dimer Coagulation - Biological invariance Test was already requested in 24 hours with results in the normal range
Antithrombin III Coagulation - Biological invariance Test was already requested in 24 hours with results in the normal range
Beta HCG Other - Incompatibility Test cannot be requested for male subjects**
Alpha Fetoprotein (AFP) Other - Incompatibility Test cannot be requested with nonspecific tumor markers (CEA, CA125, CA19-9, CA15-3, TPA)
Procalcitonin Other - Biological invariance Test was already requested in 24 hours
Vitamin B12 Other - Biological invariance Test was already requested in 30 days
Open in a new tab

* Complete Blood Count includes: Red Blood Cells (RBC), Hemoglobin (Hb), Hematocrit (Ht), Mean Corpuscular Volume (MCV), Mean Corpuscular Hemoglobin (MCH), Mean Corpuscular Hemoglobin Concentration (MCHC), Red Cells Dispersion Width (RDW), Platelets (PLTS).

** The test can be requested in male patients only for the diagnosis and monitoring of testicular seminoma.

Analysis

Descriptive statistics were performed analyzing, over a period of 20 months for each laboratory exam, the monthly mean and standard deviation of requests and violations of the rules. Based on them, the overuse rates were calculated.

Moreover, Student’s t-test and ANOVA were used to assess differences between quantitative variables. The level of significance was set at 0.05. Statistical analyses were conducted with STATA software ver. 13.1 (Statacorp, College Station, TX, USA).

According to the Italian National Health System (NHS), all treatments were carried out free of charge for the hospitalized patients. Hospitalization costs (including laboratory tests) were reimbursed by the NHS according to Medicare Diagnosis Related Groups (MS-DRGs) [18]. The monthly comprehensive cost of the laboratory tests was calculated in euro (€), according to the 2019 reimbursement fees of the Lazio Region.

Results

Overuse

During the observation period, a total of 5,716,370 requests were analyzed (285,819 per month) and 809,245 violations were counted (40,462 per month). The global rate of overuse was 14.2% ± 3.0%.

The rate was 15.2% ± 3.3% for basic panel, 8.1% ± 3.4% for microbiology, 7.2% ± 1.9% for immunology, 5.8% ± 1.1% for coagulation laboratory tests. The overall difference among groups was significant (p<0.001, ANOVA).

Among basic panel exams, the rate was 18.8% ± 4.9% for liver panel, 9.0% ± 2.3% for lipid panel, 16.0% ± 3.8% for electrolytes. The overall difference among sub-groups was significant (p<0.001, ANOVA).

Among microbiology exams, the rate was 8.4% ± 1.7% for cultural tests, 7.7% ± 12.9% for antibodies. No significant differences were observed among sub-groups (p = 0.811, t-test).

The most inappropriate exams were Alpha Fetoprotein (85.8% ± 30.5%), Chlamydia trachomatis Nucleic Acid Amplification (48.7% ± 8.8%) and Alkaline Phosphatase (20.3% ± 6.5%). The most appropriate exams were Sedimentation Rate (0.8% ± 0.5%), HIV1-2 Antibodies (2.7% ± 1.6%) and Total Cholesterol (3.4% ± 1.3%).

Table 2 shows the number of monthly requests and the rate of overuse for each monitored exam.

Table 2. Mean and standard deviation of monthly requests and over-utilization rate, unit cost and monthly avoidable cost, for each monitored exam.

Exam Monthly requests Over-utilization rate (%) Unit cost (€) Total monthly cost (€)
Alanine Transaminase 17,361 ± 2,144 18.0 ± 4.0 1.00 3,121
Albumin 11,302 ± 1,403 19.3 ± 4.8 1.42 3,086
Alkaline Phosphatase 8,978 ± 1,092 20.3 ± 6.5 1.04 1,898
Alpha Fetoprotein 176 ± 31 85.8 ± 30.5 7.40 1,103
Amylase 7,394 ± 800 19.4 ± 1.6 1.84 2,640
Antinuclear Antibody 115 ± 23 6.6 ± 4.2 9.56 73
Antithrombin III 1,382 ± 205 9.3 ± 2.1 5.02 640
Aspartate Aminotransferase 4,482 ± 902 19,5 ± 8.0 1.04 905
Beta HCG 405 ± 48 5.9 ± 1.3 6.02 144
Bilirubin 16,239 ± 1,921 19.1 ± 4.5 1.41 4,371
Blood Culture—Aerobic 1,692 ± 226 3.4 ± 1.1 13.86 809
Blood Culture—Anaerobic 1,692 ± 226 12.1 ± 2.9 13.86 2,841
Chlamydia trachomatis nucleic acid amplification (NAATs) 56 ± 39 48.7 ± 8.8 9.41 247
Chloride 6,316 ± 934 19.8 ± 6.5 1.13 1,395
Complete Blood Count* 23,604 ± 2,873 15.3 ± 3.7 3.17 11,334
Creatinine 21,159 ± 2,553 17.1 ± 2.9 1.13 4,084
Cytomegalovirus Antibody (IgG) 275 ± 70 10.4 ± 17.7 8.07 234
D-Dimer 1,592 ± 208 12.8 ± 2.9 4.99 1,013
Epstein-Barr Virus Antibody (IgG) 172 ± 49 9.6 ± 18.0 12.45 202
Fibrinogen 10,862 ± 1,276 7.8 ± 1.5 12.18 10,238
Gamma-Glutamyl Transferase (GGT) 10,016 ± 1,431 19.1 ± 6.2 1.13 2,149
HDL Cholesterol 2,089 ± 487 8.5 ± 1.9 1.43 252
HIV1-2 Antibodies 350 ± 93 2.7 ± 1.6 10.90 106
Immunoglobulin A 351 ± 82 5.8 ± 2.2 4.99 99
Immunoglobulin G 338 ± 78 8.1 ± 2.1 4.99 134
Immunoglobulin M 327 ± 77 7.8 ± 1.9 4.99 125
Lactate Dehydrogenase 12,710 ± 1,333 19.1 ± 4.8 1.13 2,748
LDL Cholesterol 1,857 ± 454 9.8 ± 2.0 0.67 120
Magnesium 6,865 ± 818 12.0 ± 3.7 1.55 1,301
Partial Thromboplastin Time 10,961 ± 1,265 3.7 ± 3.0 2.85 1,122
Potassium 19,619 ± 2,217 18.0 ± 4.0 1.02 3,608
Procalcitonin 2,051 ± 257 12.1 ± 3.0 14.41 3,611
Protein Electrophoresis 1,896 ± 315 8.3 ± 1.6 4.23 667
Prothrombin Time 11,200 ± 1,276 4.4 ± 3.0 2.85 1,408
Rubella Antibody (IgG) 308 ± 102 9.8 ± 19.2 7.88 215
Sedimentation Rate 1,596 ± 377 0.8 ± 0.5 1.95 24
Sodium 19,548 ± 2,211 14.2 ± 3.4 1.02 2,841
Total Cholesterol 8,081 ± 1,390 3,4 ± 1.3 1.04 273
Total Serum Protein 10,794 ± 1,447 18.8 ± 4.9 4.23 8,522
Toxoplasma Antibody (IgG) 163 ± 41 8.1 ± 18.8 7.79 104
Triglycerides 6,833 ± 1,251 15.5 ± 4.0 1.17 1,219
Vitamin B12 638 ± 135 6.9 ± 1.8 7.32 318
White blood cell count and differential 21,974 ± 2,539 5.5 ± 1.7 3.91 4,619
Total 285,819 14.2 ± 3.0 - 85,967
Open in a new tab

* Complete Blood Count includes: Red Blood Cells (RBC), Hemoglobin (Hb), Hematocrit (Ht), Mean Corpuscular Volume (MCV), Mean Corpuscular Hemoglobin (MCH), Mean Corpuscular Hemoglobin Concentration (MCHC), Red Cells Dispersion Width (RDW), Platelets (PLTS).

Cost evaluation

All the exams, globally considered, generated an estimated avoidable cost of 1,719,337€ (85,967€ per month) for the hospital.

The monthly cost of over-utilization was 56,534€ for basic panel, 14,421€ for coagulation, 4,758€ for microbiology, 432€ for immunology exams.

The greatest monthly cost of overuse was for Complete Blood Count (11,334€), Fibrinogen (10,238€) and Total Serum Protein (8,522€) that cover 35% of the total over-utilization cost. The least monthly cost of overuse was for Sedimentation Rate (24€), Antinuclear Antibody (73€) and Immunoglobulin A (99€).

Table 2 shows the unit cost and the total monthly cost for each monitored exam.

Discussion

This study evaluated the overuse of laboratory tests providing data on the prescriptive activity of a large university hospital (5,716,370 requests) over a long period of time (20 months).

Several authors show variable rates of inappropriateness in laboratory tests: 4.5–95.0% [19], 25.0–75.0% [20] 5.0–95.0% [21], 45.4–93.9% [22]. Zhi et al. [6] reported that this variability is due to the great variety in tests, clinical settings, timing (initial vs. repeat testing), adopted criteria (restrictive vs. permissive, subjective vs. objective) and test volume (low-volume vs. high-volume).

Unlike the most recent works on laboratory overuse (compared to which it is necessary to consider a reasonable variability in terms of settings and tests considered), our rate (14.2%) was lower than reported by other authors: Zhi et al. (20.6%) [6], Feldhammer et al. (27.0%) [23].

The basic panel exams showed an over-utilization rate (15.2%) similar to that reported by Zhi et al. (10,2–19,1%) [6] and May et al. (11,5%) [24]. Differently, Rao et al. reported an higher rate (38%) [25]. Electrolytes registered a rate (16.0%) lower than reported by Wang et al. (31.0–40.0%) [26].

Microbiology tests registered a rate (8.1%) lower than reported by and Zhi et al. (23.1%) [6]. Antibodies showed a rate (7.7%) lower than reported by Crump et al. (25.0%) [27].

Coagulation tests registered a rate (5.8%) lower than reported by Iturratte et al. (19.7%) [10].

As reported by other authors, we also registered an evident variability between single exams (85% of Alpha Fetoprotein to 0.8% of Sedimentation Rate) and groups (15.2% of basic panel to 5.8% of coagulation tests).

The plurality of medical specialties and hospital policies, as well as health, legislative, economic and political context could explain the variability in inappropriateness found in different countries.

Regarding costs, for the only 43 exams considered (to which only one rule has been applied), we estimated consistent economic savings for the hospital (85,967€ per month). The highest cost was due to basic panel exams, in particular Complete Blood Count, Fibrinogen and Total Serum Proteins, that covered 35% of the total costs for laboratory over-utilization. Antibodies (in microbiology and immunology categories) and Immunoglobulins (in immunology category) had the least impact on the total costs.

In addition to the costs, inappropriateness may also affect other aspects of the health care, such as additional procedures or treatments based on redundant tests, avoidable medical errors, waste of time for doctors, work overload for the laboratory, delays in reporting times [28,29].

In a context of high incidence of medical disputes [30,31], the use of defensive medicine has generated an increasing number of tests performed per patient, usually generated through predetermined panels of tests, that are easily ordered [32]. Utilization and decisions based on routine tests, given the widespread nature of their use, the ease of measurement and low cost, are not always appropriate, leading to undesirable consequences for the patients. An inappropriate use of routine tests minimizes their utility and favors erroneous interpretations, increasing safety risks for the patient. This practice could also lead to order potentially harmful complementary tests, to rule out or corroborate the results obtained.

Additionally, uncertain results and test repetition could cause anxiety in the patients, as they are faced with the possibility of an uncertain diagnosis [33]. For these reasons, the use of routine tests should be based on clear and sufficient scientific evidence.

According to Lanzoni et al. [15], although the definition of test panels for the diagnosis/monitoring of different health conditions is a good tool to increase the clinical governance, a deep evaluation must be done on the real needs on requiring the same tests all the time for a single patient.

Safety risks of inappropriateness should be analysed to implement appropriate strategies to improve their correct use.

Strengths and limitations

Only a small part of the total performed test types (over 1000) were monitored, as well as the potential rules applicable to each exam are much more varied and numerous than those reported in this study. For this reason, the rate of overuse and the sum of avoidable costs were underestimated. However, the most representative exams and rules (applicable in the context of an automatic CCDSS) were included and fully monitored with over 5 million of laboratory tests performed over a period of 20 months.

The CCDSS counted violations of the over-utilization rules without blocking or generating pop-ups and physicians were not alerted. Although it was not possible to evaluate CCDSS effectiveness in reducing the over-utilization rate, we were able to assess the rate of laboratory tests overuse without the confounding factor of physician’s awareness of a control system on laboratory requests. For the cost estimation we assumed the ideal 100% reduction in tests overuse.

Conclusions

A recent systematic review compared strategies to change the behavior of doctors and to reduce the inappropriateness rate of the laboratory tests [34]. Various interventions, such as educational strategies, feedback, audit, modification of test modules and reminders (a form of CCDSS) should be planned because effective [35].

This study confirms the wide variability in over-utilization of laboratory tests. For these reasons, the real impact of inappropriateness is difficult to assess, but the generated costs for patients, hospitals and health systems are certainly high and not negligible [28,36,37].

It would be desirable for international medical communities to produce a complete panel of prescriptive rules for all the most common laboratory exams (which consider the variability of settings and clinical conditions) that is useful not only to reduce costs, but also to ensure standardization and high-quality care.

Supporting information

S1 Dataset

(XLSX)

Click here for additional data file. (75.2KB, xlsx)

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References

  • 1.Health, United States, 2017. Hyattsville; 2017.
  • 2.Delvaux N, Van Thienen K, Heselmans A, van de Velde S, Ramaekers D, Aertgeerts B. The Effects of Computerized Clinical Decision Support Systems on Laboratory Test Ordering: A Systematic Review. Arch Pathol Lab Med [Internet]. 2017. April;141(4):585–95. Available from: http://www.archivesofpathology.org/doi/10.5858/arpa.2016-0115-RA [DOI] [PubMed] [Google Scholar]
  • 3.R.W. F. Why is the laboratory an afterthought for managed care organizations? Clin Chem. 1996; [PubMed]
  • 4.Driskell OJ, Holland D, Hanna FW, Jones PW, Pemberton RJ, Tran M, et al. Inappropriate requesting of glycated hemoglobin (Hb A1c) is widespread: Assessment of prevalence, impact of national guidance, and practice-To-practice variability. Clin Chem. 2012; [DOI] [PubMed] [Google Scholar]
  • 5.Kwok J, Jones B. Unnecessary repeat requesting of tests: An audit in a government hospital immunology laboratory. Journal of Clinical Pathology. 2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Zhi M, Ding EL, Theisen-Toupal J, Whelan J, Arnaout R. The landscape of inappropriate laboratory testing: A 15-year meta-analysis. PLoS One. 2013; [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Brook RH. Assessing the appropriateness of care—Its time has come. JAMA—Journal of the American Medical Association. 2009. [DOI] [PubMed] [Google Scholar]
  • 8.Haynes RB, Wilczynski NL. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: Methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010; [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Plebani M, Aita A, Padoan A, Sciacovelli L. Decision Support and Patient Safety. Clin Lab Med [Internet]. 2019. June;39(2):231–44. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0272271219300034 [DOI] [PubMed] [Google Scholar]
  • 10.Iturrate E, Jubelt L, Volpicelli F, Hochman K. Optimize Your Electronic Medical Record to Increase Value: Reducing Laboratory Overutilization. Am J Med. 2016; [DOI] [PubMed] [Google Scholar]
  • 11.Morgen EK, Naugler C. Inappropriate repeats of six common tests in a Canadian city: A population cohort study within a Laboratory Informatics framework. Am J Clin Pathol. 2015; [DOI] [PubMed] [Google Scholar]
  • 12.Vegting IL, Van Beneden M, Kramer MHH, Thijs A, Kostense PJ, Nanayakkara PWB. How to save costs by reducing unnecessary testing: Lean thinking in clinical practice. Eur J Intern Med. 2012; [DOI] [PubMed] [Google Scholar]
  • 13.Procop GW, Yerian LM, Wyllie R, Harrison AM, Kottke-Marchant K. Duplicate laboratory test reduction using a clinical decision support tool. Am J Clin Pathol. 2014; [DOI] [PubMed] [Google Scholar]
  • 14.Meidani Z, Farzandipour M, Farrokhian A, Haghighat M. A review on laboratory tests’ utilization: A trigger for cutting costs and quality improvement in health care settings. Med J Islam Repub Iran. 2016; [PMC free article] [PubMed] [Google Scholar]
  • 15.Lanzoni M, Fornili M, Felicetta I, Maiavacca R, Biganzoli E, Castaldi S. Three-year analysis of repeated laboratory tests for the markers total cholesterol, ferritin, vitamin D, vitamin B12, and folate, in a large research and teaching hospital in Italy. J Eval Clin Pract. 2017; [DOI] [PubMed] [Google Scholar]
  • 16.Lang T. National Minimum Re‐testing Interval Project: A final report detailing consensus recommendations for minimum re‐testing intervals for use in Clinical Biochemistry. London; 2013.
  • 17.Lang T, Croal B. National minimum retesting intervals in pathology: A final report detailing consensus recommendations for minimum retesting intervals for use in pathology. London; 2015.
  • 18.Design and development of the Diagnosis Related Group (DRG) [Internet]. Baltimore; 2019. https://www.cms.gov/icd10m/version37-fullcode-cms/fullcode_cms/Design_and_development_of_the_Diagnosis_Related_Group_(DRGs).pdf
  • 19.Van Walraven C, David Naylor C. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. Journal of the American Medical Association. 1998. [DOI] [PubMed] [Google Scholar]
  • 20.Bayram C. Evidence-practice gap in GP pathology test ordering A comparison of BEACH pathology data and recommended testing. Sydney; 2009.
  • 21.Silverstein MD. An Approach to Medical Errors and Patient Safety in Laboratory Services. Atlanta; 2003.
  • 22.Cappelletti P. Appropriateness of diagnostics tests. Int J Lab Hematol [Internet]. 2016. May;38:91–9. Available from: http://doi.wiley.com/10.1111/ijlh.12502 [DOI] [PubMed] [Google Scholar]
  • 23.Feldhammer M, Menasco D, Zhang W, Ritchie J. A Targeted, Evidence-Based Clinical Decision Support Intervention to Reduce Unnecessary Complete Blood Count Orders. Am J Clin Pathol. 2017; [Google Scholar]
  • 24.May TA, Clancy M, Critchfield J, Ebeling F, Enriquez A, Gallagher C, et al. Reducing unnecessary inpatient laboratory testing in a teaching hospital. Am J Clin Pathol. 2006; [DOI] [PubMed] [Google Scholar]
  • 25.Gopal Rao G, Crook M, Tillyer ML. Pathology tests: Is the time for demand management ripe at last? J Clin Pathol [Internet]. 2003. April 1;56(4):243–8. Available from: http://jcp.bmj.com/cgi/doi/10.1136/jcp.56.4.243 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Wang TJ, Mort EA, Nordberg P, Chang Y, Cadigan ME, Mylott L, et al. A utilization management intervention to reduce unnecessary testing in the coronary care unit. Arch Intern Med. 2002; [DOI] [PubMed] [Google Scholar]
  • 27.Crump JA, Corder JR, Henshaw NG, Reller LB. Development, Implementation, and Impact of Acceptability Criteria for Serologic Tests for Infectious Diseases. J Clin Microbiol. 2004; [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Cadamuro J, Ibarz M, Cornes M, Nybo M, Haschke-Becher E, von Meyer A, et al. Managing inappropriate utilization of laboratory resources. Diagnosis. 2018. August; [DOI] [PubMed] [Google Scholar]
  • 29.Ferraro S, Panteghini M. The role of laboratory in ensuring appropriate test requests. Clinical Biochemistry. 2017. [DOI] [PubMed] [Google Scholar]
  • 30.Zeng Y, Zhang L, Yao G, Fang Y. Analysis of current situation and influencing factor of medical disputes among different levels of medical institutions based on the game theory in Xiamen of China. Medicine (Baltimore) [Internet]. 2018. September;97(38):e12501 Available from: http://journals.lww.com/00005792-201809210-00107 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Liu Y, Bai Y, Wang P, Xu Z. Study of the factors causing medical disputes in a Third‐Level Grade A Hospital in Shanghai. Int J Health Plann Manage [Internet]. 2018. October 10;33(4):e1137–46. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/hpm.2599 [DOI] [PubMed] [Google Scholar]
  • 32.Lee M. The Effects of Criminal Punishment on Medical Practices in the Medical Environment. Int J Environ Res Public Health [Internet]. 2019. February 19;16(4):604 Available from: http://www.mdpi.com/1660-4601/16/4/604 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Moreno-Campoy EE, Mérida-De la Torre FJ, Martos-Crespo F, Plebani M. Identifying risk in the use of tumor markers to improve patient safety. Clin Chem Lab Med [Internet]. 2016. January 1;54(12). Available from: https://www.degruyter.com/doi/10.1515/cclm-2015-0760 [DOI] [PubMed] [Google Scholar]
  • 34.Cadogan SL, Browne JP, Bradley CP, Cahill MR. The effectiveness of interventions to improve laboratory requesting patterns among primary care physicians: a systematic review. Implement Sci [Internet]. 2015. December 5;10(1):167 Available from: http://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0356-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Miyakis S, Karamanof G, Liontos M, Mountokalakis TD. Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy. Postgrad Med J. 2006; [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Lippi G, Brambilla M, Bonelli P, Aloe R, Balestrino A, Nardelli A, et al. Effectiveness of a computerized alert system based on re-testing intervals for limiting the inappropriateness of laboratory test requests. Clin Biochem. 2015; [DOI] [PubMed] [Google Scholar]
  • 37.Ma I, Lau CK, Ramdas Z, Jackson R, Naugler C. Estimated costs of 51 commonly ordered laboratory tests in Canada. Clin Biochem. 2019. March;65:58–60. 10.1016/j.clinbiochem.2018.12.013 [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Pal Bela Szecsi

7 Feb 2020

PONE-D-20-01565

Evaluation and cost estimation of inappropriateness in laboratory tests through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

PLOS ONE

Dear Dr. Tamburrano,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

In addition to the issues raised by the reviewers, please address the following.

The use (misuse) of laboratory tests is a common issue that not only affect the laboratories but also create unnecessary work for clinicians that may harm the patients.

As far as I can understand, is the title of the paper wrong, the authors focus only on repeating test, not the appropriateness? For example is an ASAT test seldom relevant.

Furthermore is only a fraction of test investigated. I miss the most frequently requested test, hemoglobin. These factors may explain the relative low frequency of misuse, I agreement with the literature is my observation that the correct number is around 60-70%.

We would appreciate receiving your revised manuscript by Mar 23 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Pal Bela Szecsi, M.D. D.M.Sci.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have performed an observational study on the appropriateness of laboratory test orders in a large university hospital. Inappropriate testing is an important issue in health care systems facing increasing demands and diminishing resources.

There might be more information in the data set to be presented. I also have some concerns, mainly regarding the description of the setting.

Major comments:

1) The economic setting should be thoroughly described and discussed because economic incentives have an impact on test ordering behavior. A description of who is paying for the analyses is essential to understand the setting. Is it the patient paying anything (a fixed fee or the total cost)? Does the ordering unit pay to the laboratory, or is the laboratory completely budget financed? Are the hospital and the laboratory owned by the state, a company, or a trust?

2) Does the laboratory accept requests from primary health care centers? Were samples from non-hospital units included?

3) The cost in this paper is defined as the reimbursement fee set by the authorities. The reimbursement system should be explained, allowing the reader to understand the context. The reimbursement fee seems to includes reagents, staff, rental costs of the laboratory site, overhead costs, and so on. Most of the costs, excluding the reagent costs, are more or less constant so that the net savings might be substantially lower than the figures described in the paper. Who would save money with more appropriate testing? Would it have a negative impact on the laboratory economy?

4) Which staff categories are allowed to order laboratory tests? Is it only physicians, or are nurses also allowed to order tests?

5) Were there any temporary physicians from staffing companies? Physicians from staffing companies tend to order more tests.

6) It has previously been described that the demographics of the physicians are of importance for test ordering behavior. Were there any differences between violation rates between medical interns, residents, and consultants? The large number of requests would probably allow for subgroup analysis, which would add substantial value to the paper.

7) This study reports a lower proportion of inappropriate tests than previous studies. Was there any information about the study available to the hospital staff? Was the staff aware of the rules of appropriate testing? Was there any education about this before or during the study? If so, this should be described and discussed, as information and education have been shown to impact test ordering patterns.

8) The use of test panels in the study setting should be described in more detail. Are the tests usually ordered in bundled panels, or as single tests?

9) How were tests included in panels counted towards the total test count? Was the panel counted as one test, or are all the included analytes counted as separate tests?

Minor comments:

1) The Conclusion section could be shortened, and some of the text in this section would be more suitable in the Discussion section.

Reviewer #2: The authors have collected a remarkable number of datapoints regarding inappropriately ordered laboratory tests. They impressively show the financial benefits of addressing this issue.

I have some major comments which need to be addressed imho in order to improve the ms.

The term „Inappropriateness” is used icorrectly. In order to cover all inappropriate lab orders, those underused have to be included. To do so, mostly the anamnesis, physical examination and indication for testing is needed (i.e. signs of mucosal bleeding and positive family history but only APTT is ordered). As this is not presented in this study, I suggest rephrasing. Additionally, overuse does not only consist of tests ordered too soon after baseline measurement (re-testing interval). There are several other causes for laboratory overuse.

See:

Greenberg J, Green JB. Over-testing: why more is not better. Am J Med 2014; 127:362-363.

Cadamuro J, Gaksch M, Wiedemann H, et al. Are laboratory tests always needed? Frequency and causes of laboratory overuse in a hospital setting. Clin Biochem 2018; 54:85-91.

Fryer AA, Smellie WSA. Managing demand for laboratory tests: a laboratory toolkit. J Clin Pathol 2013; 66:62-72.

Cadamuro J, Ibarz M, Cornes M, et al. Managing inappropriate utilization of laboratory resources. Diagnosis (Berl) 2019; 6:5-13.

van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. Jama 1998; 280:550-558.

Throughout the entire manuscript is reads as if the authors have identified ALL inappropriate testing, which is by not true. This is also the reason why percentages differ compared to those of other authors.

The title reads as if all parameters from the labs portfolio were considered in this study, which is also not true. Therefore, I would also suggest changing the title to reflect the content more precisely: Evaluation and cost estimation of inappropriateness in selected laboratory tests through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

Table 1:

AFP – “Test is not compatible with the other markers” Which other markers are referred to here?

The mentioned “biologic invariance rules” are so-called minimal re-testing intervals, therefore the following sentence is not completely correct: “The rules, based on the most recent international guidelines on the prescription appropriateness for laboratory tests” – this covers only the appropriateness concerning the time of testing. Again this statement suggests that all causes for inappropriateness were covered, which is not the case (by far).

The authors focus on reimbursement costs – I am not familiar with the Italian reimbursement system, however, it would be nice to compare these numbers with purely analytical costs (Costs for reagent, instrument and personnel)? Within a hospital setting, there mostly is something like an inner-hospital-reimbursement system, which does not necessarily reflect costs an external, insurancy companies or non-insured patient would have to pay.

Additionally, if possible, try calculating the add-on secondary costs due to inappropriate testing in your patients (costs due to prolonged time to diagnosis and length of stay, unnecessary follow-up diagnostics/therapy; etc) – or at least estimate these numbers within the discussion.

When comparing the amount of inappropriateness to findings of other studies, please keep in mind that the settings of Zhi, Meidani or Feldhammer differed to those in this study. As mentioned above, the wording “inappropriate testing” is misleading, as not all possible causes were included but just a few.

The discussion is way to short. I would urge the authors to discuss also the patient risk of inappropriate lab testing, which is far more important than costs of testing. Additionally, possible causes for inappropriate testing within your health care setting as well as ways of prevention should be discussed.

The conclusion on the other hand is too long – most parts thereof belong to the discussion.

Minor:

Please provide information on where your hospital/lab is located.

The fact that the most inappropriately ordered tests are AFP and Chlamydia PCR is biased by the low number of orders. Please discuss this in the discussion section.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Aug 6;15(8):e0237159. doi: 10.1371/journal.pone.0237159.r002

Author response to Decision Letter 0

14 Mar 2020

Dear editor,

I am thankful for your precious suggestions made to improve our work. They were carefully analysed and, as you can see from the attached reviewed document, taken into consideration and completely embraced.

On the basis of your suggestion, we reconsidered the whole paper and the meaning of “inappropriateness” and limited it only to the topic of “test overuse”. In “Discussion” section only comparisons with over-utilization rates of other authors have been included. Even the title has been changed in this light.

Although it is not feasible to apply appropriateness criteria and/or overutilization rules on all possible laboratory tests (over 1000), according to the other authors mentioned, we considered the most frequently performed tests and representative rules (applicable by automatic systems, considering a CCDSS use). We proceeded to specify this aspect in the “Setting” section (row 88).

Despite these limits (better explained in the “Strengths and limitations” section – row 194), the study collected data on a significant number of laboratory tests performed over a long period of time.

The common tests you missed probably fall into the “Complete Blood Count” panel (RBC, Hb, Ht, MCV, MCH, MCHC, RDW, PLTS). However, as you mentioned, we considered it appropriate to better specify the composition of the panel by adding a footnote to the tables and a sentence in the “Materials and methods” section (row 113).

Finally, we kindly ask you to include in the authors list Professor Maurizio Sanguinetti (President elect and Secretary General of ESCMID, Head of Department of Laboratory Sciences and Infectious Diseases of IRCCS Gemelli Hospital), for his fundamental contribution, as supervisor, in the final stages of this work.

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Dear reviewer #1,

I am thankful for your precious revision made to improve our work. Your suggestions were carefully analysed and, as you can see from the attached reviewed document, taken into consideration and completely embraced.

We report point-by-point answers to your questions:

1) As you suggested, we proceeded to explain the context of the Italian National Health System by specifying in the “Analysis” section that all the treatments were carried out free of charge for the hospitalized patients; hospitalization costs (including laboratory tests) were reimbursed by the Regional Health System according to Medicare Diagnosis Related Groups (MS-DRGs) (row 132).

2) The origin of the requests and the samples is described in the “Setting” section (row 89): only requests from hospital internal departments were monitored.

3) As explained at Point 1, Italian NHS fully covers hospitalization costs according to MS-DRG. This is a flat cost and it’s linked to the main diagnosis and procedures performed during hospitalization. A reduction in inappropriate laboratory tests does not affect the reimbursement costs and generates net savings for the hospital.

4) Only medical staff can make exam requests in Italian Health System. We proceeded to better specify this information (row 91).

5) No temporary physicians were included in the analysis. Only internal staff could make exam requests through the CPOE login interface.

6) Although your suggestion is acceptable and very interesting, unfortunately, the Local Ethical Committee did not allow us to track information related to physician demography, it’s specific role and the department of origin.

7) As described in “Strengths and limitations” section (row 199) physicians were not alerted and the CCDSS worked silently, without blocking or generating pop-ups. In this way we were able to assess the rate of inappropriateness in laboratory tests overuse without the confounding factor of physician’s awareness of a control system on laboratory requests.

8 and 9) All the examined tests have been ordered and counted as single tests, except for “Complete blood count” in which a panel of tests was ordered and counted in bulk (all together). As you suggested, we better explained this aspect in “Computerized Clinical Decision Support System (CCDSS)” section (row 113). Also, a footnote to the two tables was added.

Following your suggestion, we have reduced the “Conclusion” section by moving part of the text into the “Discussion” section (row 173).

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Dear reviewer #2,

I am thankful for your precious revision made to improve our work. Your suggestions were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

On the basis of your suggestion, we reconsidered the whole paper and the meaning of “inappropriateness” and limited it only to the topic of “test overuse”. Even the title has been changed in this light.

Although it is not feasible to apply appropriateness criteria and/or overutilization rules on all possible laboratory tests (over 1000), according to the other authors mentioned, we considered the most frequently performed tests and representative rules (applicable by automatic systems, considering a CCDSS use). We proceeded to specify this aspect in the “Setting” section (row 88) and to better explain this topic in the “Strengths and limitations” section (row 194).

In this light, as you correctly mentioned, in “Computerized Clinical Decision Support System (CCDSS)” section (row 104) we rephrased the over-utilization criteria specifying the number of biological invariance and incompatibility rules and the preponderance of minimal re-testing intervals criteria. Also, the sentence “The rules, based on the most recent international guidelines on the prescription appropriateness for laboratory tests” was rephrased (row 105).

In Table 1 we provided a better explanation of the rule applied to Alpha Fetoprotein (AFP): “Test cannot be requested with nonspecific tumor markers (CEA, CA125, CA19-9, CA15-3, TPA)”.

As you suggested, we proceeded to explain the context of the Italian National Health System by specifying in the “Analysis” section that all treatments were carried out free of charge for the hospitalized patients; hospitalization costs (including laboratory tests) were reimbursed by the Regional Health System according to Medicare Diagnosis Related Groups (MS-DRGs) (row 132). Italian NHS fully covers hospitalization costs according to MS-DRG. This is a flat cost and it’s linked to the main diagnosis and procedures performed during hospitalization. A reduction in inappropriate laboratory tests does not affect the reimbursement costs and generates net savings for the hospital.

Although your suggestion to consider analytical and secondary costs is acceptable and very interesting, unfortunately, our data does not allow to make a deep economic evaluation. However, in case of over-utilization criteria and minimal retesting intervals, costs due to a prolonged length of stay, unnecessary diagnostic tests and therapies are substantially negligible.

Following your indications, in the “Discussion” section (row 177) we proceeded to better clarify that comparisons with other authors’ results are affected by a reasonable variability in terms of settings and tests considered. However, only comparisons with over-utilization rates of other authors have been included.

According to your suggestion, we have reduced the “Conclusion” and increased the “Discussion” sections.

As you cited, some tests have a lower number of monthly requests (due to particular clinical needs that motivate the request, related to the epidemiology of the population). Nevertheless, over-utilization rates of each exam were calculated on the total of the requests on a period of 20 months. In this case, rates were sufficiently reliable because they were calculated on 3,520 alpha fetoprotein tests and 1,120 chlamydia trachomatis nucleic acid amplifications (NAATs).

Finally, information on hospital’s location (row 83) was added.

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (21.6KB, docx)

Decision Letter 1

Pal Bela Szecsi

4 May 2020

PONE-D-20-01565R1

Evaluation and cost estimation of laboratory tests overuse through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

PLOS ONE

Dear Dr. Tamburrano,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

The reviewers find that their suggestions are not sufficiently addressed and I concur.

However, as the manuscript has value, I allow another round of revision. I recommend taking the raised issues seriously.

==============================

We would appreciate receiving your revised manuscript by Jun 18 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Pal Bela Szecsi, M.D. D.M.Sci.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have performed an observational study on the overuse of laboratory tests in a large university hospital.

The manuscript has been revised, and the setting is now clearly described.

Major issues:

1) The Conclusions section is still far too long. Most of it could be transferred to the Discussion section.

Reviewer #2: Several of my suggestion were met inappropriately or not at all:

The mentioned biological invariance rules are basically minimal re-testing interval rules. In the CCDSS-section this should at least be put in brackets behind the term “biological invariance rules” so that the readers are readily able to follow your chain of thoughts.

My suggestion regarding the change of the title was not considered. Once again, I would urge the authors changing it to “selected laboratory tests” or “43 laboratory tests” instead of “laboratory tests”, otherwise readers may assume that all parameters from the labs portfolio were considered in this study, which is not true. Suggestion: “Evaluation and cost estimation of laboratory overuse in 43 parameters by the use of a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.”

The authors misunderstood my suggestion regarding the calculations of costs. Calculating these numbers based on local reimbursement fees may be interesting for the region the study was performed in. However, for all the other readers of PLOS One these numbers are meaningless, since they are not comparable. Therefore, I would suggest a clear and transparent cost calculation based on reagent costs and if possible also personnel and material (blood collection set) costs. As supplemental table a list of single costs for each analyte should be provided. Thereby, readers are able to estimate respective savings in their own setting.

Please also consider the publication rules of PLOS One in this context: “The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.”

I understand the authors stating that deep economic evaluation of secondary costs is not feasible. In addition thereof however, the statement “costs due to a prolonged length of stay, unnecessary diagnostic tests and therapies are substantially negligible.” is just a subjective belief and imho severely false. – this is just a personal statement and does not have to be discussed in the manuscript.

My suggestion regarding the discussion section was completely ignored: The discussion is way to short. I would urge the authors to discuss also the patient risk of inappropriate lab testing, which is far more important than costs of testing. Additionally, possible causes for inappropriate testing within your health care setting as well as ways of prevention should be discussed.

As for the next revision, please provide a sheet including my suggestion and your direct response to these, so that reviewers do not have to switch between documents and estimate which response belongs to which suggestion.

Reviewer #3: The study by Tamburrano et al addresses a highly relevant question: overuse of laboratory tests. The study focuses on the extra costs generated by overuse of laboratory tests mainly defined as tests repeated within a too short timespan. While this is one important factor causing overuse, another important factor is the increasing number of tests performed per patient usually generated through predefined panels of tests that are easily ordered. Regardless of the cause of overuse, the consequences reach beyond the immediate cost of the tests themselves: Time needed by clinicians to evaluated the results, additional procedures or treatments based on test results outside of reference ranges, sometimes by chance without any connection to the actual morbidity of the patient. A few lines in the discussion section on these points could improve the manuscript.

Specific points:

Line 4 in the “Setting section”: Ematology should be hematology

It is not clear how the mean and SD of the monthly requests and the over-utilization rates in table 2 are calculated. Please specify in the “analysis” section.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Henrik L. Jørgensen

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Aug 6;15(8):e0237159. doi: 10.1371/journal.pone.0237159.r004

Author response to Decision Letter 1

15 Jun 2020

Dear editor,

I am thankful for your precious suggestions made to improve our work. They were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

Reviewers’ suggestions were now fully addressed, and all the sections of the paper were improved.

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

 

Dear reviewer #1,

I am thankful for your precious revision made to improve our work. Your suggestions were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

Following your indications, the “conclusion” section has been shortened and the “discussion section” has been improved.

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

 

Dear reviewer #2,

I am thankful for your precious revision made to improve our work. Your suggestions were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

Following your indications, the title was changed, also “discussion” and “conclusions” sections were improved.

We report point-by-point answers to your questions:

• The mentioned biological invariance rules are basically minimal re-testing interval rules. In the CCDSS-section this should at least be put in brackets behind the term “biological invariance rules” so that the readers are readily able to follow your chain of thoughts.

As you requested, we put in brackets the term “minimal re-testing intervals” next to “biological invariance rules” (row 100).

• My suggestion regarding the change of the title was not considered. Once again, I would urge the authors changing it to “selected laboratory tests” or “43 laboratory tests” instead of “laboratory tests”, otherwise readers may assume that all parameters from the labs portfolio were considered in this study, which is not true. Suggestion: “Evaluation and cost estimation of laboratory overuse in 43 parameters by the use of a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.”

As you reported, we changed the title of the paper, following your suggestion, in “Evaluation and cost estimation of laboratory test overuse in 43 commonly ordered parameters through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital”.

• The authors misunderstood my suggestion regarding the calculations of costs. Calculating these numbers based on local reimbursement fees may be interesting for the region the study was performed in. However, for all the other readers of PLOS One these numbers are meaningless, since they are not comparable. Therefore, I would suggest a clear and transparent cost calculation based on reagent costs and if possible also personnel and material (blood collection set) costs. As supplemental table a list of single costs for each analyte should be provided. Thereby, readers are able to estimate respective savings in their own setting.

Please also consider the publication rules of PLOS One in this context: “The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.”

Regarding the calculation of the costs for each laboratory test, we applied the most reliable and independent calculation available within the Italian healthcare system. In fact, the calculation based on the breakdown of the different items and salaries would return an arbitrary evaluation based on the local hospital contracts with companies. Overall the provided estimates are official evaluation from National Italian authorities and therefore provide the best and standardized estimate we may provide to the PLOS ONE readers.

Different authors analysed this issue and drew similar conclusions.

- Irene et al., for example, say:

“To attach a cost to a single test or a panel of tests is not a straightforward task due to many factors, including the variability between direct and indirect/overhead costs, proprietary information from vendors on how consumables and equipment costs are broken down, service contracts, and the variability of how each clinical laboratory processes their laboratory tests”

and

“Although studies have presented “actual costs” of laboratory tests that are hospital- or province-specific, an “actual cost” of performing a laboratory test varies tremendously across hospitals, cities and provinces. Here, we present the reference median costs (RMC) of commonly ordered laboratory tests in a Canadian setting as a first step towards raising physician awareness in order to help enhance selective and appropriate laboratory test ordering practices.

and

“Calculating the cost of a laboratory test is complicated as there are a number of fluctuating factors associated with the costs of performing each test sample and the production of the laboratory test result. […] Due to the variability of direct expenses relating to the production of a test result, and variable volumes of tests processed between clinical laboratories, the RMC of each test presented here was determined by compiling price lists of all inclusive indirect costs.”

- Following the same concept, also Sarkar et al. say:

“Approximation of financial cost burden was based on test pricing from two commercial laboratories that might not reflect all commercial laboratories. Hence, the cost analysis should be considered only as an estimation.”

- Finally, Bertrand et al., say:

The measurement of the direct cost of these inappropriate procedures only included the direct cost of the test. For laboratory tests, each item or group of laboratory items was associated with a fixed value (e.g., 10 CHF for the CRP assay). […] The total cost was therefore obtained by adding the set of values associated with the repeated tests. This cost, therefore, did not take into account indirect costs, such as nursing services, time required to complete exams, equipment costs, etc. […] The costs generated by these repeated procedures were estimated on the basis of the unit rates charged at the HUG at CHF 20,000 over one year.”

1. Ma, I., Lau, C. K., Ramdas, Z., Jackson, R., & Naugler, C. (2019). Estimated costs of 51 commonly ordered laboratory tests in Canada. Clinical Biochemistry, 65, 58–60. https://doi.org/10.1016/j.clinbiochem.2018.12.013

2. Sarkar, M. K., Botz, C. M., & Laposata, M. (2017). An assessment of overutilization and underutilization of laboratory tests by expert physicians in the evaluation of patients for bleeding and thrombotic disorders in clinical context and in real time. Diagnosis, 4(1), 21–26. https://doi.org/10.1515/dx-2016-0042

3. Bertrand, J., Fehlmann, C., Grosgurin, O., Sarasin, F., & Kherad, O. (2019). Inappropriateness of Repeated Laboratory and Radiological Tests for Transferred Emergency Department Patients. Journal of Clinical Medicine, 8(9), 1342. https://doi.org/10.3390/jcm8091342

• I understand the authors stating that deep economic evaluation of secondary costs is not feasible. In addition thereof however, the statement “costs due to a prolonged length of stay, unnecessary diagnostic tests and therapies are substantially negligible.” is just a subjective belief and imho severely false. – this is just a personal statement and does not have to be discussed in the manuscript.

Following your indications, the statement was reported in the correct formula in the “conclusions” section (row 222-224)

• My suggestion regarding the discussion section was completely ignored: The discussion is way to short. I would urge the authors to discuss also the patient risk of inappropriate lab testing, which is far more important than costs of testing. Additionally, possible causes for inappropriate testing within your health care setting as well as ways of prevention should be discussed.

Following your indications, the issues you raised were included and argued in the “discussion” section (row 195-203).

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Dear reviewer #3,

I am thankful for your precious revision made to improve our work. Your suggestions were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

Following your indications, the issues you raised were included and argued in the “discussion” section (row 195-203).

As you reported, “ematology” was corrected in “hematology”.

Finally, a description of the descriptive statistics performed is reported in the “analysis” section (row 127-129).

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Attachment

Submitted filename: Response to reviewers.docx

Click here for additional data file. (21.3KB, docx)

Decision Letter 2

Pal Bela Szecsi

3 Jul 2020

PONE-D-20-01565R2

Evaluation and cost estimation of laboratory test overuse in 43 commonly ordered parameters through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

PLOS ONE

Dear Dr. Tamburrano,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Please correct the few remaining issues mentioned by the reviewers.

It will not require renewed review.

==============================

Please submit your revised manuscript by Aug 17 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Pal Bela Szecsi, M.D. D.M.Sci.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The Authors have performed a study on the over-utilization of laboratory tests. This is an important issue due to the ever-increasing demand for laboratory services and diminishing health care resources.

Minor issues:

1) The provider of the Prometeo Appropriatessa Software should be provided.

Reviewer #2: The autors have met most of my comments. However, I still believe that the Discussion is too thin and that the authors could dive deeper into patient safety risks of inappropriate lab testing - actually this should be the main motivation for improvement actions.

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Aug 6;15(8):e0237159. doi: 10.1371/journal.pone.0237159.r006

Author response to Decision Letter 2

14 Jul 2020

Dear editor,

I am thankful for your precious suggestions made to improve our work. They were carefully analysed and, as you can see from the attached reviewed document, completely embraced.

Reviewers’ suggestions were now fully addressed, and all the sections of the paper were improved.

The provider of the Prometeo Appropriatezza Software has been provided in parentheses (row 97).

The discussion section has been improved by adding a discussion on patient safety risks related to inappropriate routine tests (row 203-210 and 214-215).

Many thanks for your consideration.

Sincerely,

Dr Andrea Tamburrano

Section of Hygiene - Institute of Public Health

Università Cattolica del Sacro Cuore di Roma

Attachment

Submitted filename: Response to reviewers.docx

Click here for additional data file. (16KB, docx)

Decision Letter 3

Pal Bela Szecsi

22 Jul 2020

Evaluation and cost estimation of laboratory test overuse in 43 commonly ordered parameters through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

PONE-D-20-01565R3

Dear Dr. Tamburrano,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Pal Bela Szecsi, M.D. D.M.Sci.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Acceptance letter

Pal Bela Szecsi

27 Jul 2020

PONE-D-20-01565R3

Evaluation and cost estimation of laboratory test overuse in 43 commonly ordered parameters through a Computerized Clinical Decision Support System (CCDSS) in a large university hospital.

Dear Dr. Tamburrano:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Pal Bela Szecsi

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Dataset

    (XLSX)

    Click here for additional data file. (75.2KB, xlsx)
    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (21.6KB, docx)
    Attachment

    Submitted filename: Response to reviewers.docx

    Click here for additional data file. (21.3KB, docx)
    Attachment

    Submitted filename: Response to reviewers.docx

    Click here for additional data file. (16KB, docx)

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES