Development of an automated quality control and assay performance management system for biochemical genetics laboratory

Abstract

Objectives

Quality control (QC) in clinical laboratory is critical to ensuring quality and accuracy of patient results. However, QC monitoring is complicated in the multi-analyte, multi-instrument assays common to biochemical genetics laboratories. Comprehensive off-the-shelf QC management systems optimized for such highly complex assays and platforms are relatively scarce. A manual QC review process can impact laboratory productivity and increase risk of errors. Here we describe a novel software application that integrates, processes, and displays QC statistical parameters from multiple instruments in near real-time results by automated processor.

Methods

A customizable, cloud-based software application was developed to centralize the information, automate an extra review step in the QC review process and increase clinical utility. We monitored time spent on each step of QC review and QC range assignments before and one year after implementing the program and documented quality improvements.

Results

This QC program has modules for different assay platforms. The program's functions include automated collection and assay data analysis, Levey-Jennings charts with integrated data from multiple instruments, graphical data visualization, instrument data centralization, assay monitoring, and a QC audit trail. The program also generates email notifications for QC lot expiration, QC review reminder, and critical results alert enabling prompt communication to providers. After the first year, this program provided 81 %-time reduction of hands-on time.

Conclusion

This program improves assay quality and provides considerable time savings. One benefit of this software is the ease of updating program capabilities and customizing them to meet specific and changing needs, especially for high-complexity testing.

1. Introduction

Clinical Laboratory Improvement Amendments of 1988 (CLIA) regulations require all clinical laboratories to implement quality control (QC) procedures that monitor the accuracy and precision of the complete testing process. Rigorous QC procedures can help to detect errors prior to the release of patient results. A comprehensive QC process involves the integration of both Internal Quality Control and External Quality Control (Proficiency Testing). Internal QC is performed daily using multiple matrix-appropriate samples of known concentrations, which are included in every assay setup [1,2]. Various statistical parameters are calculated from the QC data to verify the accuracy of reported results and detect potential system deficiencies or errors so that they can be corrected prior to releasing patient results.

Biochemical genetics tests are primarily used for screening, diagnosis, and treatment monitoring of patients with inborn errors of metabolism (IEM) and, in some cases, carrier and prenatal testing [3,4]. Although not defined as specialties or subspecialties under the Clinical Laboratory Improvement Amendments (CLIA), biochemical genetics tests are considered high-complexity tests. Laboratories that perform these tests must meet the applicable general CLIA requirements for nonwaived testing and the personnel requirements for high-complexity testing [4]. The unique challenges of biochemical genetic testing were recognized by the College of American Pathologists (CAP), which introduced the Clinical Biochemical Genetics checklist in 2011 specifically to help these laboratories ensure the highest levels of quality.

Nearly all the most frequently ordered biochemical genetics tests, such as amino acids, acylcarnitine, and urine organic acid analyses, are multiple-analyte assays. In many cases, the analytes included in a test panel may be heterogeneous, with varying chemical properties that require complicate extraction, analysis and consequently, quality control. Therefore, the process of monitoring QC parameters has an additional layer of complexity, especially when multiple instruments are also used for a particular test. The relative scarcity of comprehensive, user-friendly, off-the-shelf QC management systems capable of managing highly complexed platforms such as LC-MS/MS with single- and multiple-analytes across different instruments only compounds the difficulties experienced by these laboratories. The commercially available QC programs do not have all features to fully support biochemical genetics laboratory QC activity.

As a result, biochemical genetics test QC review processes are often time-consuming and lack automated or semi-automated features for assessing and tracking assay performance. A manual QC review process can not only impact laboratory productivity but also increase the risk of data entry errors. In contrast, commercially available software was not designed to serve clinical biochemical genetics laboratories specifically and may lack the ability to nimbly adapt to new assay requirements, which is crucial for large laboratories with comprehensive test menus and various assay platforms.

Here we describe a novel software application that integrates, processes, and displays QC statistical parameters from multiple instruments in near real time.

2. Methods

A customizable, cloud-based software application was developed to facilitate QC data review and monitor assay performance, with modules for different assay platforms displayed on a clickable home page. The program's functions include automated data collection directly from the instrument, automated analysis of assay data, generation of Levey-Jennings (LJ) charts with integrated data from multiple instruments and/or multiple HPLC/UPLC channels over customizable date ranges, graphical data visualization, instrument data consolidation and centralization, assay monitoring, tracking of positivity or abnormal results rates, and a time-stamped QC audit trail. It also monitors internal standard (IS) data for those assays using stable isotope dilution and can automatically identify samples in which IS recovery is low.

The software was developed and has been maintained by the Quest Diagnostics Bioinformatics team. The program was written in PHP and JavaScript using the Symfony 4 framework and Microsoft SQL Server and hosted on internal servers via a Docker container instance. The servers are securely connected to Quest's network, and all users must have security clearance from Quest Diagnostics before gaining access to the program. There are several access levels to restrict a user's permission based on their role as the laboratory user.

The software is designed to support the streamlined implementation of new assays in clinical laboratory settings. To incorporate a new assay, the database configuration is updated to include assay-specific parameters such as instrument types, analytes, lot numbers, default total allowable error (TEa), significant figures, calibration settings, applicable Westgard rules, and the appropriate email notification groups. When necessary, a custom parsing algorithm is developed to extract relevant data from assay input files. The new assay is also integrated into the user interface to ensure accessibility, with modifications made as needed to accommodate any deviations from standard functionality.

To evaluate process improvements, from 2019 to 2020, we monitored time spent on each step of QC review and QC range assignments before and after implementing the automated QC program and documented quality improvements. We also compared the critical functions of our QC program with a commercial QC program.

3. Results

Program development was completed in December 2019 and launched in 2019 with 12 clinical biochemical genetics assays. From 2019 until March 2025, the program has expanded to cover 26 clinical biochemical genetics and clinical chemistry assays (Fig. 1). This program provides the following features:

Fig. 1.

QC program homepage.

Data collection

• -Automated collection and analysis of processed assay data
• -Data upload capability directly from the instrument, eliminating the need for a USB drive or copying and pasting files from the instrument to a computer desktop (Fig. 2A and B)

Fig. 2.

A. Data upload page. Data from the instrument is uploaded directly from the output file to the QC program. The batch file is uploaded after the correct file is selected. B. Uploaded data page. After the batch file is selected and data is uploaded, the batch file name, date run, date reported, technician name who uploaded the file, instrument name and QC lot names are shown at the top of the page. The table demonstrates the value of each QC analyte, +/-2SD and +/-3SD range limits, applicable Westgard flags, and the status of each QC point (passes or fails).

Data analysis and monitoring

• -Graphical visualization of assay data with customizable date ranges
• -
  Levey-Jennings (LJ) chart data integration across multiple instruments (Fig. 3)

  • o
    Consolidation, centralization, and streamlining of instrument data
  • o
    Color-coded data points on LJ charts to identify individual instruments, with options to view data from any or all instruments at any time
• -Plotting of trendlines in customizable date ranges to more easily identify biases or trends in large data sets
• -Calculation of statistical parameters including mean, standard deviation (SD), and coefficient of variation (CV) for individual instrument or multiple instruments combined, p and t values, and Westgard rules and warnings (Fig. 4)
• -Calculation of percent deviation from QC phase-in values, patient results distribution, and internal standard mean response per batch
• -Retrospective data is stored and is available to re-evaluate existing reference intervals.
• -Assay monitoring, including positivity rate, patient median values per batch, and percent abnormal values (Fig. 5)

Fig. 3.

Integrated Levey-Jennings (LJ) chart. Color-coded data of integrated LJ chart for 3 amino acids from 3 different instruments, LC1, LC3 and LC5.

Fig. 4.

Statistical parameters. Data from urine porphobilinogen (PBG) illustrated in a table format, demonstrating statistical parameters such as mean, standard deviation (SD), and t and a p value for low, mid, and high QC pools from one instrument.

Fig. 5.

Assay monitoring. Percent positive rate and median illustrated in graph and table formats after the time window is selected (at the top of the page). The positive rate is calculated by the total number of samples which had values outside their age specific reference intervals divided by the total number of samples in the same batch.

Review, reminder and notification

• -Generates automated email notifications for pending QC lot expiration
• -Generates reminders for weekly/monthly supervisor/Director reviews
• -Option for Director performing monthly review to automatically alert the assay general supervisor if an issue that needs follow-up action is observed or noted (Fig. 6)
• -Abnormal patient results alert

Fig. 6.

Records of weekly and monthly QC reviews. Laboratory personnel can add comments about QC performance and action items. The reviewer's name and the date and time when the comment is entered are recorded in the QC program.

When a new lot of QC is being phased in, the software calculates the mean, standard deviation (SD), and sigma levels for each analyte based on phase-in data. Acceptable ranges, typically defined as the mean ± 4 or 5 sigma, are reviewed and approved by the laboratory director to ensure assay accuracy. Westgard rules are then applied to monitor multiple analytes within each assay, enabling the detection of systematic and random errors during routine QC assessment. Any out-of-range values and their potential clinical impact are reviewed by the laboratory director. The software flags analytes that exceed predefined thresholds within specific batches and also detect patterns or trends over defined time intervals (e.g., weeks or months). When an outlier is identified, the laboratory director reviews the entire affected batch to determine any potential clinical impact on the patients. Additionally, even when individual values remain within acceptable limits, the software can reveal emerging trends. These trends serve as early indicators, prompting investigation before values fall outside the allowable range.

The user can choose or change the Westgard rules by going to configuration and select the rule, the options are 1_3s,1_2s,2_2s,r_4s,4_1s and 10_x. Custom rules can also be programmed if the standard options are not sufficient.

The software does not evaluate the acceptability of the calibration curves. The process of creating the calibration curve, applying the suitable mathematic model to fit the best curve, and determining analyte concentrations in QC and patient samples occur during assay processing with the instrument software. The acceptability of calibration curves is determined based on criteria defined in the assay standard operating procedure (SOP). The concentration values obtained from the calibration curve are uploaded to our QC software. However, because assay batches are uploaded in the program, the calibration curve data can be accessed from within the program by clicking on the download icon for a particular batch. Depending on the formatting of the instrument data file, specific information regarding calibration curve parameters can be reviewed from within the program.

The data stored in the program is used to generate a QC audit trail. Within the QC program, the data are stored indefinitely and can be accessed either by assay date or by QC lot number. Archival assay data can be retrieved for review or trouble shooting. Data can be easily visualized and centralized, which helps to detect assay problems early and minimizes the potential of human errors. Since the results of biochemical genetic tests help clinicians to monitor and treat patients with IEM, we set up analyte- or assay-specific cut-off criteria that allow the program to automatically send an alert email to the genetic counselors and laboratory directors when a critical result is imported from an assay batch, enabling prompt communication to providers. The program sends “abnormal result” alert emails when the values of certain analytes fall outside of pre-determined cut-offs, or when the results exceed the assay analytical measurement range (AMR) and will require dilution. Critical analyte threshold values can be customized according to clinical significance. In addition, in the future, predefined ratios of critical analytes can be configured to detect specific patterns associated with specific metabolic disorders. This abnormal alert notification has already helped physicians receive critical results and patients receive appropriate treatment before the final results were released.

One year after implementing this program in our laboratory, we have reduced time spending on each step of QC including time consuming manual QC processes and repetitive tasks by 81 %, leading to savings of approximately 2.75 laboratory technician full-time-equivalents (FTE) (Fig. 7). The program allows for a paper-free QC process, which reduced the usage of paper from 660 to 700 pages of QC documents per year for a manual process to almost none.

Fig. 7.

Time spent on different QC review processes. The graph demonstrates the time spent on each QC review step before the QC program was implemented (manual process) and after the program was implemented (automated process). Critical results management includes 2 steps: critical results check and director assay review.

Assay acceptability includes 5 steps: assay set up, QC entry and review, positive rate check, internal standard check, and report results.

QC phase-in includes 4 steps: run phase-in samples, recording phase-in values, calculating mean, SD and CV, and QC ranges establishment.

Commercially available QC software solutions can import data from middleware and perform most QC tasks described in our program. However, the commercially available software cannot provide integrated data from multiple analytes and multiple instruments or abnormal result alerts. The advantages of our software over the commercial ones are summarized (Table 1).

Table 1.

Functionality comparison: the QC program versus the commercial program.

Features	QC program	Commercial software
Access
	A web browser	Commercial software is installed on a device
Data collection
QC and assay's batch data	Information is automatically uploaded from instrument output files, or the Excel files are automatically uploaded in the program	Manual process to transfer data to the Excel files
Automated Assay acceptability	The program provides information if a batch meets QC acceptability requirements	The program provides information if a batch meets QC acceptability requirements
Data analysis and monitoring
LVJ chart data comparison if single analyte is run by more than one instrument	Data from all instruments can be presented at the same time, on one page or on one plot	Separate LVJ chart for each instrument
Visual Presentation	Graphical visualization	Graphical visualization
Statistical calculation	Program provides all necessary statistical data including mean, standard deviation, and coefficient of variation (%CV) for individual instrument or combined results from multiple instruments	Only Mean, Std Dev and %CV are calculated. No option to review the combined results from multiple instruments
	Flagging for Westgard rules such as 1-2S, 2-2S, R-4S, 1-3S	Flagging for Westgard rules such as 1-2S, 2-2S, R-4S, 1-3S
	P and t values, trendlines and percent differences from phase-in values	Nonexistent feature
	Abnormal patients' results distribution by assay/batch	Nonexistent feature
	Internal standard response means and %CV per batch	Nonexistent feature
Ability to customize Westgard rules	The appropriate Westgard rules can be customized for any assay	The appropriate Westgard rules can be customized for any assay
Phase-in QC range determination	Data is uploaded automatically as an Excel file and the program calculates observed, 3 Sigma, 4 Sigma, 5 Sigma, and 6 Sigma SD based on the assay total allowable error (TEa) limits to allow for flexibility in phase-in QC range determination	Data is entered manually, and the program provides phase-in ranges based on the assay total allowable error limits
Monitor changes which may impact QC	Dates of starting new reagent and lot numbers, instrument service, new calibration	Dates of starting new reagents and lot number, instrument service, new calibration
Review and reminder
Test performance monitoring	Real time monitoring and auditable	Real time monitoring and auditable
Reminder and alert systems	There is a reminder when QC lot is nearing expiration	There is a reminder when QC lot is nearing expiration
Expiration of Reagents	Program alert 2 months before the expiration date	Program alert 1 month before the expiration date
QC weekly and monthly review due	For weekly QC reviews, a reminder is sent every Tuesday	Nonexistent feature.
	For monthly QC reviews, a reminder in sent from 2nd Tuesday of the month
Notification
Critical values	Critical results which are above or below a designated cut off are emailed to directors and genetics counselors. Clients are notified verbally	Nonexistent feature

4. Conclusion

This program has improved overall assay quality in our laboratory and has saved labor time by automating and streamlining time-consuming data entry steps, reducing clerical errors, and facilitating the monitoring of assay performance at the batch level. Integrated assay data can be visualized by individual channel for multichannel instruments as needed, and statistical parameters are calculated with automated flagging when the QC requirements or rules for an assay are not met. The long-term data storage is cloud-based, and data can be retrieved and reviewed for trouble-shooting purposes, for retrospective studies, or for assessing assay performance over time.

One of the major benefits of our software is the ease with which we can update the program capabilities and customize them to meet our specific and changing needs. The modular design ensures that as new tests are developed and operationalized, they can readily be added to the program. The ability to be nimble and to adapt to new assay or regulatory requirements is especially crucial for large laboratories with a comprehensive and varied test menu that incorporates multiple assay platforms. Development and implementation of program modifications in some commercially available software can be a much more arduous process.

CRediT authorship contribution statement

P. Tanpaiboon: Writing – review & editing, Writing – original draft, Visualization, Supervision, Methodology, Formal analysis, Conceptualization. D. Salazar: Writing – review & editing, Writing – original draft, Formal analysis, Data curation, Conceptualization. M. Pan: Writing – review & editing, Writing – original draft. L. Xu: Software, Data curation. R. Sharma: Writing – original draft, Methodology, Formal analysis, Data curation, Conceptualization.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

All authors disclose current/previous employment by Quest Diagnostics.

Data availability

The authors do not have permission to share data.