Data pipeline quality: development and validation of a quality assessment tool for data-driven algorithms and artificial intelligence in healthcare

Abstract

Objectives

To develop and validate a tool for standardised quality assessment of data-driven algorithms in healthcare, focusing on the underlying data pipeline.

Methods

Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence (DATA-CARE) was iteratively developed from the established Quality In Prognosis Studies framework, selected after reviewing 10 existing quality assessment tools for observational and artificial intelligence studies. DATA-CARE evaluates five quality domains of the data pipeline: study population, data, algorithm, outcome and report transparency. Each domain comprises three to five quality criteria. With a total score of 75 points, study quality is categorised as low (<45), moderate (45–59) or high (≥60). DATA-CARE was validated during a systematic review on data-driven algorithms using continuous physiological monitoring data within the paediatric intensive care unit. Two independent reviewers performed quality assessment using DATA-CARE of included studies. Tool validation was evaluated using inter-rater agreement and intraclass correlation coefficient (ICC).

Results

DATA-CARE demonstrated robust inter-rater agreement (93.5%) with ICC 0.98 (95% CI 0.96 to 0.99). Of 3858 screened studies, 31 were reviewed in the use case, describing diverse algorithms. Studies were predominantly low (32.3%) to moderate (41.9%) and sporadically (25.8%) high quality.

Discussion

Predominance of low-to-moderate quality studies reveals critical barriers to clinical implementation of data-driven algorithms, including low quality data capture and processing, lacking validation strategies and non-transparent reporting of findings.

Conclusions

DATA-CARE allows standardised and reliable critical appraisal for a wide variety of algorithms, addressing current gaps in standardised and reproducible algorithm development.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Algorithm quality hinges on its underlying data pipeline, specifically source data, processing and analytical methodology and overall reproducibility. Existing quality assessment tools often neglect this, limiting the ability to review and reproduce algorithms and thus hindering their clinical implementation.

WHAT THIS STUDY ADDS

• This study introduces and validates Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence (DATA-CARE), a quality assessment tool that evaluates five key domains of the data pipeline. It demonstrates high inter-rater reliability and reveals that most reviewed studies in a paediatric intensive care use case are of low-to-moderate quality.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• DATA-CARE provides a standardised and reliable framework for evaluating algorithms, supporting reproducible research and transparent reporting. Its adoption could guide researchers, reviewers and policymakers in improving the quality and clinical readiness of data-driven algorithms in healthcare.

Introduction

Data-driven healthcare, powered by artificial intelligence (AI) and big data analytics, has emerged as a transformative force in modern healthcare.¹ The ability to harness data for bedside monitoring and decision support via actionable algorithms holds the promise of improved and personalised care.¹ Despite increasing research on this topic, a critical gap persists between algorithm development and clinical implementation, often attributed to lack of standardised methodology.2,4

In data-driven healthcare, continuously measured data are used to guide clinical decision making.⁵ Patients constitute the source from which data follows a pipeline where raw digitalised signals from bedside monitors and devices are collected, processed and used in algorithmic analysis to produce actionable insights (figure 1).⁶ The quality of the data and integrity of each step in the data pipeline affects derived insights and their validity. A robust and reproducible data pipeline is therefore of critical importance to ensure data-driven healthcare is effective, reliable and generalisable. However, to our knowledge, there is no tool for quality assessment of the data pipeline.

Figure 1. Schematic overview of the data pipeline. Raw data are acquired from various sources and ingested (in batches from files, via streaming of cloud databases) into the data pipeline where it undergoes systematic processing (such as noise cleaning) to prepare it for analysis (machine learning, statistical modelling, signal analytics, dashboarding). This is a simplified overview; there may be more details to the data pipeline (eg, aggregation of preprocessed data) or feedback loops (eg, following analysis, preprocessing is enhanced).

Quality assessment tools are often developed for a specific study design or objective, focusing on individual components rather than providing a comprehensive assessment of the data pipeline.^{7 8} As such, available tools tend to be fragmented and limited in scope, fail to capture critical domains of the data pipeline or are too focused on specific algorithm types.^{7 8} Algorithm quality and returned output are largely dependent on source data and how this is ingested and processed in the pipeline.⁹ Perhaps most important of all is the ability to reproduce and validate the data pipeline. This requires transparency in research, in particular on study population selection (data source), data quality and processing, algorithm development and validation and (desired) outcomes. Available tools that address these domains may pose a suitable basis for quality assessment of data-driven healthcare, but need to be adjusted to become widely applicable to studies on data-driven healthcare.

A quality assessment tool for data-driven healthcare enables critical appraisal of existing research and guides towards standardised and reproducible data pipelines for actionable clinical algorithms. Therefore, the aim of this study was to develop and validate a quality assessment tool for data-driven healthcare, based on the domains study population, data, algorithm, outcome and report transparency adjusted from the Quality In Prognosis Studies (QUIPS) framework.

Methods

Tool development

Tool development occurred iteratively in a six-member working group, including an epidemiologist (RdJ), clinicians (RdJ, JWK), data scientists (EvT, BvW) and engineers (AS, DT) at Erasmus MC Sophia Children’s Hospital and Delft University of Technology (figure 2). The tool was developed through refinement and expansion of the QUIPS, which were most suitable among ten existing tools identified for observational and AI studies (online supplemental table S1).10,24 The QUIPS tool was chosen as it guides systematic and comprehensive critical appraisal in a user-friendly and widely applicable format, with domains that adhere to the data pipeline. Original domains study participation, prognostic factor measurement, outcome measurement, study confounding and statistical analysis and reporting were translated to data-driven healthcare. The domains were divided into criteria that determine the quality per domain, covering the data pipeline from input to output. Quality domains and criteria were ranked based on applicability and clinical relevance for selection. Guidelines were written with signalling questions and a scoring system was adopted from de Jonge et al.^{25 26} The score per domain reflects robustness of that domain and the overall score reflects overall quality and risk of bias in a study with regard to the data pipeline. Per domain, 15 points can be allocated, distributed over 3–5 criteria, yielding 17 criteria in total with a maximum score of 75 points. Each criterion is scored out of 3 or 5 points depending on the number of criteria within the domain, with 50% penalty for incomplete or methodologically flawed information. A score of 80% (≥60 points) constituted high quality, between 60% and 80% (45–59 points) moderate quality and below 60% (<45 points) low quality.²⁶

Figure 2. Schematic overview of stepwise development of DATA-CARE in working group. Criteria for quality assessment were formulated and reformulated iteratively during consensus meetings. AI, artificial intelligence; DATA-CARE, Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence.

Quality domains

The study population domain guides readers to assess whether the target population (data source) was captured and whether risk of bias was introduced when certain participants were (not) included in the study, for example, due to selection bias, ascertainment bias or loss to follow-up.^{12 27} It is based on the three criteria: recruitment, inclusion and exclusion criteria and baseline study population.

The data domain assesses whether data capture was adequate and of sufficient quality. It is the only domain containing five criteria: data acquisition, data set size and balance, missing data, preprocessing and feature derivation. The criteria adhere to the data quality dimensions of accuracy, completeness, redundancy, readability, accessibility, consistency, usefulness and trust.²⁸ During data acquisition, sample frequencies determine data resolution and may cause aliasing if not appropriate to signal bandwidth.²⁹ Characteristics like size, balance and completeness significantly impact algorithm performance and stability. Imbalanced data lead to overrepresentation of the majority class and may bias the algorithm to better distinguish this class.³⁰ Missing data can introduce bias similar to loss to follow-up, as eligible participants have incomplete data or data of insufficient quality to be included in analysis.³¹ Missing data can be missing completely at random, missing at random or missing not at random, where the latter two introduce high risk of bias if not accounted for during analysis.³¹ Preprocessing and feature derivation are important as medical data cannot be mistaken for ground truth, affecting algorithm generalisability and computational efficiency while also posing a risk of bias and/or overfitting.32,34

The algorithm domain assesses whether the computational approach to derive an outcome of interest was standardised, robust and valid across participants and its generalisability to the target population, based on algorithm architecture, development and evaluation. An algorithm should be appropriate to the intended use and requires systematic data partitioning, configuration and validation.^{35 36} Data partitioning refers to the train–test split, ideally on a participant level to prevent data leakage which may cause overfitting and reduce algorithm generalisability.³⁵ If algorithms are patient-tailored, this should be explicitly stated. To advance to population inference, algorithms require internal (unseen test set) and external validation (newly sampled data).³⁷ Complementary performance metrics are vital to clinical interpretation, including metrics of significance (eg, p values) and uncertainty (eg, CIs) or of discrimination (eg, balanced accuracy) and calibration (eg, R² curves).^{37 38} Discriminative metrics allow the reader to interpret how well the algorithm can identify positive and negative instances, and calibration metrics allow interpretation of how reliable this identification is.³⁸

The outcome domain assesses whether the outcome was standardised and measured reliably across participants and judges the risk of bias due to mislabelling, as algorithms can only be implemented in clinical practice if they can reflect on an outcome of interest. To ascertain clinical implications, the outcome and its labelling must represent a ground truth, that is, the objective reality based on (reliable) measurement or observation.¹⁵ Outcomes need clear definitions and standardised assessment, ideally via the reference standard. Labelling maps outcomes to individual data points is especially essential for supervised algorithms which must learn to reproduce outcome labels, while unsupervised algorithms create custom labels.³⁴ Standardisation of outcome and labelling may be limited by inter-rater variability.³⁹ If a ground truth is not available (eg, clinical deterioration), labels may be engineered (unsupervised) or derived from clinically relevant endpoints (eg, therapeutic intervention), with implications considered in the discussion.

The report transparency domain assesses the risk of bias due to incomplete reporting or inappropriate statistical methodology, based on presentation of data and findings, reporting of results and statistical analysis. Data and findings should reflect the study objective and methods but are not overstated and limitations are discussed. Selective reporting is avoided by presenting all results, including algorithm subtypes and subgroups where applicable. On indication, sensitivity analysis and/or post hoc analysis is reported. Statistical evaluation was specified, statistical assumptions have been met and results are consistently presented throughout the study (eg, OR as positive decimal). To minimise bias and contribute to fairness, adequate measures of significance or uncertainty are provided with correction for multiple testing where applicable.⁴⁰ This allows interpretation of findings, which is dependent on objective, sample sizes and assumptions.⁴⁰

Tool validation

Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence (DATA-CARE) was validated during a use-case on data-driven healthcare in the paediatric intensive care unit (PICU). Studies were identified using a systematic review conducted in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.⁴¹ The search strategy is detailed in online supplemental A. Study selection was based on study design, setting, data and sampling frequency as described in online supplemental B. Data synthesis included information on study methodology and characteristics of the data pipeline (online supplemental C). Included articles were qualitatively assessed by two reviewers (EvT, BvW). After quality assessment, scoring was compared, with disagreements resolved during consensus meetings (EvT, BvW). Tool validation was based on inter-rater agreement (%) and intraclass correlation coefficient (ICC) for total quality score and quality category.

Results

The DATA-CARE tool

We present DATA-CARE, with quality domains and criteria summarised in table 1. The full tool and guidelines are available in online supplemental D.

Table 1. Overview of DATA-CARE and point allocation for scoring.

Criteria	Score*
	+	±	−
Study population
Recruitment	5	2.5	0
Inclusion and exclusion criteria	5	2.5	0
Baseline study population	5	2.5	0
Data
Data acquisition	3	1.5	0
Data set size and balance	3	1.5	0
Missing data	3	1.5	0
Data preprocessing	3	1.5	0
Feature derivation	3	1.5	0
Algorithm
Algorithm architecture	5	2.5	0
Algorithm development	5	2.5	0
Algorithm evaluation	5	2.5	0
Outcome
Definition of outcome	5	2.5	0
Method and setting of outcome assessment	5	2.5	0
Outcome labelling	5	2.5	0
Report transparency
Presentation of data and findings	5	2.5	0
Reporting of results	5	2.5	0
Statistical analysis	5	2.5	0

Adapted from Hayden et al and de Jonge et al with permission.12 25 26

^*Scoring symbols refer to maximum score (+), average score (±) and minimum score (−).

DATA-CARE, Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence.

Data-driven healthcare in the PICU

Out of 3858 studies identified, 31 were included for quality assessment after duplicates removal, screening and full-text retrieval (online supplemental figure S1). Main reasons for exclusion were non-monitoring objectives and discontinuous data. The included studies and characteristics of the data pipeline are presented in online supplemental tables S2 and S3. Studies were generally retrospective (23 (74.2%)) with median sample size 90 (28–215) participants. Algorithms were mainly AI (16 (51.6%)), specifically machine learning, followed by signal analysis (12 (38.7%)) and were mainly intended for prediction (13 (41.9%)) or monitoring (10 (32.3%)). Studies on AI typically evaluated multiple classifiers, most commonly neural networks and random forest (8 (25.6%)). None of the included studies developed dashboards or provided decision support. Data sampling and window sizes varied widely. 19 studies (61.3%) used separate data sets for algorithm development and validation and specified data partitioning, 9 of which introduced data leakage. Cross-validation was reported by 12 studies (38.7%), optimisation by 16 studies (51.6%) and handling of imbalanced data by 6 studies (19.4%), all mostly AI. Additional validation post train–test procedure was mentioned by three studies (9.7%). One study (3.2%) performed external validation using prospectively collected data. Performance metrics were mainly discriminatory, including area under the receiver-operator curve (AUROC), accuracy, sensitivity and specificity.

Tool validation

Mean quality score was 50.6 (12.6) points, with 10 (32.3%) low quality studies (<45 points), 13 (41.9%) moderate quality studies (45–59 points) and 8 (25.8%) high quality studies (≥60 points). Quality scores per study are available in table 2. Most points were withheld in data and report transparency domains as studies neglected to report on data set size and balance, missing data and/or lacked statistical rigour. Most points were allocated in the outcome and algorithm domains. Inter-rater agreement was 63.6% for total score and 93.5% for quality category, with ICC 0.98 (95% CI 0.96 to 0.99) and 0.95 (95% CI 0.90 to 0.98). Across quality domains, inter-rater agreement for domain scores was 64.5% for study population, 67.7% for data, 70.1% for algorithm, 70.1% for outcome and 67.7% for report transparency.

Table 2. Quality assessment of included studies using DATA-CARE.

Study	Study population	Data	Algorithm development	Outcome	Report transparency	Score
Singh et al 32	15	12	10	15	12.5	64.5*
Azriel et al49	7.5	10.5	12.5	12.5	10	53
Rooney et al50	15	4.5	10	15	10	54.5
Badke et al51	12.5	10.5	15	15	12.5	65.5*
Joram et al52	15	10.5	10	10	10	55.5
Amiri et al53	5	6	10	15	2.5	38.5
Castineira et al54	7.5	13.5	7.5	12.5	2.5	43.5
Sorensen et al55	12.5	7.5	10	10	12.5	52.5
Bose et al56	5	10.5	10	12.5	12.5	50.5
Marsillio et al57	15	6	15	5	15	56
Messinger et al58	15	9	7.5	5	7.5	44
Matam et al (2019)59	7.5	7.5	12.5	10	7.5	45
Kamaleswaran et al60	7.5	6	12.5	12.5	7.5	46
Rusin et al61	12.5	6	10	10	12.5	51
Zhang et al62	5	7.5	12.5	0	5	30
Biswas et al63	12.5	7.5	12.5	5	12.5	50
Si et al64	2.5	10.5	5	7.5	2.5	28
Martin et al65	15	9	7.5	15	10	56.5
Kirschen et al66	15	7.5	12.5	12.5	15	62.5*
Matam et al (2014)67	6.5	9	7.5	2.5	2.5	28
Izquierdo et al68	0	9	10	5	0	24
Zoodsma et al69	15	9	10	12.5	10	56.5
Tabassum et al70	5	10.5	7.5	7.5	7.5	38
Liu et al71	15	13.5	15	15	15	73.5*
van Twist et al† (EEG)72	12.5	12	12.5	15	12.5	64.5*
Macabiau et al73	7.5	12	12.5	10	2.5	44.5
Le et al74	7.5	12	12.5	10	2.5	44.5
Kwon et al75	12.5	7.5	10	12.5	12.5	55
Hunfeld et al76	15	9	12.5	12.5	12.5	61.5*
van Twist et al† (ECG)77	12.5	17	12.5	15	15	72*
Silva et al78	15	7.5	12.5	12.5	12.5	60*
Mean (SD)	10.5 (4.5)	9.4 (2.7)	10.9 (4.1)	10.6 (4.1)	9.2 (4.6)	50.6 (12.6)

Note there are two pairs of studies with a similar author, where additional information is provided in brackets for distinction.

^*Studies with a high quality (≥60 points).

^†Study by same author as the present study.

DATA-CARE, Data Assessment Tool for Algorithm Critical Appraisal and Robust Evidence.

Discussion

We have developed DATA-CARE, a quality assessment tool for systematic critical appraisal of data-driven algorithms in healthcare. This tool, based on the widely recognised QUIPS, addresses five quality domains of the data pipeline, including study population, data, algorithm, outcome and report transparency. Validation of DATA-CARE during a use-case on data-driven healthcare in the PICU showed the tool can be applied to a wide variety of algorithms, obtaining robust consensus in our working group with 93.5% agreement and 0.98 correlation. As such, DATA-CARE supports reproducible and transparent research through structured critical appraisal of data-driven algorithms.

To our knowledge, DATA-CARE is the first quality assessment tool suited to the diverse and fast-growing field of data-driven healthcare. While available quality assessment tools for observational studies were relevant for epidemiological aspects (eg, study population), they lacked domains that directly address the data pipeline (eg, data processing).^{10 11 13 14} Quality assessment tools addressing the data pipeline were mainly intended for AI and typically occurred as reporting checklists, spanning between 4 and 27 domains with variable numbers of items per domain.15,21 Such checklists, however, require fundamental knowledge on data science and may provoke inter-rater variation.⁴² Checklists also omit the issue that computerised algorithms lack human judgement and can therefore not identify inherent bias in the data.³⁵ For example, all checklists included data partitioning, but partitioning on a subpatient level (eg, event level) introduces data leakage as patients can occur in both train-set and test-set, hampering generalisability.⁴³ Partitioning should also be done at the beginning of the pipeline, as preprocessing techniques such as scaling on the entire data set cause similar data leakage. Hence, critical appraisal of information is just as important as ascertaining its presence. Non-checklist quality assessment tools included APPRAISE-AI and Prediction model Risk of Bias Assessment Tool (PROBAST-AI), intended for clinical decision support and predictive AI, respectively.^{22 23} APPRAISE-AI reported ICC between 0.71 and 1.00 for criteria scores, 0.89 and 0.99 for domain scores and 0.98 for overall scores.²² A similar agreement was obtained with DATA-CARE, though the agreement varied across quality domains. Development of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-AI) and Standards for Reporting of Diagnostic Accuracy Study (STARD-AI) was ongoing at the time of publication, but all were specifically intended for AI studies.^{16 24} DATA-CARE uniquely shifts critical appraisal to the data pipeline and uses key principles of transparent research reporting. As such, DATA-CARE is widely applicable and practical, without compromising on high reliability.

Progression of data-driven healthcare critically hinges on study quality, common barriers being low quality data, lack of external validation and incomplete or non-transparent reporting of findings.44,46 These barriers were also encountered during validation of DATA-CARE. None of the reviewed algorithms were implemented, DATA-CARE quality scores varied widely and the majority of studies were regarded as low-to-moderate quality. While the lack of progression and low study quality may reinforce one another, research has shown that qualitative issues persist even among algorithms approved as medical devices.⁴⁷ Predominant low scores in the data and report transparency domain, contrary to higher scores in the algorithm domain, imply that the current bottleneck of data-driven healthcare is poor quality data or studies simply neglect to reproducibly report their data pipeline. Among reviewed studies, common issues included heterogeneity in design, small and imbalanced data sets, inconsistent data processing and partitioning and lacking validation strategies with only singular metrics (eg, AUROC). While specific train–test sets may be less relevant in non-AI and/or non-prediction studies, alternatives such as stratification were rarely reported. Altogether, these inconsistencies in the data pipeline hamper reproducibility. However, they also extend as significant barriers on a regulatory level when it comes to implementation of data-driven healthcare, in particular under international bodies such as the Medical Device Regulation (MDR).⁴⁸ Despite stringent demands with regard to validation and transparency, such regulations lack guidelines on how to achieve this. International regulations such as the MDR could therefore benefit from tools like DATA-CARE to establish guidelines for standardised and reproducible algorithm development.

The strengths of the present study are that it was conducted in a transdisciplinary working group with experts from medical, engineering and research methodology fields. While clinicians are familiar with algorithm output (ie, a clinical outcome), engineers are familiar with the input (ie, data and underlying measurement principles). DATA-CARE comes with comprehensive guidance, including examples, signalling questions and a scoring system. Moreover, DATA-CARE is practical and can be applied to a wide variety of studies on data-driven studies in healthcare. Nevertheless, this study is not without limitations. Because DATA-CARE is intended to be widely applicable to data-driven healthcare, some criteria may be open to interpretation. This is especially the case for criteria in the algorithm development domain, as precise configurations of algorithms (eg, classifier type, intended objective) may vary. However, regardless of the algorithm type, it still requires an architecture with a dedicated input and output or objective, which must be developed and validated. As shown here, agreement between raters using DATA-CARE was overall high, but varied across quality domains. Furthermore, quality assessment was only performed within our own working group, and the possibility of a learning curve within the process was not considered. We still recommend users of DATA-CARE to always carry out quality assessment with two independent reviewers and reach consensus in interpretation of criteria prior to scoring. The compelling need for a widely applicable quality assessment tool for data-driven healthcare supports the present approach.

We encourage further refinement of DATA-CARE. By using the tool prospectively, criteria can be further specified and/or novel criteria can be formulated. Potentially, instead of equal points per domain, some domains may need to be prioritised and receive more points than others. While DATA-CARE is a quality assessment tool, its use highlights recurring methodological issues and reporting issues that could inform the development of future guidelines or checklists for standardised and reproducible data-driven healthcare. Although such guidelines exist, they rarely address the full data pipeline. DATA-CARE’s focus on this aspect represents its key novelty and potential contribution to improving study quality. Ultimately, this will contribute to bridging the gap between algorithm development and implementation in clinical care.

Conclusion

DATA-CARE, a quality assessment tool based on the QUIPS, allows reliable critical appraisal for a wide variety of algorithms within data-driven healthcare. The tool is widely applicable, spanning five quality domains that adhere to the data pipeline, addressing current gaps in standardised and reproducible algorithm development.

Data availability statement

Data sharing is not applicable as no data sets were generated and/or analysed for this study. All data relevant to the study are included in the article or uploaded as supplementary information.