Assessing the quality of reporting in artificial intelligence/machine learning research for cardiac amyloidosis

Abstract

Objectives

Despite the rapid development of AI in clinical medicine, reproducibility and methodological limitations hinder its clinical utility. In response, MINimum Information for Medical AI Reporting (MINIMAR) standards were introduced to enhance publication standards and reduce bias, but their application remains unexplored. In this review, we sought to assesses the quality of reporting in AI/ML studies of cardiac amyloidosis (CA) an increasingly important cause of heart failure.

Materials and Methods

Using PRISMA-ScR guidelines, we performed a scoping review of English-language articles published through May 2023 which applied AI/ML techniques to diagnose or predict CA. Non-CA studies and those with selective feature sets were excluded. Two researchers independently screened and extracted data. In all, 20 studies met criteria and were assessed for adherence to MINIMAR standards.

Results

The studies showed variable compliance with MINIMAR. Most reported participant age (90%) and gender (85%), but only 25% included ethnic or racial data, and none provided socioeconomic details. The majority (95%) developed diagnostic models, yet only 85% clearly described training features, and 20% addressed missing data. Model evaluation revealed gaps; 80% reported internal validation, but only 20% conducted external validation.

Discussion and Conclusion

This study, one of the first to apply MINIMAR criteria to ML research in CA, reveals significant variability and deficiencies in reporting, particularly in patient demographics, model architecture, and evaluation. These findings underscore the need for stricter adherence to standardized reporting guidelines to enhance the reliability, generalizability, and clinical applicability of ML/AI models in CA.

Introduction

Cardiac amyloidosis (CA) is a disease of heart muscle caused by the extracellular deposition of amyloid fibrils that is responsible for up to 15% of heart failure cases. The most common subtypes include transthyretin (ATTR) and immunoglobulin light chain (AL) amyloidosis, and ATTR represents the majority of cases.^1,2 ATTR CA disproportionately impacts older, male, and Black persons. In addition, 3%-4% of US Black person are carriers for TTR mutation and carrier status is associated with an average 2 years of reduced life expectancy.³ Early diagnosis of CA is critical to prevent progression and initiate disease-modifying therapies, yet clinical recognition remains delayed due to nonspecific presentations and reliance on expert-driven diagnostic pathways.^4–6 As a result, there is growing interest in applying machine learning (ML) and artificial intelligence (AI) to accelerate CA detection using data from imaging, electronic health records (EHRs), and clinical biomarkers.^7–9

Although ML methods show promise for improving diagnostic efficiency, substantial concerns persist regarding the reproducibility, generalizability, and transparency of these models.^10,11 Several studies across multiple medical specialties, including obstetrics, orthopedics, oncology, rheumatology, and radiology have shown that critical details about data handling, model development including patient demographics, and model evaluation are often omitted, limiting the ability of clinicians and researchers to appraise, reproduce, and apply the results clinically.^12–20 For example, Bozkurt et al found that only 36% of studies reported race/ethnicity while a mere 8% reported on the socioeconomic descriptors of populations used to generate ML models.¹³

To address these concerns, several reporting guidelines have been developed. Notably, the MINimum Information for Medical AI Reporting (MINIMAR) guideline was published in 2020 to establish baseline expectations for AI/ML studies in healthcare.²¹ Complementary reporting standards have been developed including CONSORT-AI (2020), CLAIM (2020), and TRIPOD-AI (2024), which provide guidelines for reporting predictive modeling, clinical trials, and medical imaging, respectively.^22–24 Despite the availability of these tools, evidence suggests that adherence to reporting standards remains inconsistent across medical specialties, including a recent study noting that 93% of ML models in oncology failed to provide details on how to access model code and 92% did not report strategies for model pretraining.^{12,17–20,25}

In this scoping review, we systematically evaluated the quality of reporting in ML studies focused on CA using the MINIMAR framework. We chose MINIMAR as it has broad applicability across data types and was one of the earliest proposed standards for AI in healthcare and was available when we began this work in 2022. Here, we aim to quantify adherence to MINIMAR criteria across critical reporting domains by ML studies focused on CA. Our findings provide a foundation for improving the rigor, reproducibility, and transparency of ML applications in this emerging field of cardiovascular medicine.

Materials and methods

We conducted a scoping review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist.²⁶ Our objective was to assess how well ML studies in CA adhere to reporting standards defined by the MINIMAR guideline, which outlines essential information for evaluating ML models in healthcare.

Eligibility criteria

We included peer-reviewed studies and preprints published in English through May 2023 that employed ML to develop diagnostic or prognostic models for CA using patient data, clinical imaging, or laboratory tests. Exclusion criteria included studies focused exclusively on non-cardiac forms of amyloidosis or those that used domain-specific knowledge to pre-select highly curated features.

Information sources and search strategy

We searched PubMed, MEDLINE, EMBASE, Elsevier, Google Scholar, and preprint servers (bioRxiv, arXiv) using a standardized query containing terms related to “cardiac amyloidosis,” “machine learning,” “deep learning,” and “artificial intelligence.” Boolean logic was applied to expand the search. Reference lists of included articles were reviewed to identify additional eligible studies.

Study selection

Two independent reviewers screened titles and abstracts using the Covidence systematic review software.²⁷ Full-text articles were obtained for those meeting initial criteria, with duplicates identified and removed. The titles and abstracts of the remaining citations were then screened to determine their eligibility based on the predefined inclusion and exclusion criteria. Any discrepancies or disagreements between the reviewers were resolved through consensus. The full texts of the articles that met the initial screening criteria were obtained and thoroughly reviewed in detail. During the full-text review, any studies that did not meet the inclusion criteria were excluded, and the reasons for their exclusion were documented.

Data charting

Data were extracted into a standardized form reflecting the domains of the MINIMAR guideline. These domains included:

1. General Study Information—authorship, country, publication year, and study setting.

2. Patient Demographics—age, sex, race/ethnicity, and socioeconomic indicators.

3. Model Architecture—type of ML algorithm used, input features, training and test split, handling of missing data, and model optimization techniques.

4. Model Evaluation—optimization, internal and external validation, performance metrics, and transparency (code and data availability).

Results

We identified 20 studies that met the inclusion criteria for this scoping review (Figure 1; Tables S1 and S5). The majority were published after the release of MINIMAR in 2020 (n = 16). These studies applied ML approaches to the diagnosis or prediction of CA using structured clinical data, imaging, or electronic health records. Using the MINIMAR framework, we evaluated adherence to 4 reporting domains: general study information, patient demographics, model architecture, and model evaluation.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 flow diagram for study selection.

General study information

Across the category of general study information (Table 1 and Table S1), all 20 studies (n = 20, 100%) stated their objectives and described the data sources used. Most (n = 17, 85%) reported the study setting, such as academic medical centers or hospital networks. However, 15% (n = 3) of studies did not describe the institutional or clinical context in which the data were collected. Patient selection criteria were clearly specified in 75% (n = 15) of studies, while 25% (n = 5) did not provide sufficient detail to determine how patients were chosen.

Table 1.

Summary statistics of MINIMAR criteria-quality of reporting (Metadata).

Key attributes	Reported	Not reported
Well-defined objectives	20 (100%)	0 (0%)
Well-defined patient selection criteria	15 (75%)	5 (25%)
Study setting	17 (85%)	3 (15%)
Data sources	20 (100%)	0 (0%)
Dataset details
Age	18 (90%)	2 (10%)
Gender	17 (85%)	3 (15%)
Race/Ethnicity	12 (60%)	8 (40%)
Socioeconomic status (SES)	0 (0%)	20 (100%)
Gold standard	18 (90%)	2 (10%)
List of all features (N/A for articles which did not used features)	5 (45%)	6 (55%)
Model optimization (N/A for articles which did not develop models)	13 (65%)	7 (35%)
Validation
Internal	17 (85%)	3 (15%)
External	6 (30%)	14 (70%)
Dealing with missing data (N/A for image/video/ECG data)	4 (44%)	5 (56%)
Transparency
Code availability	5 (25%)	15 (75%)
Data availability	11 (55%)	9 (45%)
Model type
Decision trees and variants28–35	n = 8
Neural network36–42	n = 7
Clustering35,43,44	n = 3
Logistic regression29,45	n = 2
Support vector machine46,47	n = 2
Data sources
EHR28,30,45	n = 3
ECG31,36,41,43	n = 4
Echocardiogram36,38	n = 2
Lab parameters and test results29,44	n = 2
PET image39	n = 1
CMR image37,40,42	n = 3
Claims data34,45	n = 2

Patient demographic characteristics

In terms of reporting patient demographics (Table 1 and Table S2), 90% (n = 18) of studies included information on participant age, and 85% (n = 17) reported gender. However, only 25% (n = 5) of studies reported racial or ethnic data, and none included socioeconomic indicators such as insurance status, income, or education level.

Model architecture

Model architecture was generally well described (Table 1 and Table S3). Nearly all studies (n = 19, 95%) developed diagnostic models, while only one study focused on prognosis. Most (n = 17, 85%) detailed the features used for training, and 90% (n = 18) reported how the dataset was split for model development and testing. Nevertheless, only 20% (n = 4) of studies addressed how missing data were handled, an important component for reproducibility and model robustness.

Model evaluation

Model optimization was performed for more than half of studies (n = 13, 65%); however, not all studies reported the optimization method (n = 10, 50%) (Table 1 and Table S4). Validation methodology was reported for almost all studies (n = 19, 95%). Moreover, 16 studies performed internal validation (n = 16, 80%) whereas only 4 performed external validation (n = 4, 20%); 4 studies performed both internal and external validation (n = 4, 20%). Model code was made available for 5 studies (n = 5, 25%). Data were made available for 11 studies (n = 11, 55%).

Discussion

This scoping review assessed the quality of reporting in ML-based studies of CA using the MINIMAR framework. We found occasional deficiencies in general study information (ie, study setting, patient selection) but noted significant deficiencies in the reporting of demographic characteristics (ie, race/ethnicity, socioeconomic indicators), model architecture (ie, missing data), and model evaluation (ie, transparency). These limitations mirror similar findings from other medical fields, including radiology, oncology, and rheumatology, where studies have shown incomplete reporting across multiple domains.^{12,17–20,25} This study is the first to systematically evaluate ML reporting in CA and, to our knowledge, the first to apply the MINIMAR framework to assess ML literature within a specific clinical domain.

Demographic reporting was among the weakest areas. Only 1 in 4 studies included racial or ethnic information, and none reported socioeconomic data. These omissions are particularly concerning given the established racial disparities in the incidence of transthyretin amyloidosis⁴⁸ and the socioeconomic barriers to diagnosis and care. Black Americans develop CA at twice the rate of CA compared to the White population.⁴⁸ Furthermore, there is under detection in certain US regions such as the South despite over 50% of the Black population residing there, thus highlighting need for improved equity of care.⁴⁸ Incomplete reporting in this domain limits the generalizability of ML models and perpetuates structural bias.^49–51 The inclusion of social determinants of health has been shown to improve models predicting hospital readmission for sepsis.⁵² Our findings are also consistent with prior analyses of EHR-based ML studies, which reported similarly low rates of demographic disclosure¹³ and highlight the need to address potential sources of bias early in model development.⁵³

Model architecture was generally well reported with the exception of handling of missing data. Most studies did not report how missingness was handled or whether imputation strategies were employed, potentially biasing model performance and reducing replicability.⁵¹ This omission is particularly concerning given the frequency of missing data in real-world clinical datasets, where improper handling can introduce systematic bias and compromise generalizability, especially in CA, where models are often developed on small, heterogeneous cohorts.

We found major deficiencies in the reporting of model evaluation parameters, particularly with transparency and reproducibility. Despite growing calls for enhancing reproducibility in AI/ML science, only 25% of the reviewed studies shared their model code, and 55% provided information on data availability. Without access to code, data, and a clear description of modeling pipelines, independent replication is impossible, resulting in decreased trustworthiness and clinical adoption.^54–57 This aligns with other reports that found incomplete reporting related of transparency measures in AI studies.^11,17,55,58

Model evaluation reporting was also often deficient with regard to validation practices. Internal validation was commonly reported, but only 20% of studies reported external validation using independent datasets, a vital step in demonstrating model generalizability. This rate is slightly higher than that reported in imaging-based AI literature but remains far below what is needed for clinical deployment.^55,58,59 Given the rarity and clinical heterogeneity of CA, external validation using multi-institutional or prospective datasets is particularly important.

Our review has several strengths and additional considerations. We applied a standardized evaluation framework (MINIMAR) to assess the quality of studies and included all available studies of ML in CA, including both peer-reviewed and preprint literature. Second, most of the studies that were evaluated were reported after MINIMAR was published in 2020. Studies published after MINIMAR release trended toward improved reporting, particularly with regard to model evaluation. Our analysis is limited by the small number of eligible studies, precluding formal statistical comparisons particularly in comparing studies completed before and after the release of MINIMAR. It should be acknowledged that other reporting standards have recently been created or updated to incorporate ML studies, and MINIMAR is not a universally accepted reporting standard. However, the goal of this work is to evaluate the quality of ML reporting to inform this rapidly evolving specialty in cardiovascular medicine. In addition, given the explosive growth of ML studies in medicine, our study does not address studies published since 2023. Fourth, other reporting standards tools such as TRIPOD-AI and CLAIM might offer complementary insights not captured in our evaluation. For example, TRIPOD-AI, which was released in 2024, is intended to be a comprehensive guide to reporting multivariable predictive models.²² It shares the key reporting domains espoused in MINIMAR (with emphasis on fairness, openness and trustworthiness) and provides additional guidance and on how to report each aspect of model development. Nonetheless, because of the similarity of MINIMAR and TRIPOD-AI, and the availability of MINIMAR before TRIPOD-AI, our choice of MINIMAR was appropriate to provide a snapshot of the ML studies in this field. We believe there is concordance between MINIMAR and CLAIM standards, and furthermore our review was not exclusively focused on medical imaging.

Future directions

Given the recent introduction of several new reporting standards for AI studies in healthcare, it is possible that the quality of reporting of AI classification and prediction models may improve over time. However, current research suggests that their adoption remains low^17,60 and is predicated on self-reporting, which has been shown to be prone to error.^61,62 Widespread utilization of standardized reporting practices will require buy-in from researchers and enforcement from publishers and funding agencies⁶³ to ensure systematic change. As new AI “solutions” come to market to improve diagnosis and treatment of CA, it will be critical to carefully examine the data and the population those data represent. Better reporting of ML studies will accelerate the safe, equitable, and effective integration of these technologies and lead to improvements in the health of patients with CA.

Conclusion

This study is one of the first to systematically assess the quality of reporting of AI/ML research within the CA field, a rapidly developing therapeutic area in medicine. By employing the MINIMAR reporting standards as a benchmark, we have adopted a rigorous and broadly applicable framework to evaluate the reporting quality of studies in this field. This systematic approach has enabled us to identify and document key areas where reporting deficiencies are prevalent, such as in demographic details, model architecture, and evaluation processes. Addressing these deficiencies is crucial for enhancing the transparency and reproducibility of future research in CA.

Author contributions

Asiful Arefeen (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing—original draft, Writing—review & editing), Simar Jeet Singh (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing—original draft, Writing—review & editing), Crystal Razavi (Data curation, Formal analysis, Investigation, Writing—original draft), Hassan Ghasemzadeh (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing), and Sandesh Dev (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Writing—original draft, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This material is the result of work supported with resources and the use of facilities at the Southern Arizona Veterans Affairs Health Care System Tucson, AZ and Arizona State University, Complex Adaptive Systems Initiative. The contents do not represent the views of the U.S. Department of Veterans Affairs or the United States Government.

Conflicts of interest

The authors have no competing interests or disclosures to declare.

Data availability

The data underlying this article are available in the primary publications identified in the systematic search. No new datasets were generated or analyzed in this study.