Table 1.

Checklist of points to consider when assessing a commercial AI solution in radiology

1. Relevance	1.1. What problem is the application intended to solve, and who is the application designed for? Define the scope of application; end-users; research vs. clinical use; usage as double reader, triage, other; outputs (diagnosis, prognosis, quantitative data, other), indications and contra-indications
	1.2 .What are the potential benefits, and for whom? Consider benefits for patients, radiologists/referring clinicians, institution, society
	1.3. What are the risks associated with the use of the AI system? Consider risks of misdiagnosis (including legal costs), of negative impact on workflow, of negative impact on quality of training
2. Performance and validation	2.1. Are the algorithm’s design specifications clear? Check robustness to variability of acquisition parameters; identify features (radiomics) or network architecture (deep learning) used
	2.2. How was the algorithm trained? Assess population characteristics and acquisition techniques used, labeling process, confounding factors, and operating point selection
	2.3. How has performance been evaluated? Check proper partitioning of training/validation/testing data, representativeness and open availability of data. Assess human benchmarks, application scope during evaluation, source of clinical validation
	2.4. Have the developers identified and accounted for potential sources of bias in their algorithm? Assess training data collection, bias evaluation, stratification analyses
	2.5. Is the algorithm fixed or adapting as new data comes in? Check whether user feedback is incorporated, if regulatory approval is maintained, and if results are comparable with previous versions. *
3. Usability and integration	3.1. How can the application be integrated into your clinical workflow? Consider integration with your information technology (IT) platform, check for compliance with ISO usability standards, consider issues related to practical management of the software
	3.2. How exactly does the application impact the workflow? Identify modifications to bring to your current workflow, identify roles in the new workflow (physicians and non-physicians)
	3.3. What are the requirements in terms of information technology (IT) infrastructure? Consider on-premise vs. cloud solutions. Identify requirements in terms of hardware and network performance, consider network security issues
	3.4. Interoperability - How can the data be exported for research and other purposes? Check whether the export formats are suitable
	3.5. Will the data be accessible to non-radiologists (referring physicians, patients)? Check whether the form of the output is suitable for communication with patients/referring physicians
	3.6. Are the AI model’s results interpretable? Check whether and which interpretability tools (i.e. visualization) are used
4. Regulatory and legal aspects	4.1. Does the AI application comply with the local medical device regulations? Check whether the manufacturer obtained regulatory approval from the country where the application will be used (CE, FDA, UKCA, MDSAP, or other local guidance), and for which risk class
	4.2. Does the AI application comply with the data protection regulations? Check whether the manufacturer complies with local data protection regulations and provides contractual clauses protecting patient’s data
5. Financial and support services considerations	5.1. What is the licensing model? Assess one-time fee vs. subscription models, total costs, scalability
	5.2. How are user training and follow-up handled? Check whether training sessions are included and at which conditions further training can be obtained
	5.3. How is the maintenance of the product ensured? Check whether regular maintenance is included, assess the procedure during downtime and for repair
	5.4. How will potential malfunctions or erroneous results be handled? Assess the procedure in the event of malfunction and post market surveillance and follow-up

* Note that at the time of writing of these guidelines, no adaptative AI application exists on the market.