Coates and de Koning 1 warn us wisely. An appropriate, well-understood, scientific evidence base is always needed before attempting automation. Racial, gender and socioeconomic bias may creep in, even in critical care training data. Informatic oversight via comprehensive clinical reviews and meta-analyses is essential.
Without a ‘human in the loop’, medicine is not healthcare. Machine learning (ML) is just a complicated statistico-mathematical modelling, and as Box 2 says ‘All models are wrong but some are useful’. ML or artificial intelligence (AI) still requires the validating and qualifying steps described 3 as the TRL5-8 stages of development. Not just clinical validity in critical care, but utility, contraindications, cost effectivity, impact on health budgets, operationalisation and continuing quality audit and reviews need examining. Moreover, the right data to probe this critically may not currently be in the digital electronic health records; therefore, specially designed trials may be needed.
Relying solely upon ‘Computer says … yes/no’ is not an acceptable way to deal with human beings, even under the urgent demands of rapid responses. Blind acceptance of AI in clinical decision-making runs the risk of promulgating Automated Idiocy. Informed informatic and trusted clinical judgement must be embedded to challenge, put into context or indeed correct automated decisions.
Patients are not widgets and ML/AI is not exceptional.
Declarations
Competing Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: I am a Royal Society Industrial Fellow with the Oxford Centre for Industrial Applied Mathematics and Life Fellow of the Royal Society of Medicine. These are personal opinions.
References
- 1.Coates JT, de Koning C. Machine learning-driven critical care decision making. J R Soc Med 2022; 115: 236–238. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Box GEP. All models are wrong. See https://en.wikipedia.org/wiki/All_models_are_wrong (last checked 10 August 2022).
- 3.European Commission. Technology Readiness Levels (TRL). Horizon 2020 – Work Programme 2014–2015. See https://ec.europa.eu/research/participants/data/ref/h2020/wp/2014_2015/annexes/h2020-wp1415-annex-g-trl_en.pdf (last checked 10 August 2022).