Artificial intelligence and predictive algorithms in medicine

Artificial intelligence (AI) is poised to reshape preventive medical practice; however, its benefits to patients, specific social groups (eg, racialized populations), and businesses remain to be seen. Artificial intelligence is defined as a “machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments.”¹ Although AI does not play a major role in Canadian primary care yet, the merit of some algorithms developed elsewhere has been diminished by unintended and intended biases. For example, a US algorithm tasked to identify American patients whose current illnesses might predict future needs for increased care erroneously concluded that Black patients are healthier than equally sick White patients, effectively denying resources to Black patients.² The authors found that Black patients generated lower health care costs than White patients and that Black patients visited doctors less frequently than White patients; and yet, the AI was coded to interpret this less frequent access of care by Black patients as having a lesser disease burden.² Avoiding biases such as this requires awareness of risks and bold but thoughtful action by researchers and governments.

Problems

Artificial intelligence can account for the health effects of any available data set (eg, genetics, environment, behaviour); that is, for a patient’s context and biology, AI can “put the patient back together”³ to optimize and personalize prevention and diagnosis. It therefore has the potential to level the playing field and correct existing health inequalities.^4,5 By incorporating volumes of real-world electronic medical record (EMR) data, AI addresses the effectiveness rather than only the efficacy of clinical trials, while concurrently incorporating key social determinants of health as predictors.⁶ By using EMRs of patients whose medical outcomes are known, algorithms can predict future risk of those outcomes in others.⁷ Artificial intelligence–generated algorithms could correct historic underrepresentation or misrepresentation of women and those of particular racial or ethnic backgrounds and could account for social circumstances like education, income, and social capital. For example, an algorithm developed to identify cardiovascular disease risk might include not only risk scores, but also socioeconomic status, which is a strong predictor of heart disease.⁸ Curating, aggregating, and analyzing longitudinal information derived from EMRs hold the promise of risk prediction, prevention, and treatment by accounting for individual biology and context.

Promises

Artificial intelligence algorithms are not without medical, ethical, and social challenges, some because AI incorporates the values of data writers (eg, physicians), developers, and funders,^9-11 and others because entries in EMRs are neither standardized nor written to answer research questions. Information entered in EMRs is circumscribed by patient access to care (no clinical encounter means no recorded data about that patient) and clinicians’ values and determinations about that care. Ambiguity, lack of standard terminology, inaccuracy, and misleading acronyms are inevitable.¹² Electronic medical record data are limited by the time span of the record, and can be flawed for many reasons, including inconsistent measurement or documentation of biomarkers, absent modifiers of diagnosis such as severity, or even uninterpretable spelling. To merge known social and biological predictors of health, AI requires availability and precision of sociodemographic parameters such as race and socioeconomic status, data not routinely included in Canadian EMRs. This gap means algorithms must either ignore these data or risk imputing these classifications.¹³ Furthermore, recorded diagnoses, even those using ICD codes, are often vague (eg, does a breast cancer entry name the patient’s fear, the subject of a discussion, or a diagnosis?), as are terms such as worse and improved. Algorithms could use only biomedical measures without factoring in social determinants, but such oversight could diminish the “promise” of AI and might result in harmful blindness to the medical effects of social circumstances.¹⁴ Doctors likely consider a patient’s social determinants of health in addition to their health issues; failure to do so would, at most, harm 1 patient. Aggregation of EMR data to predict health risks aggregates blindness to sociodemographic characteristics and can undermine accurate predictions for many patients.^2,15,16

Misdiagnosis and overdiagnosis

Predictive AI models that are incomplete or unrepresentative can bias outcomes and precipitate misdiagnoses.¹⁷ Conversely, overdiagnosis may result from accepting a single abnormal test result as diagnostic, thereby attributing greater prognostic weight to some measures (eg, blood glucose level in prediabetes) than is validated by research, narrowing definitions of what is normal (eg, lowering the upper limit of a normal hemoglobin A_1c level), or defining screening outcomes in ways that favour false-positive results over false-negative ones.^9,18,19 All precipitate overdiagnosis and accompanying excessive investigation, patient fear, and unnecessary intervention. For example, if an algorithm’s default is that “1 elevated blood pressure indicates risk,” overdiagnosis will be inevitable, as more doctor visits will produce more recorded data, creating bias in what is, essentially, a time series with irregular and unequal sampling across participants.¹⁸

Artificial intelligence may, conversely, underdiagnose groups of people by excluding patients with missing data owing to limited access to care,² ignoring groups for whom data are not recorded or using biased disease definitions instead (eg, using a lower glomerular filtration rate to diagnose renal failure in Black versus White populations). Consider an algorithm to identify diabetes risk that selects as a marker of concern a blood glucose level recording that is higher than a specified level. As with current screening, those individuals who attend doctors infrequently will be tested less consistently and may be underdiagnosed. In terms of AI, if population subgroups do not attend physicians frequently and are underdiagnosed, their group characteristics (eg, age, race, sex) will be erroneously interpreted as conferring lower risk of the specific disease being tracked. This is 1 example of the more general possibility of distorting disease prediction when underlying data oversample those with specific medical traits, or are limited to either interesting or diagnosed cases rather than entire populations.²⁰

Who might have an interest in AI?

Although ethical principles of human rights, doing no harm, and protecting the individual are fundamentals of research, AI is only starting to be subjected to similar scrutiny.¹⁵ Without explicit patient consent, individuals whose medical records are electronic (almost 90% of primary care records in Canada in 2019²¹) have a digital health footprint that is being commodified, aggregated, shared, sold, and re-identified.²² Re-identification possibilities are magnified when EMR data are merged with self-quantification tools such as smart watches.²² Industry’s drive to maximize business is a powerful underlying force in science and technology.²³ Marketers’ self-interest can skew selection of input and output measures to amplify an advantageous (to industry) outcome by maximizing those identified as being at risk and by creating demand for unnecessary treatments. Statistical evidence has been misrepresented by looking at relative rather than absolute risk, again to inflate diagnoses and markets.^5,24,25 Artificial intelligence algorithms that are proprietary, as most have been to date, tend to lack transparency, necessitating clever workarounds to audit them for such disparities or distortion, and to find bias, inequality, and sources of misdiagnosis or overdiagnosis.^2,24,26 The ethos of AI and precision medicine serves industry better than it does patients when it is propelled by and fosters the belief that “more is better,” “new is better,” “prevention is better than cure,” and that diagnosing those at even minimal risk of disease is advantageous.

What is next?

Human bodies integrate social traits and life experiences with individual biology to interdependently determine health and illness.²⁵ Artificial intelligence algorithms are able to do this as well²⁴ but could exacerbate blindness to race, gender, and social identity, and obscure all behind a perceived certainty of science and numbers.⁴ An assessment is urgently needed of the empirical risks AI poses in terms of bias to inputs, processes, or the diagnoses that arise from outputs. The development of regulatory and practice frameworks to safeguard Canadians’ health is also needed. This process is under way elsewhere and only beginning among Canadian researchers, physicians, ethicists, and regulators.^27-29

Regulations and policies should address the commodification and monetization of health data; explicit patient consent; safeguarding of patient privacy; re-identification possibilities; the need to test algorithms using large, diverse, and representative data; transparent coding that enables auditing; when to inform patients that their diagnoses are AI generated; and chains of accountability for errors. Other jurisdictions are operationalizing oversight and regulations of both products (eg, algorithms) and processes (eg, data access, analytics) to ensure ethical standards and equitable resource allocation. Since 2010 in the United Kingdom, clinical decision support software requires approval from the Medicines and Healthcare Products Regulatory Agency. Developers must demonstrate that benefits outweigh risks, the effectiveness of the software, and adherence to criterion standards, and must conduct postmarket surveillance.³⁰ The US Food and Drug Administration has draft guidance for clinical decision support systems that use AI.^31,32 Health Canada is considering regulatory requirements for AI as a medical device but not yet for stand-alone AI software systems.³³ A 2019 guide on the use of AI in clinical practice by the Canadian Medical Protective Association stated that responsibility to assess quality, functionality, reliability, and privacy of AI systems rests with the physician,³⁴ raising the question of whether ensuring efficacy, accountability, and privacy while not stymieing development of beneficial health frameworks should rest with individual doctors.

Conclusion

Only when there is assurance that values, AI, and underlying training and validation data sets align with and are as expansive and inclusive as are the bodies they purport to help will their outputs foster social and racial equity and better health for all. The real question is whether regulators, data scientists, medical researchers, and clinicians will prioritize equity and ethics.