Fluid Intelligence: AI’s Role in Accurate Measurement of Ascites

See also the article by Hou et al in this issue.

Alex M. Aisen, MD, is a retired academic radiologist who specialized in gastrointestinal and body imaging. He began his career at the University of Michigan and moved in mid-career to Indiana University where he is currently a professor emeritus of radiology and imaging sciences. Following his retirement from clinical practice, Dr Aisen began working in the commercial sector. He is presently employed as a clinical scientist at Philips Healthcare.

Pedro S. Rodrigues, PhD, has a doctorate in physics engineering from Instituto Superior Técnico, Lisbon, Portugal. He is a clinical scientist at Philips Healthcare.

In this issue of Radiology: Artificial Intelligence, Hou et al describe another tool in the ever-expanding portfolio of artificial intelligence (AI) algorithms to augment radiologic image interpretation (1). The application both detects and quantifies free peritoneal fluid. In clinical practice, a radiologist can usually quickly detect the presence of ascitic fluid, but quantifying the amount of fluid would be impractical for a human reader; thus, this tool provides a new function and information impractical to obtain manually during routine interpretation. It will be interesting to learn after the algorithm is deployed clinically how useful this information proves to be for patient management.

The method described to develop the algorithm is sound, although, as the article notes, there are a few significant limitations. The basic deep learning model is nnU-Net framework, a commonly used and well-regarded tool (2). Four case cohorts were employed: a public database of abdominal-pelvic CT scans of women with ovarian cancer—composed of both enhanced and unenhanced studies, used for model training and development—and three additional cohorts used for performance testing. Importantly, one of the latter databases is external, which is good practice for assessment of the accuracy of an AI algorithm. The initial segmentation and labeling of both the training and test data were done by a postdoctoral fellow, with each set of labeled data then reviewed by one board-certified radiologist. This segmentation procedure provided the reference standard both to train the model and to evaluate the accuracy of the completed algorithm.

Because the process of labeling can be time-consuming, an iterative approach termed active learning was used both to train the algorithm and to define the reference standard for performance testing. A small number of images from each cohort was initially segmented manually; this first group was then used to train a preliminary AI algorithm. The preliminary AI model was then used to segment a larger number of patients. Rather than manually segmenting from scratch, a radiologist adjusted the output segmentations, a much faster process, and repeated the process iteratively. Such iterative labeling is frequently employed in algorithm development.

The scan cohort used for training is composed of only female patients seen at several U.S. hospitals with a single diagnosis (ovarian cancer), which may limit generalizability. Additionally, there are few noncontrast scans in the test cohorts. The use of only a single experienced radiologist to review final segmentations—one radiologist for the training set and another for the test set—is also a limitation.

There are multiple clinical reasons to evaluate a CT study for ascites and to gauge the quantity of fluid. Large-volume ascites is often present in cirrhosis or cancers with peritoneal involvement; other causes include pancreatitis and heart failure. Detecting ascites is useful for diagnosis, and quantification may be useful for patient management (3). In CT scans performed in the emergency department for blunt abdominal trauma, the presence or absence of peritoneal fluid can be used to assess the severity of abdominal injury. A sensitive algorithm may be useful for triaging (4); however, the present algorithm was not developed or tested for this use case.

The detection of free peritoneal fluid is not straightforward, as the fluid’s configuration is highly variable. The fluid’s attenuation may take on a range of values, either from blood in the case of hemorrhagic ascites, or from intravenous contrast material, which in some circumstances can diffuse from the bloodstream; these conditions were not evaluated in this study. Nonetheless, AI algorithms based on deep learning are proving to be robust. Most commonly, deep learning algorithms are developed by first choosing a mathematical model and training it by allowing it to process many images that have been labeled as to the medical condition in question. The current tool was trained with scans from patients with ovarian cancer, with the ascitic fluid delineation accurately and meticulously confirmed by an expert. The algorithm “learns” the features of the labeled finding and can then recognize similar findings when new, unlabeled cases are presented. The image characteristics that define the algorithmic learning are often obscure; algorithms are often said to be “black boxes” (5). Nonetheless, algorithm accuracy is testable, and good algorithms often perform as well as or better than radiologists at the specific tasks and in the patient populations for which they are trained.

The algorithm under consideration was trained to both detect and quantify peritoneal fluid. A threshold of 50 mL was used to enable a binary classification of whether significant fluid was present, that is, classification of the scans in the cohorts as positive or negative for ascites. As the authors note, this threshold is arbitrary. A 50-mL cutoff may be too high for some clinical purposes, including triaging. This threshold may be reasonable for use cases such as cirrhosis or tumor evaluation. If a revised algorithm is developed for triaging blunt trauma patients, further consideration of the selection of a threshold will be necessary.

In this study, the model’s accuracy both for detection of ascites and for measurement of the volume was good; the volume estimation error (the discrepancy between the volume estimated by the algorithm and the true volume, expressed as a percentage of the true volume) was 5%–20% on the three test cohorts. However, in one of the test datasets the recall (sensitivity) for detection of ascites was only 75%, suggesting a meaningful false-negative rate. This result and the procedure used to evaluate the model in patients with low fluid volumes suggest that the algorithm is not (yet) suitable to triage emergency department trauma cases; this use case is not discussed in the manuscript. For most other use cases, the accuracy for fluid volume is not perfect, but probably reasonable and sufficient for most clinical applications.

The use of deep learning to evaluate ascites is relatively recent, but algorithms have become available and validated for other similar functions. Uses include the identification and volumetric measurement of individual organs, and the segmentation and measurement of specific tissues, such as muscle, visceral fat, and subcutaneous fat (2). Recently developed algorithms can reliably differentiate between subcutaneous and visceral fat. These indicators of body composition are important in the diagnosis and tracking over time of muscle loss (sarcopenia) and metabolic syndrome.

An algorithm for quantification of ascites is novel; even experienced radiologists do not routinely quantify fluid, as the process would be time-consuming. In view of the lack of clinical experience with practical methods for quantitative measurement of ascitic volume, we can only speculate as to the clinical utility of the tool. Volume measurement conveniently incorporated into workflow, such as an algorithm that runs in the background and provides a value to the radiologists as soon as a case is opened for interpretation, may be medically useful in assessing disease status in patients with malignant peritoneal carcinomatosis (3). The information may also help indicate when a patient with late stage cirrhosis should be treated, perhaps with palliative therapeutic paracentesis (6).

For a tool to be available and useful clinically, it must achieve regulatory approval in the country in which it will be deployed, and it must be conveniently incorporated into radiologist workflow. Usually, this entails development by a commercial entity. Achieving regulatory clearance of AI algorithms in the United States involves submission of a lengthy application to the U.S. Food and Drug Administration (7); there are similar processes for most other countries, including the Conformité Européenne mark process under the recently revised Medical Device Regulation in Europe. These regulatory processes are becoming more rigorous and expensive; agencies appropriately require performance testing to document accuracy and detailed analysis and mitigation of potential risks. It remains to be seen what accuracy regulators would require, and this will likely depend on the use cases for which a commercial version of the tools is marketed and labeled.

The tool must also be seamlessly incorporated into the routine interpretation workflow, probably with a result calculated in advance and available to the radiologist as soon as the case is opened for interpretation. Relevant technical standards are still being developed; many companies are implementing so-called AI marketplaces that manage multiple AI algorithms and efficiently and conveniently present their results. Standards and profiles, such as the Integrating the Health Enterprise (IHE) AI Results and AI Workflow profiles may also be enabling (8). Presently, most AI algorithms are not directly reimbursed through clinical billing, though it is possible that may change going forward. For an application to be deployed clinically, it must either clearly benefit patient management or improve workflow efficiency (9). These requirements apply to the tool just described: For it to be accepted, it must clearly enhance patient care.

We also express our appreciation that the authors are making their curated annotated training data publicly available. It is often stated that only a minority of the effort expended in algorithm development is software construction, and the substantial majority is obtaining appropriate training and test data. By making annotated data publicly available, the authors are providing significant public benefit (10).