Summary
In this Q&A, Scientific Editor Emily Marcinkevicius talks with author Peter Horvath about work from his group entitled “Fisheye transformation enhances deep-learning-based single-cell phenotyping by including cellular microenvironment,” his cross-disciplinary background, and the future of artificial intelligence methods in the biomedical sciences.
Main text
Peter Horvath, PhD director, Institute of Biochemistry, Biological Research Centre, Szeged, Hungary; principal investigator, Institute for Molecular Medicine Finland, FIMM-EMBL, University of Helsinki, Finland
Artificial intelligence (AI) has revolutionized the fields of biomedical imaging and pathology. Inspired by discussions with trained pathologists about how they practically assess and diagnose samples, Toth et al. developed an image classifier that increases the accuracy of cell phenotyping by utilizing a fisheye-like transformation that incorporates information about a given cell’s neighbors and microenvironment. The paper is being published as part of a joint special collection of papers across Cell Reports portfolio journals focused on AI.
First, please tell us a little bit about yourself. Where are you from, where did you train, and how did your scientific interests develop?
Peter Horvath: I am a computer scientist by training and received my BSc, MSc, and PhD degrees in computer vision and image analysis. I did my PhD in France (INRIA-Sophia Antipolis) on satellite image analysis; we developed methods that digitalize Earth maps automatically, detecting roads, trees, and houses. I would have never thought that I would work on the fields of biology and healthcare until it turned out that the questions in microscopy image analysis are very similar. We essentially hunt for circles (i.e., cell nuclei) and lines (i.e., vesicles). So, I went to ETH Zurich (ETHZ), and we analyzed the first dozen image-based siRNA whole-genome screens. After ETHZ, I opened a lab in Helsinki FIMM-EMBL and in Hungary (Biological Research Centre Szeged) and recently became the director of the Biochemistry Institute. Generally speaking, we work on single-cell analysis.
An earlier working title of your paper was “Show me your neighbor and I tell what you are: Fisheye transformation for deep-learning (DL)-based single-cell phenotyping.” For a reader from any field of biology, what does this work allow you to do?
P.H.: Probably the title speaks for itself. I was very curious about the influence of the microenvironment on a single entity. More precisely, can we more accurately analyze the phenotype of a single cell if we know its neighbors? One can think of an analogy with a real-life neighborhood and the prediction of some individual properties. This method allowed us to increase the accuracy of phenotypic profiling by 10%.
What initially motivated the project and how did it develop?
P.H.: The motivation came from discussions and long-lasting nights spent in the pathology institute with the pathologist co-author Farkas Sukosd. We observed that the practitioner looks at the tissue sample under the microscope, often changing the magnification from high to low to high, sometimes taking it away and looking by the naked eye. So, we can conclude that in making diagnostics decisions, the pathologist strongly considers the cellular microenvironment. We first published a method where we generated a graph from the tissue so that single cells were the points, and edges came from neighbors based on distance. Each cell was analyzed so that its neighbors were also taken into account (https://www.nature.com/articles/s41598-018-28482-y). In our current work, we created a transformation and incorporated it into DL-based image classifiers. This transformation is similar to the classical fisheye-transform and provides several advantages, such as the machine seeing the current cell of interest in full resolution and the entire cellular microenvironment being incorporated at a decreasing resolution as a function of spatial distance. This way we have more detailed information from closer neighbors but still have some from larger proximity.
I’m wondering how you generally select your AI/machine learning (ML)-themed projects. Is it driven primarily by a biological question or need, or by a type of ML approach that you are interested in applying?
P.H.: Definitely the former one. My lab has a global interest of understanding why and how single cells are organized and behave in order to provide clinical advancement for cancer and other disease treatments. We have numerous challenges while working in depth on these fields, and often the solutions include AI and ML, where my group moves very comfortably, as the majority of my colleagues are trained AI engineers.
What are some of the fields where you think AI can make the strongest contribution? Or put another way, what fields are on the cusp of having a breakthrough driven by AI?
P.H.: My forecast for the coming 5 years is that AI will not bring many breakthroughs but rather more incremental improvements. On the other hand, I would not underestimate these improvements. In my opinion, we will see a fantastic revolution in digital pathology—similar to what has happened in medical imaging (CT/MRI) about a decade ago—and this will rewrite modern cancer treatment.
Reflecting on the first half of the question, with no question DL is a game-changer. Therefore, the strongest contribution is expected on the fields where DCNNs are appropriate, meaning that the data are spatially dependent. Therefore, digital images and time-resolved questions are my best bets.
What were some of the most interesting papers or developments in the AI space that caught your attention in 2022, and what are you expecting or hoping to see more of in 2023?
P.H.: I would definitely watch how single-cell segmentation is developing. A nice read is the CellPose2 (https://www.nature.com/articles/s41592-022-01663-4). Another field is the navigation in gigantic image-based datasets, such as whole-slide images. I would recommend this paper: https://www.nature.com/articles/s41551-022-00929-8.
In 2023, I would hope to see more intelligent annotation methods and identification of the most efficient combinations of annotated and non-annotated data.
Do you have any advice for researchers working with complex datasets who might be looking to see whether ML-approaches can help accelerate their research but who do not have expertise in this area? How should they assess whether a given ML model or software package can help them?
P.H.: Recently, large imaging communities put serious effort into enabling software tools that utilize ML/DL, such as ImageJ/FiJI or Napari. Some of them also received funding for more proficient developments (CZI and Napari). For non-expert users, I would recommend looking at these software and contact the community behind them if there are questions. Assessing the quality of the analysis is very difficult; there is also a big debate in the community about what evaluation methods to use. There are large studies coming to address this in early 2023.
Does your group have a “favorite” ML architecture to work with or compare new approaches to?
P.H.: I would say that we have standard ones that we know and use, so all pros and cons are benchmarked. These include MaksRCNNs, UNETs, YOLOs, Resnets, NuleAIzer, and derivatives of StarDist. The majority of the problems we face can be solved with these. We pay attention to being up-to-date and benchmark any promising new approaches.
What are some of the major directions your group’s research will be moving in over the coming years?
P.H.: (1) 3D DL-based segmentation and phenotyping, (2) image similarity search applied in large pathology datasets, (3) correlative microscopy, and (4) metrics.
Do you have any advice for junior trainees who are interested in getting involved with AI/ML methods development?
P.H.: (1) Learn Python (optionally Matlab, too); (2) gain practical knowledge and familiarity with at least a few basic architectures in terms of behavior, convergence, problem-solving capabilities, and parameters; and very importantly, (3) learn and understand the theoretical foundations of ML. Having experts who understand why DL works and does the job is important. We need people who not only can run and play with parameters but also understand and build new strategies and architectures that are tuned to the question of interest. This will change the game!