Smart microscopes of the future

Abstract

We dream of a future where light microscopes have new capabilities: language-guided image acquisition, automatic image analysis based on extensive prior training from biologist experts, and language-guided image analysis for custom analyses. Most capabilities have reached the proof-of-principle stage, but implementation would be accelerated by efforts to gather appropriate training sets and make user-friendly interfaces.

When we started writing CellProfiler software for bioimage analysis in 2003 (A.E.C.) or began contributing to ImageJ in 2001 (K.E.), we never imagined a day where we would say that most microscopy-based phenotypes are fairly easy to robustly quantify. Yet 20 years later it is largely true, thanks to the veritable buffet of software tools that solve most problems in bioimage analysis. Even more unfathomable 20 years ago is today’s reality: that most of these tools are free and open source. The bioimaging developer community is a model for others and marked by collegiality and cooperation, as evidenced by the Scientific Community Image Forum (https://forum.image.sc)¹, our one-stop shop for all image analysis questions, from beginners to experts and across software projects.

When ImageJ² had its 25th anniversary in 2012 and CellProfiler³ celebrated its tenth birthday in 2016, our teams wanted to look to the future. Discussions involved the community and ranged from standard brainstorming for the near future to whimsical dreaming further over the horizon. There were lots of ideas for new functionality and improving the interoperability and usability of interfaces. On the dreamier end of the spectrum, one participant imagined importing -omics data to integrate it with images at the single-cell level. A panoply of spatial techniques have made this capability even more in demand today. Another dreamed of software where you could deposit two piles of images into folders (perhaps samples from healthy people and from those with a particular disease) and receive a report on any significant differences in morphology found in the two groups. This is relatively technically feasible today, though user-friendly tools lag behind and safeguards are needed to avoid finding spurious associations in the samples.

Some wished for image analysis that would work fast and seamlessly with image acquisition, to allow a smart microscope that could make decisions about what, and how, to image on the fly. Several groups have proposed and demonstrated the concept^4–10. For both ImageJ and CellProfiler, much of the futuristic brainstorming focused on deep learning; it was already clear back then that eventually trained deep learning models might exist so software would ‘just know’ what different cellular structures look like, such that those structures are automatically identified in images as they are acquired, with no need to configure any software at all. One brainstormer, Allen Goodman, went even further into the future: what if you could just talk to image analysis software in plain language?

These experiences and concepts have shaped our vision of the light microscope of the future, which might have three main capabilities (Fig. 1). The first is language-guided image acquisition. Once a sample is loaded, the microscope asks: “What would you like to image?” (in any desired language, of course). At the simplest end of the spectrum, one could say “Three images from the center of each well of this 96 well plate.” If given permission, the microscope could scan an appropriate portion of the sample to discern the properties of stains that are present, to suggest, “It appears your sample contains green fluorescent protein, Hoechst 33342 and MitoTracker; should I acquire all three channels?” At the more complex end of the spectrum, one could tell the microscope, “Scan this sample for cells in the anaphase stage of mitosis, then acquire images of them every 5 minutes for the next 12 hours” or “Scan the sample at low magnification to find well-formed organoids larger than 100 micrometers in diameter, then image them at high magnification in 3D” and the microscope would perform the necessary image analysis on the fly. The microscope would interact with the user not just about what to image, but also how. Some decisions might be automatic, such as autofocusing or adjusting exposure times to be consistent across a set of samples. For others, the microscope may ask clarifying questions such as “Shall I use confocal or wide-field imaging?” and explain the pros and cons of each to the scientist, if needed. In the end, of course, the microscope would record the protocol used, so it could be published, shared and reused.

Fig. 1 |. A vision for a microscope of the future.

The microscope and scientist work together using plain language to produce desired images and report results. The microscope makes as many automated analyses and decisions as possible, though the scientist can choose to override them. The microscope also performs custom image analysis as directed.

Second, a microscope of the future could automatically analyze images, without being given explicit instructions. In fact, based on extensive training, its ability to detect and identify cell structures and phenotypes might surpass that of a typical cell biologist, whose expertise may be limited to a few cell structures and who has not observed cells under millions of different treatment conditions. For example, once images begin to be acquired, the system might report to the scientist automatically, for each image, the number of cells and the proportion of cells with particular phenotypes, including cell cycle distributions based on nuclear DNA content. It could also avoid regions of the sample that are of poor technical quality. How would this work? Already, deep learning models can automatically identify common cellular structures in images, such as nuclei or cell borders, as well as particular phenotypes, such as mitosis, apoptosis, metastasis and various differentiation states. However, until recently, such models must have already been trained on images from a particular protocol and laboratory. The idea that generalizable models – ones that work across experiments and sample types without any user training or parameter tuning – might be effective in biology was demonstrated in the 2018 Data Science Bowl¹¹, which aimed to create deep learning models to identify nuclei across a variety of samples, stains, microscopes and laboratories, all with zero user input needed. Models trained on that dataset and more, with ever-improving training schemes and networks, have yielded trained models such as Stardist¹² that ‘know’ what nuclei look like and in fact have already expanded to ones such as such as Cellpose¹³ and Mesmer¹⁴ that can identify cell borders with relative ease. In the future this capability could be expanded to other cell structures and common phenotypes. GPUs, thanks to the gaming boom in the 2000s, provide the necessary computing power to train these models, and many user-friendly open-source software tools now have deep learning segmentation capability, including CellProfiler, ilastik¹⁵, ImJoy¹⁶, DeepImageJ¹⁷, CSBDeep¹⁸ and ZeroCostDL4Mic¹⁹.

Third, language-guided image analysis would allow customized analyses, on top of whatever automatic information has been extracted as described above. This would allow queries like, “Are the GFP-positive cells nearer to the vasculature than expected by chance?” or “Show me a histogram of the number of red speckles per cell.” This now seems just visible over the horizon, with a variety of natural language processing models, datasets and training approaches coming online, mixing generalist and specific tasks^20–24. This seems especially likely after the debut of OpenAI’s ChatGPT, catching the public’s attention and imagination when it was released to great fanfare in November 2022, and the broader varieties of artificial intelligence models now becoming available (reviewed in ref. 25). As has become obvious for language bots, which may compose text that is ‘truthy-sounding’ but not true, automated methods may fail in subtle ways that require oversight by experts so acquisition and analyses may proceed as intended. Still, if these limitations are carefully explained and, to the degree possible, controlled for, such systems could vastly simplify and democratize quantitative image analysis.

What are the challenges for getting there? The envisioned capabilities have largely already been proven in principle, as described above, and the user interfaces could readily be built given sufficient funding, as could generalizable interfaces between microscopy acquisition and analysis; major technical obstacles are not likely. That leaves the major limiting factor as collecting sufficiently large and diverse training sets to be able to (1) crystallize cell biologists’ knowledge about cell structures and phenotypes into trained models that can be deployed and (2) interpret plain-language requests from scientists about image acquisition and analysis. The latter is a greater challenge currently; hundreds of custom image analysis pipelines exist in the community, but they span dozens of software languages and tools and formats, and there is no collection of them associated with a plain language explanation of their capabilities. An effort to gather these necessary missing pieces, representing the breadth of the biology community, might relieve this bottleneck. Alternatively, unsupervised learning models might learn from all existing public data without requiring supervision nor annotation by experts. Either way, advancements here could give biologists in the 2030s capabilities we can now only dream of.