Imaging of cellular dynamics from a whole organism to sub-cellular scale with self-driving, multi-scale microscopy

Abstract

Most biological processes, from development to pathogenesis, span multiple time and length scales. While light-sheet fluorescence microscopy (LSFM) has become a fast and efficient method for imaging organisms, cells and sub-cellular dynamics, simultaneous observations across all these scales have remained challenging. Moreover, continuous high-resolution imaging inside living organisms has mostly been limited to few hours as regions of interest quickly move out of view due to sample movement and growth. Here, we present a self-driving, multi-resolution light-sheet microscope platform controlled by a custom Python-based software, to simultaneously observe and quantify sub-cellular dynamics in the context of entire organisms in vitro and in vivo over hours of imaging. We apply the platform to the study of developmental processes, cancer invasion and metastasis, and we provide quantitative multi-scale analysis of immune-cancer cell interactions in zebrafish xenografts.

Introduction

Over the past two decades, light-sheet microscopy has emerged as a powerful approach for fast and efficient fluorescence imaging^1–3. At the sub-cellular level, high-resolution light-sheet instruments such as lattice light-sheet microscopes⁴, field synthesis⁵, dual-view inverted selective plane illumination microscopes⁶ or axially swept light-sheet microscopes (ASLM)^7,8 enable studies of spatiotemporal organization and dynamics of cell signaling, cell morphology, and local cell-cell interactions. At the organismal scale, low-resolution light-sheet imaging⁹ provides capabilities to rapidly image several entire organisms over days^10,11 and to study processes such as cell dissemination, migration patterns or to find rare cellular events such as immune-cancer cell interactions. Progress has been made towards combining low- and high-resolution imaging for cleared tissue imaging¹². However, to our knowledge, no light-sheet imaging platform has been described to dynamically image living organisms in their entirety over several hours while simultaneously imaging at sub-cellular resolution to study biological processes across scales.

The lack of technologies for imaging a whole, living organism at sub-cellular resolution is due to technical and practical limitations¹³. There is a trade-off in optical design between numerical aperture (NA) and working distance as well as field of view¹⁴, i.e., the higher the imaging resolution, the smaller the region to image (Fig. 1a). From a data collection perspective, there are practical limitations as imaging an entire organism such as a zebrafish larva with sub-cellular resolution would require several hours, preventing any dynamic time-lapse imaging. Moreover, proper Nyquist sampling of the 3D specimen would result in enormous data sizes, e.g., over half a Terabyte per timepoint and channel (Fig. 1b). To overcome these trade-offs, novel, adaptive imaging schemes are required that perform high-resolution imaging only in selected regions of interest and timepoints.

Fig. 1: Self-driving, multi-scale microscopy.

a Schematic of the field of views of low- (brown) and high- (violet) resolution light-sheet microscopy modalities to image biological processes across scales. b Calculations of the duration and resulting data sizes for imaging a 2.5-day old zebrafish larva in one fluorescent channel with high-resolution imaging using axially swept light-sheet microscopy (ASLM), low-resolution imaging and a multi-scale imaging approach that captures the whole organism and one selected region of interest with high-resolution. c (Left panels) Schematic top-view light-sheet profiles of dual-sided multi-directional selective plane illumination microscopy (mSPIM) (top, brown) for low-resolution imaging, and single-sided ASLM for high-resolution imaging (bottom, violet), with the dark blue arrow indicating the scan direction of the thin ASLM light-sheet. (Middle panels) Representative images of 0.2 μm fluorescent nanospheres for both modalities. (Right panels) PSF values measured with MetroloJ (n=20 for each modality). d Simplified rendering of the multi-scale microscope. The low- and high-resolution modalities share the same illumination and detection objectives. In the detection path, a motorized flip mirror selects either the low-resolution or the high-resolution detection path. e Representative images from the multi-scale microscope displaying a 2.5 days old zebrafish larva (cyan: vascular marker Tg(kdrl:Hsa.HRAS-mCherry), magenta: macrophage marker Tg(mpeg1:EGFP)) xenografted with U-2 OS osteosarcoma cells (white). Left: Low-resolution image of the whole zebrafish larva. Right: Zoom in to boxed image to compare the high-resolution image acquired with ASLM (bottom left) to the mSPIM image (top right). f Representative low-resolution stills from a time-lapse imaging experiment of zebrafish larva expressing the vascular marker Tg(kdrl:Hsa.HRAS-mCherry) are overlaid onto each other, with the color indicating imaging time (starting at 3 days post fertilization). This highlights sample movement and growth over the observation window and the shift of a region of interest over time (boxed region). g High-resolution imaging with a static, pre-defined imaging volume within the microscope coordinate system (top) and self-driving microscopy (bottom). Same color scheme as in e. Scale-bar lengths are as follows: c 0.5 μm; e 250 μm (left), 60 μm (right); f 250 μm; g 50 μm.

Leveraging adaptive imaging schemes with different imaging modalities, Almada et al.¹⁵ performed unsupervised, event-driven transition from live-to-fixed cell imaging. In another approach, Alvelid et al.¹⁶ performed automated multiscale imaging using STED super-resolution imaging guided by widefield microscopy. Similarly, smart lattice microscopy has been described for imaging rare cellular processes in a culture dish using epifluorescent imaging for detection¹⁷. While these approaches integrate information from different modalities, they do not allow for multi-resolution imaging of entire, complex 3D samples such as spheroids and organisms, away from a culture dish, with simultaneous observations of sub-cellular processes of interest in situ.

To overcome these limitations, we describe a self-driving, multi-resolution light-sheet microscope platform that enables observation and quantification of biological processes across scales in vivo over many hours. To this end, we have integrated the strengths of low-resolution light-sheet imaging¹⁰ with sub-cellular ASLM imaging^7,8 into one microscopy platform. Guided by custom hardware control software, the microscope is self-driving, i.e., it adjusts the region for high-resolution imaging on-the-fly as the specimen develops, grows, or changes. We demonstrate the power of this self-driving, multi-scale imaging platform for diverse applications in vitro and in vivo, including imaging of cancer spheroids, developmental processes, and cancer cell metastasis in a zebrafish xenograft model. In depth, we detect, visualize, and quantify immune–cancer cell interactions that lead to successful phagocytosis or immune evasion of human cancer cells in a micro-metastasis niche.

Results

Multi-sample, multi-resolution system design.

We modularly combined dual sided multi-directional selective plane illumination (mSPIM)¹⁸ for low-resolution imaging and single sided axially swept light-sheet microscopy (ASLM)^7,8 for high-resolution imaging (Fig. 1c, Extended Fig. 1, Supplementary Note 1). mSPIM excels at whole organism imaging to reveal cellular and tissue level processes such as cell spreading and migration patterns, and detect rare cellular processes in large volumes. Complementarily, ASLM provides the necessary resolution enhancement to reveal sub-cellular signaling, cell-cell interactions and local changes of cell morphologies (Fig. 1a, c). Both modalities were combined into the same system using a shared illumination path and the same detection objective (Fig. 1d). A description of all hardware components is available as Supplementary Table 1.

Specifically, for illumination, we applied dual-sided mSPIM to illuminate instantaneously a large field of view for whole organism imaging. For sub-cellular imaging, the ASLM modality employed a thin light-sheet (Extended Fig. 1), which was scanned in its propagation direction synchronized with the rolling shutter on a camera. The scanning of the light-sheet allowed us to extend the otherwise beam-waist limited field of view of a thin light-sheet. To change the light-sheet properties for the two modes, we employed a motorized slit conjugate to the pupil of one of the illumination objectives. Changing the slit opening adjusted the effective numerical aperture of the illumination objective, up to an NA of 0.367 for high-resolution imaging (Extended Fig. 1a, Supplementary Note1, Supplementary Fig. 1–2). To improve the switching time between low- and high-resolution, we also designed an illumination layout that dispenses with the motorized slit and only requires flip mirrors to change the imaging modality (Extended Fig. 1b, Supplementary Note 1, Supplementary Fig. 3–4). This second configuration leverages two paths for light-sheet generation that are combined by a polarizing beamsplitter cube onto the same light path for illumination of the sample. In both layouts, we employed a temperature control system of the sample chamber for environmental control (Supplementary Note 1, Supplementary Fig. 5).

For detection, we determined that using a medium magnification objective (20X Olympus XLUMPLFLN) with a high numerical aperture (NA) of 1.0, a large working distance (2 mm) and a large field of view allowed us to use it for both modalities. A motorized flip mirror after the detection objective enabled us to image at different magnifications by selecting a different tube lens and camera pair (Fig. 1d). In the low-resolution path, a 100 mm tube lens yielded 11.1x magnification, while a 500 mm tube lens enabled 55.56x magnification in the high-resolution detection path. For low-resolution imaging, we used a camera (Iris 15, Photometrics) with a large sensor size of 5056 x 2960 pixels. This provided a 1.9x1.1 mm² large field of view with a deliberate slight under-sampling (382nm demagnified pixel size) for organism scale imaging. Due to the field number of the objective, the effectively usable field of view was 1.3x1.0 mm². In the high-resolution detection path, we used a sensitive camera (Prime BSI Express, Photometrics) with 2048 x 2048 pixels, enabling Nyquist sampled (117nm demagnified pixel size) high resolution imaging using ASLM with the Photometrics programmable scan mode (Supplementary Fig. 6). Notably, we did not require auto-focusing routines during imaging. The focus was maintained over the time window of observation once aligned at the start of the experiment.

To characterize the optical performance of our multi-scale microscope, we determined the point spread function values of 0.2 μm YG Nanospheres embedded in agarose (Fig. 1c, Supplementary Note 2, Supplementary Fig. 7). The ASLM resolution was measured laterally as: 0.39 ± 0.02 μm (x) and 0.39 ± 0.01 μm (y); and axially as: 0.65±0.04 μm (n=20). For the low-resolution mode, due to a combination of the thicker light-sheet and lower lateral sampling, we measured a lateral resolution of 1.01 ± 0.14 μm (x), 0.92 ± 0.13 μm (y), and an axial resolution of 2.42 ± 0.49 μm (n=20; Fig 1c). These values considerably improve over a recently published multi-scale cleared tissue microscope with 2.91 ± 0.31μm axial PSF in the high-resolution mode and a 5.48 ± 1.08 μm axial PSF in the low-resolution mode¹². Importantly, the resolution remained uniform within the usable field of view (Supplementary Fig. 8). In principle, high-resolution imaging without ASLM was possible with a motorized slit, however, this led to a reduced axial resolution (Supplementary Note 1, Supplementary Fig. 9).

To characterize whether repeated switching between low- and high-resolution imaging induced any artefacts such as image drifts or focus shifts (Supplementary Note 3; Supplementary Fig. 10), we performed time-lapse imaging of 0.2 μm YG Nanospheres embedded in agarose gel over 200 imaging cycles (repeated low- and high-resolution imaging). Using TrackMate¹⁹ in Fiji²⁰, we tracked and quantified the positional displacement of nanospheres over time. We found that the high-resolution view was displaced by 0.207 ± 0.01 μm (n=1459 tracks) over 200 imaging cycles, comparable to the size of a single pixel (0.117 μm). The low-resolution view was displaced by 2.82 ± 0.02 μm (n=7698), less than 0.3% of the total image size. Analyzing the image quality with normalized DCT Shannon Entropy²¹ over time revealed no degradation in image quality and thus focus drift (Supplementary Fig. 10). All these measurements indicate that our system is robust enough for repeated imaging between low- and high-resolution imaging.

In summary, our low-resolution imaging mode provided a 1.3x1.0 mm² field of view, allowing efficient tiling of an entire cancer spheroid or a zebrafish larva in one or three tiles, respectively. Our high-resolution ASLM mode targeted sub-cellular imaging and spanned typically a field of view of 0.24 × 0.12 mm². Applied together, the low-resolution mode enabled efficient imaging and scanning of large volumes complemented by a high-resolution modality to zoom in on selected regions of interest for sub-cellular imaging (Fig. 1e). Importantly, in the configuration without motorized slit, the time to switch between both modalities was only limited by the time it took for the flip mirror to change the light-sheet properties, typically within one second (Supplementary Note 4, Supplementary Table 2).

Multi-scale control software

To control all aspects of multi-scale imaging, we implemented our own hardware control software (Supplementary Note 5). The software is designed for easy integration of the hardware components and image processing algorithms, inclusion of features for simultaneous acquisition and data processing, handling of all multi-scale aspects such as imaging modality selection, and with a clear model-view-controller (MVC) design pattern that enables future transition of the software to different microscope hardware.

We implemented the software in Python, a popular choice for microscope control software solutions^22,23. Precise timing of all hardware components was ensured with Python-based control of an NI Data Acquisition card (NI PCIe-6738) with sufficient digital and analog outputs (Supplementary Fig. 11). Importantly, Python offers many optimized image processing libraries such as OpenCV²⁴ that can be easily integrated into a Python based hardware control software for automated microscope control. To enable simultaneous processing and acquisition of image data, we leveraged concurrency tools from Thayer, York et al.²⁵ such as shared memory arrays, threads, and subprocesses. Briefly, these tools span child processes to make hardware control IO-limited rather than CPU-limited. Moreover, by using a buffer queue of shared memory arrays, acquired 3D images could be processed, while newly acquired images were saved to a different shared memory array. Furthermore, the MVC design separated the GUI from the controller and the model, including the routines for on-the-fly image processing, hardware control and concurrency to improve the logic (Supplementary Fig. 12). Using Napari²⁶, we further implemented a state-of-the art and open source image viewer to visualize acquired images (Supplementary Fig. 13). The whole control software is available on GitHub: https://github.com/DaetwylerStephan/self_driving_multiscale_control, with a dedicated documentation https://daetwylerstephan.github.io/self_driving_multiscale_control/ and video tutorial for the installation: https://youtu.be/alAD5PIA9Ps.

Self-driving multi-scale microscopy

While longitudinal imaging by LSFM over days has been previously performed at cellular resolution^10,11, high resolution imaging (i.e. at sub-cellular resolution) inside a living organism has so far been restricted to a few hours^27,28. This limitation arises as a (small) selected region of interest for high-resolution imaging will quickly move out of view due to non-linear sample growth and changes or sample movement (Fig 1f). Additionally, cells of interest might migrate within the organism away from the selected region. As outlined in the introduction and later in the discussion, this cannot be remedied by simply imaging a larger volume with high-resolution imaging (Fig. 1b).

To image a selected region of interest over hours, we implemented “self-driving microscopy” within our software (Fig. 1g). We leveraged the low-resolution information of the microscope to guide the acquisition of the high-resolution region using a custom 3D region tracking algorithm (Extended Fig. 2). To successfully apply region tracking on-the-fly, rapid image processing is paramount. Moreover, the tracking should be independent of the content of the high-resolution region so that any fluorescent channel or even brightfield transmission imaging (Extended Fig. 3) could be used to automatically guide high-resolution acquisitions.

To fulfill these requirements, our on-the-fly region tracking algorithm consisted of two steps, i.e., an initialization step at the start of high-resolution imaging of the region of interest, and an update step from the previous timepoint to the current acquisition. During the initialization, we established correspondence using calibrated stage positions between the high-resolution region and the corresponding low-resolution data in any channel (fluorescence or brightfield). We found experimentally that increasing the tracked volume by 1.5 times in the lateral directions compared to the size of the high-resolution region resulted in more robust tracking. To achieve fast tracking, we mapped the 3D volume of the tracked region into x-y, y-z, x-z maximum intensity projections and saved them in an image library. For each update, we applied OpenCV’s template matching²⁴ on multiple scales to find the previous x-y projected region in the current low-resolution data. Subsequently, the axial projections (y-z, x-z) were generated and registered to the axial projections of the previous timepoint. This resulted in the 3D shift vector of the region of interest and allowed us to update the stage positions for high-resolution imaging. Lastly, the image library was updated with the latest projections to account for changes in the region of interest over time.

In vitro and In vivo imaging of development and disease

We demonstrate the application of our self-driving, multi-scale platform to a variety of biological processes: an in vitro assay of cancer cell invasion using spheroid models, in vivo developmental processes of zebrafish gastrulation and vascular growth, and an in vivo larval zebrafish xenograft model of cancer metastasis. These biological processes extend over long-time periods and are inherently multi-scale in nature.

Invasion of cancer cells from a primary tumor into the surrounding tissue is a critical stage of cancer progression²⁹. Multi-scale imaging of this process is necessary to understand the overall behavior of a tumor, such as how many cells start migrating away from the tumor, while also monitoring single cell processes that drive individual cells at the invasive front. Cancer spheroids have become instrumental models to stage the course of these events³⁰. We formed cancer spheroids in a collagen matrix from SUM159 breast cancer cells^30,31 that are known for their high invasiveness³². Using our self-driving, multi-scale imaging platform, we visualized the entire spheroid with low-resolution, and simultaneously imaged cancer cells (trailblazers) at the invading front with sub-cellular resolution. By mixing SUM159 cells of two colors 1:1 (Lifeact-GFP and the Lifeact-mCherry), individual cells were discernible. This allowed us to visualize actin reorganization during cell division (Fig. 2a), while observing overall spheroid growth (Supplementary Video 1, 2).

Fig. 2. Applications of self-driving, multi-scale microscopy.

a Multi-scale imaging of a SUM 159 breast cancer cell spheroid embedded into a collagen matrix (Supplementary Video 1,2) with the low-resolution modality shown on top and the high-resolution modality on the bottom. The spheroids consisted of a 1:1 mixture of cells expressing the actin marker Lifeact-GFP (cyan) and Lifeact-mCherry (magenta). Boxed region on top indicates the location of the high-resolution region on the bottom. The white arrowhead points at a cell division at the invasive front. b Multi-scale imaging of zebrafish gastrulation with cells expressing the histone marker Tg(h2afva:h2afva-GFP), starting at around 6 hours post fertilization (Supplementary Video 3,4). Low-resolution imaging (on top) captured the entire zebrafish embryo (color scale: depth of data in 3D volume from 0 to 450 um), while the high-resolution imaging (bottom) enabled near-simultaneous imaging of cell division (yellow arrowheads), including sister chromatid separations. c Multi-scale imaging of human breast cancer cells MDA-MB-231 expressing F-tractin-EGFP (left: magenta, right: grayscale) in situ in a larval zebrafish xenograft model (Supplementary Video 5,6). The cancer cells were xenografted at 2.25 days post fertilization into a zebrafish larvae expressing the vascular marker Tg(kdrl:Hsa.HRAS-mCherry) (cyan). Cell spreading patterns as imaged in the low-resolution mode are shown on the left, and a 3D rendering of the high-resolution data is shown on the right. The white arrowhead points at a network of protrusions. Scale-bar lengths are as follows: a 150 μm (top), 20 μm (bottom); b 200 μm (top), 30 μm (bottom); b 300 μm (left); 40 μm (right).

Starting from a single cell, human development relies on the growth of thousands of specialized tissues encompassing trillions of cells⁴⁶. To study some of the underlying regulatory processes, researchers find zebrafish a powerful model organism³³. Zebrafish provides a short generation time, fast development and optical translucency, particularly in models without pigmentation³⁴. This is in stark contrast to the highly scattering tissues in murine models, where detailed fluorescence images can only be superficially acquired with two- or three-photon raster scanning microscopy³⁵. To demonstrate the self-driving multi-scale microscope for imaging developmental processes, we visualized cellular divisions during zebrafish gastrulation with the nuclear marker Tg(h2afva:h2afva-GFP)³⁶ (Supplementary Video 3, 4). While whole organism imaging allowed precise developmental staging and understanding of overall tissue flow, the high-resolution modality enabled precise characterizations of cell divisions, including sister chromatid separation (Fig. 2b). At a later developmental stage, we imaged the growing vascular network by combining high-resolution ASLM imaging of fluorescently tagged vasculature, labeled with Tg(kdrl:EGFP)³⁷, with LED brightfield illumination for guidance. Over more than 11 hours, we followed the growth of the vasculature at the tip of the tail, while the whole embryo was growing and expanding over 100 μm. The resulting images revealed angiogenesis and anastomosis events, including rearrangements of the nuclei during this process (Extended Fig. 3).

Cancer metastasis implicates a cascade of complex and dynamic processes during which cells disseminate from their site of origin to invade distant organs ^38,39. Multi-scale imaging permits the in vivo observation of cell behavior in different metastatic niches in an entire organism, while imaging the sub-cellular processes that drive disease progression. For this purpose, zebrafish xenografts (Extended Fig. 4a) have been established as potent in vivo assays^40,41 with different metastatic niches and a physiological environment including vasculature, blood flow, and immune immune cells^42,43. We injected MDA-MB-231⁴⁴ human breast cancer cells stably expressing the filamentous actin marker F-tractin-EGFP⁴⁵ into zebrafish expressing the vascular marker Tg(kdrl:Hsa.HRAS-mCherry)⁴⁶. By low-resolution mSPIM imaging, we visualized dissemination patterns, i.e. where cancer cells resided over time. The high-resolution ASLM imaging provided a detailed view on the cellular shapes, actin distribution and intricate network of protrusions that MDA-MB-231 cells formed inside the vasculature to attach to the vessel walls and other cells (Fig. 2c, Supplementary Video 5, 6).

In vivo imaging of macrophage- cancer cell interactions

To demonstrate the use of our self-driving, multi-scale microscopy platform for quantitative biology beyond visualization, we studied immune-cancer cell interactions in a zebrafish xenograft model. Zebrafish xenografts have been exploited to study interactions of cancer cells with cells of the innate immune system such as macrophages⁴⁷. However, high-resolution images and quantification of these interactions at different scales over many hours are still lacking. We injected human U-2 OS osteosarcoma cancer cells⁴⁸, labelled with pVimentin-PsmOrange⁴⁹, into zebrafish larvae with macrophages labelled with Tg(mpeg1:EGFP)⁵⁰ and vasculature labelled with Tg(kdrl:Hsa.HRAS-mCherry)⁴⁶ for tissue context (Extended Fig. 4). Two hours after injection, zebrafish with intact circulation and few isolated cancer colonies in the tail were selected for imaging.

Our low-resolution, long-term time-lapse movie revealed overall macrophage dissemination in the xenografted larvae. Macrophages clustered around sites where cancer cells were present (Fig. 3a, Extended Fig. 4b, Supplementary Videos 7,8). In contrast, in zebrafish larvae without injected cancer cells, macrophages were uniformly distributed (Fig. 3b, Extended Fig. 4c, Supplementary Video 9). Applying our self-driving microscope feature, the microscope observed and followed a selected micro-metastatic niche over 16 hours with high-resolution imaging to reveal how zebrafish macrophages interacted with and ultimately phagocytosed U-2 OS cancer cells, as the zebrafish larva grew (Fig. 3c, Extended Fig. 4d). Interestingly, several macrophages were present at the site of cancer cell ingestion, hinting at collective phagocytosis. While cancer cells formed a metastatic niche, they were remodeling the vascular network around them. After phagocytosis, the vasculature relaxed back to its normal state.

Fig. 3: Analysis of self-driving, multi-scale data.

a Low-resolution, 3D rendering of a zebrafish xenograft with U-2 OS osteosarcoma cancer cells (red), segmented macrophages (color scale: violet to green from head to tail), and a vasculature label, Tg(kdrl:Hsa.HRAS-mCherry) (gray). Boxed region indicates the location of high-resolution region in panel c. b Low-resolution, 3D rendering of a zebrafish larva without xenograft (control) with segmented macrophages (color scale: violet to red from head to tail) and a vasculature label, Tg(kdrl:Hsa.HRAS-mCherry) (gray). c 3D rendering of high-resolution stills with segmented macrophages (in color), U-2 OS cancer cells (red, white arrowhead) and vasculature (gray). Boxed region indicates individual macrophage highlighted in panel d. d 3D renderings of individual macrophages from the high-resolution data of xenograft experiment (top) and control (bottom). Color indicates mean cellular curvature. e Hopkins statistics (10% sampling, dimensionality d=3) to analyze macrophage cluster tendency inside a zebrafish. A higher number indicates a higher cluster tendency with U-2 OS xenograft data in violet, control data (no xenograft) data in yellow, and a simulation of a randomized, uniform distribution in black. Data displayed as average ± 95% confidence interval (n=350 iterations for Hopkins statistic calculation). Dashed line indicates moving average of 10 timepoints. f Global morphological feature analysis of macrophage shapes (gray points, n=825 cells) from the high-resolution data of the U-2 OS cancer cell xenograft experiment over time, displayed in a principal component analysis (PCA) plot. Bagplots of all macrophage shapes in the first hour (hour 0, turquoise) and hour 16 (brown) of observation are overlaid. The bagplot consists of the Tukey median (cross), an inner polygon (darker color) that contains the 50% observations with largest Tukey depth, and the outer polygon (lighter color) with all data excluding outliers. g Change of macrophage shapes over time: all macrophage shapes within an hour of observation were compared to macrophage shapes at different hours by applying a permutation test (one-sided, no adjustment for multiple comparison, n=3000 iterations) using the Tukey median as a metric. The resulting p-values were plotted as a heat map. Scale-bar lengths are as follows: a,b 500 μm; c 50 μm.

In contrast, in more metastatic cancer cells such as A375 melanoma cells⁵¹, we did not observe such phagocytosis events over the same observation window despite close interactions between immune cells and A375 cells (Supplementary Note 6; Supplementary Fig. 14, Supplementary Video 10). This was also reflected in the high survival rate of A375 cells in the tail over 48 hours since xenografting (Supplementary Fig. 15). However, we observed single apoptosis events of A375 cancer cells with subsequent clearance by macrophages (Supplementary Fig. 16, 17).

To quantify the macrophage dynamics during phagocytosis of U-2 OS cancer cells across scales, we segmented the individual macrophages in both the low- and high-resolution data (Supplementary Note 7). For the low-resolution data, we applied GPU-accelerated segmentation with py-clesperanto based on CLIJ⁵² (Fig 3a, b; Supplementary Fig. 18, Supplementary Videos 11, 12). However, the watershed-based approach failed to segment macrophages in the high-resolution data. This was due to the complex, non-spherical macrophage morphologies (Extended Fig. 4d, Extended Fig. 5a). Circulating and resident macrophages have large cytoplasmic branches, likely to survey as much tissue as possible for tissue damage or pathogens. Moreover, during phagocytosis, more compact macrophages clustered together (Fig. 3c; Extended Fig. 5a), rendering the separation of neighboring cells a difficult task. To overcome these challenges, we developed a custom segmentation workflow (Fig. 3c, Extended Fig. 5c–g). This approach first used multi-Otsu thresholding⁵³ and connected component labeling, to segment individual macrophages and identify macrophage clusters. To resolve individual macrophages inside the clusters, we complemented the initial segmentation with a 3D consensus segmentation⁵⁴ based on 2D Cellpose⁵⁵ segmentations of x-y, x-z, y-z views. Finally, manual curation was employed to resolve any segmentation ambiguities. The resulting segmentation provided a detailed characterization of macrophage morphologies in 3D over time (Supplementary Video 13) and different conditions, e.g., xenograft vs control (Fig. 3d, Supplementary Video 14). The high quality of the segmentation allowed us to represent individual cell surfaces as meshes and to calculate the mean cellular curvature on them (Fig. 3d).

The segmentations in low- and high-resolution data enabled a quantitative comparison of macrophage behavior across scales. In the low-resolution data, we quantified the spatial dissemination pattern of macrophages using the Hopkins statistic H⁵⁶. Originally introduced to study plant distribution, the Hopkins statistic measures cluster tendency of the segmented macrophage positions against an equivalent number of randomly sampled positions in the zebrafish (Methods, Supplementary Fig. 19). According to literature, a value of H>0.75 indicates at cluster tendency at the 90% confidence level⁵⁷. In our experiments, the qualitative observation of clustering tendency of macrophages around cancer cells in the xenograft was confirmed by the high value of the Hopkins statistics over the 16 hours of time-lapse imaging. Except for one time point at the end of the recording, H was always above 0.75. In contrast, the macrophage dissemination without xenografted cancer cells (H< 0.65) was close to a uniform distribution, and H was always less than in the xenograft experiment (Fig. 3e).

In the high-resolution imaging data, we used the segmentations to measure global cell shape features (Supplementary Table 3)⁴² and track putative morphological changes during interaction with U-2 OS cells. We mapped the macrophage shapes into a 2D principal components analysis (PCA) space (Fig. 3f), explaining 82.7% of the observed shape variability in cell shapes, justifying the cut off after 2 components (Supplementary Fig. 20). In this 2D PCA space, the shapes of different macrophage populations could be compared with each other using bagplots⁵⁸ as the 2D equivalent of boxplots. To obtain insights into whether macrophages change their morphologies over the course of the time-lapse imaging, we grouped macrophage shapes defined by the hour of observation. Next, we compared each of these populations with each other using a permutation test based on the Tukey median metric⁵⁸ (Fig. 3g), e.g., all macrophage shapes observed in hour zero vs. in hour sixteen. Changes in macrophage shapes correlated with changes in biological function, i.e., circulating macrophages, first attachments, tight attachment, phagocytosis and dissolving of the clusters (Fig. 3g, Supplementary Video 13). Taken together, our approach uncovers unique properties of macrophages interacting with cancer cells across spatial scales at biologically relevant temporal scales, demonstrating the utility and potential of self-driving multi-scale microscopy.

Discussion

We developed a self-driving, multi-scale light-sheet imaging platform to observe and quantify biological processes from sub-cellular to organismal scales in vivo over long time periods. It enabled imaging of large 3D volumes over long time spans to reveal cell spreading and migration patterns, and to find rare dynamic events such as immune-cancer cell interactions, in the context of dynamic organisms. Guided by custom-written self-driving hardware control, the microscope could then selectively focus on processes of interest to concurrently visualize molecular signaling and morphological changes, such as cell shape changes, chromatid segregation, cell-cell interactions, or cytoskeletal rearrangements.

The need for multi-scale platforms arises as current microscopy platforms are limited in their spatiotemporal sampling due to technical and practical limitations¹³. To overcome these limitations, multi-scale microscope systems have been described to image at different spatial scales. These microscopes have so far been dedicated to imaging fixed, cleared and expanded tissues¹², are not suited for live imaging of larger 3D samples such as spheroids or entire living organisms away from a coverslip^15–17, or do not provide the imaging resolution and technical considerations for capturing dynamic sub-cellular processes in living organisms^59,60. However, living organisms and complex in vitro samples such as spheroids are integral for establishing mechanistic understanding of dynamic biological interactions in development and disease. Our multi-scale microscope provides the necessary spatial and temporal resolution and technical considerations to meet the requirements of contextual live imaging of processes at the sub-cellular scale.

Importantly, multi-scale imaging alone is not sufficient to collect high-resolution data within living, growing organisms that move themselves. Consequently, regions of interest quickly move out of the field of view (Fig. 1g). This has hitherto prevented long-term, high-resolution acquisitions in situ in living organisms over more than a few hours. To overcome this challenge, our multi-scale platform incorporates functionality for self-driving data acquisition, where low-resolution imaging provides biological information at the level of the full organism while guiding high-resolution imaging for observation of a subset of critical events at the cellular and sub-cellular level.

To enable self-driving microscopy, real-time analysis of microscope images is required. For one single imaging modality, adaptive microscopy workflows have been implemented prior to the presented work. In a landmark paper, McDole et al.¹¹ describe adaptive light-sheet microscopy for mouse embryo imaging with automatic adjustment of the overall imaging volume, building on the previously published automated microscopy routine AutoPilot framework for near-constant image size²¹. However, these routines do not extend to multi-scale imaging, which requires integration of low-resolution and high-resolution images.

In addition, our implementation for self-driving adaptive imaging using multi-scale information provides several key advantages in terms of motion prediction. High-resolution imaging of the same volume is substantially slower than low-resolution imaging and requires considerable computational storage due to the larger amount of data. Moreover, the field of view of high-resolution imaging is much smaller than in a low-resolution modality. Hence, high-resolution motion prediction cannot handle large, unpredictable sample movements. Furthermore, high-resolution imaging incurs higher sample irradiation and thus more photo-toxicity, limiting the duration and frequency of experiments. In contrast, low-resolution light-sheet imaging has shown to be gentle on the specimen^1,61,62, as confirmed by our data (Supplementary Note 8, Supplementary Fig. 21). It also opens the door to more flexible experimental designs, e.g., by leveraging bright-field transmission imaging of non-labelled large tissue structures for guidance (Extended Fig. 3), further reducing concerns about photobleaching and phototoxicity. Thus, our platform is, to our knowledge, the first to allow high-resolution image acquisition in a living organism for extended durations of several hours to days.

The second enabling technology for a robust application of self-driving microscopy is the tracking of regions of interest and specimens such as individual cells. Data-driven methods such as deep-learning networks have outperformed classical algorithms for tasks such as segmentation⁵⁴. We have leveraged this in our segmentation of the high-resolution data. However, these networks demonstrate no advantage in performance for tasks such as image registration⁶³ or cell linking techniques⁶⁴ essential for motion tracking control. Moreover deep learning-based networks must be retrained or adapted on curated datasets to recognize specific processes¹⁷. For these reasons, we have applied established template matching implemented by industry-performant OpenCV library²⁴ to perform process- and fluorophore-independent motion tracking control. Importantly, template matching does not require training data and is tolerant to image intensity variations, making it suitable for diverse applications, including studies of novel and rare biological events, where limited or no training data is available. A possible advantage of machine-learning based methods is processing time, however, our tracking already runs in real-time. Nevertheless, our software has been designed to incorporate future developments, allowing users to integrate such machine learning algorithms if they wish.

In our applications, we selected regions of biological interest at the start of an experiment and followed these over long time periods of several hours to days. The resulting imaging in the high-resolution mode demonstrates that our resolving power is sufficient to quantify sub-cellular details such as cell shape features and mean cellular curvature, which has been shown to be an important organizer and potential driver of cell signaling^65,66. Our experiments on macrophage-cancer cell interactions indicate how context-driven high-resolution time lapse imaging provide insights into the morphological and eventually molecular programs that underlie immune system functions. Thereby, we observed that U-2 OS cells were readily cleared by macrophages, while A375 evaded the immune system. This mirrors studies in murine models, where U-2 OS cells do not metastasize well ⁶⁷, in contrast to A375 cells⁶⁸. Furthermore, leveraging these features, the proposed microscopy platform has already been applied to functional studies of the protein bridge-like lipid transfer protein family member 2 (BLTP2) in cancer migration and survival⁶⁹. We envision to extend the versatility of our platform in the future by guiding the high-resolution data acquisition with event-based approaches instead of selecting region of interest at the start of an experiment. Thereby, the availability of low-resolution data opens the path forward to search for and image specific processes of interest in high-resolution in an entire organism.

For the microscope user, the multi-resolution imaging capabilities provided an improved user experience as large volumes could be easily navigated in the low-resolution view before switching to high-resolution imaging of regions of interest. This contrasts with dedicated high-resolution microscopes (in particular light-sheet architectures), which have a fixed magnification and are constrained to a rather small field of view, making navigation within a 3D organism cumbersome.

As our system was designed for sub-cellular and organismal imaging in vivo, we leveraged a traditional light-sheet layout to optimize image performance. In turn, this constrained the sample mounting in comparison with recent open-top geometry implementations^12,70,71 that may provide more flexibility in sample preparation. However, both the system design and sample holders can be adapted towards specific research questions.

In summary, we have built, programmed, and applied a multi-scale microscope that enables observations of biological processes of interest from sub-cellular to organismal scales. Due to its versatility, we anticipate that the platform will empower a broad spectrum of studies of development and disease progression with sub-cellular insights.

Methods

The research within this work complies with all relevant ethical regulations as reviewed and approved by the University of Texas Southwestern Medical Center.

Microscope layout

We have implemented two configurations of the multi-scale microscope hardware (Supplementary Note 1). The first configuration includes cylindrical lenses and a motorized slit to change light-sheet properties. In the second configuration, the low- and high-resolution light-sheets are generated by Powell lenses and no motorized slit is required to change light-sheet properties when changing from low- to high-resolution imaging, making changes between both microscope imaging resolutions faster. A schematic of both microscope layouts is available as Extended Fig. 1 with a detailed list of all microscope hardware components available in Supplementary Table 1. Moreover, both illumination light paths are characterized in more detail in Supplementary Figures 1–4. Acquired data was stored on the BioHPC cluster, provided by UT Southwestern Medical School.

In more detail, in both configurations (Extended Figure 1), first the laser lines of the 4 different lasers were combined by dichroic beamsplitters, followed by a pinhole through which the light was focused with a 4x telescope (f1=50mm, f2=200mm). Next, a halfwave plate and a polarized beamsplitter cube adjusted how much laser light went into which illumination arms. A flip mirror in the low-resolution arm blocked light during high-resolution imaging.

In the first design, a telescope consisting of two cylindrical lenses (f1=25mm, 100mm) expanded the beam in the low-resolution illumination arm in one direction to minimize light loss at the vertical slit. A cylindrical lens (f=200mm) formed the light-sheet that was pivoted by a resonant galvanometer. Finally, another tube lens (f=100mm) and an illumination objective (IL, NA 0.4) formed the light sheet. In the high-resolution arm (top illumination arm), first the beam was further expanded by a telescope (5x). Next, a halfwave plate rotated the polarization of the beam so that all light was sent onto the refocusing unit. Before that, the beam was modified by a motorized vertical slit and a light-sheet was formed with a cylindrical lens (f=50 mm), which was pivoted by a resonant galvanometer. The slit was fully opened in the high-resolution mode and only narrowly opened in the low-resolution mode to provide a larger light-sheet. Next, a tube lens (f=200mm) and the remote focusing objective mapped the light-sheet onto the voice coil mirror. The light was sent onto the voice coil mirror through a polarizing beamsplitter cube (PBS) and a quarterwave plate, which optimized the transmission of the laser light through the PBS. The remote focusing unit was conjugated through another 4f system / telescope (f1=200mm, f2=75mm) to the illumination objective of the high-resolution arm.

In the second configuration, the low-resolution illumination arm was composed of a vertical slit in front of a 10° Powell lens , followed by a 30 mm lens, a resonant galvanometer and a second lens (f=60 mm). Another telescope (f1=200 mm, f2=75 mm) together with the illumination objective formed the low-resolution light-sheet. The high-resolution illumination arm was split by a flip mirror into another low-resolution arm and the high-resolution arm, each with dedicated optics for light-sheet generation. These two illumination arms were again combined by a polarized beamsplitter cube (PBS) into a single path. The low-resolution arm was composed of the same elements as the dedicated low-resolution arm: vertical slit, Powell lens, 30mm lens, resonant galvanometer, and 60mm lens. In the high-resolution arm, the Powell lens was followed by a 30mm lens and a resonant galvanometer. A 400 mm lens conjugated the beam to the remote focusing unit consisting of a quarterwave plate, the remote focusing objective and the voice coil with a mirror. The remote focusing unit was conjugated through another telescope (f1=200mm, f2=75mm) to the illumination objective of the high-resolution arm.

In both configurations, an LED illumination was available to position and image the sample in transmission (bright field). The microscope hardware comfortably accommodated several samples in one experiment, mounted in low melting agarose within fluorinated ethylene propylene (FEP) tubes. These samples were mechanically moved by translational and rotational stages from SmarAct, with 83 mm travel range in vertical direction (SmarAct, SLC-24120) and 21 mm travel range in horizontal directions (SmarAct, SLS-3232).

Moreover, in both configurations, samples were imaged with 20x, NA1.0 detection objective followed by a filter wheel. A motorized flip mirror then selected between low- and high-resolution detection paths. In the low-resolution detection path, a 100 mm tube lens was used for imaging onto a Teledyne Photometrics, Iris 15 camera. In the high-resolution detection path, a 500 mm tube lens was used for imaging onto a Teledyne Photometrics, Prime BSI Express camera.

Microscope control software

The microscope was controlled by a custom Python-based microscope control software based on an MVC design pattern (Supplementary Fig. 12). It featured concurrent acquisition and processing, an intuitive graphical user interface with a state-of-the art image viewer for ease of use (Supplementary Fig. 13), automated generation of maximum intensity projections for fast browsing of the acquired data, and control of different light-sheet modalities including mSPIM and ASLM. The control code is available as Github repository:

https://github.com/DaetwylerStephan/self_driving_multiscale_control

Fluorescent nanosphere measurements

Fluorescent nanosphere samples of 0.2 μm YG Nanospheres (Polyscience, 17151) were prepared in 2% low melting agarose (Sigma Aldrich, A9045) at a concentration of 1:1000 – 1:10’000. After imaging, the point spread function (Fig. 1c, Supplementary Fig. 7) was determined using the plugin MetroloJ⁷² in Fiji²⁰ (n=20 for each resolution level). The mean and confidence interval (95%) were calculated using R, and the data was checked for normality of the measurements (Supplementary Note 2). To measure the point spread functions over an entire 3D volume (Supplementary Fig. 8), we wrote our own Python code to automatically perform PSF measurements using the same algorithm as in the manual MetroloJ plugin (Github: PSF_measurements.py) (Supplementary Note 2).

Zebrafish husbandry and sample preparation

Zebrafish husbandry and experiments described here have been approved and conducted under the oversight of the Institutional Animal Care and Use Committee (IACUC) at UT Southwestern under protocol number 101805. Zebrafish (Danio rerio) adults and embryos were kept at 28.5°C and were handled according to established protocols^73,74. All zebrafish experiments were performed at the embryonic and larval stage and therefore the sex of the organism was not yet determined.

To visualize the development of the zebrafish vasculature, the zebrafish line expressing the vascular marker Tg(kdrl:EGFP)³⁷ was imaged. To visualize macrophage interaction with cancer cells, the fish line Tg(mpeg1:EGFP)⁵⁰ was crossed with zebrafish expressing the vascular marker Tg(kdrl:Hsa.HRAS-mCherry)⁴⁶ in a casper background³⁴. To visualize the nuclei inside developing zebrafish, the fish line Tg(h2afva:h2afva-GFP)³⁶ was used.

To immobilize the zebrafish for imaging, zebrafish embryos were anesthetized with 200 mg/l Tricaine (Sigma Aldrich, E10521)⁷⁵ during imaging. To mechanically mount up to five zebrafish embryos for imaging, they were individually embedded in 0.1% low melting agarose (Sigma Aldrich, A9414) within fluorinated ethylene propylene (FEP) tubes (Pro Liquid GmbH, Art: 2001048_E; inner diameter 0.8 mm and outer diameter 1.2 mm), coated with 3% methyl cellulose (Sigma Aldrich, M0387), and connected as previously described¹⁰.

Spheroid sample preparation

SUM159 breast cancer cells^30,31 were cultured in DMEM/F12 media (Gibco, 11320033) containing 5% FBS (Peak Serum), 1ug/ml hydrocortisone (Sigma Aldrich, H0888), 5ug/ml bovine insulin (Sigma Aldrich, I6634), and 1% Penicillin-Streptomycin (Gibco, 15140122). A Lifeact-GFP and a Lifeact-mCherry expressing SUM159 cell line were established through lipofectamine transfection according to the manufacturer’s protocol. pTK92_Lifeact-GFP (Addgene plasmid # 46356) and pTK93_Lifeact-mCherry (Addgene plasmid # 46357) were obtained from Addgene.

To obtain mosaic-labelled cancer spheroids, a 1:1 mixture of the Lifeact-GFP and the Lifeact-mCherry SUM159 cell lines (1000 cells each) was incubated in 96-well Nunclon Sphera low adhesion plates (Thermo Fisher Scientific, 174925) for 48 hours at 37°C and 5% CO₂. The spheroids were then resuspended in a 50 uL mixture of 70% activated rat tail Collagen 1 (Corning, catalog no. 354236) and 30% Cultrex UltiMatrix Basement Membrane Extract (R&D systems, BME001) at a final concentration of 2.1 mg/ml Collagen 1 and 3mg/ml UltiMatrix BME. The resuspended spheroids were then embedded in sterile FEP tubes containing holes punctured with 30 G needles for increased circulation. FEP sterilization was achieved by flushing the tubes with 70% ethanol and letting them dry in a closed plate in the incubator. After 24 hours of additional incubation at 37°C and 5% CO₂, the spheroids were imaged.

Cell culture for zebrafish xenograft

A375 cells⁵¹ (ATCC CRL1619) and MDA-MB-231⁴⁴ cells (ATCC HTB-26) were acquired from ATCC. U-2 OS cells⁴⁸ were obtained from Richard McIntosh (University of Colorado, Boulder CO). To test for mycoplasma contamination, the test kit Genlantis MycoScope PCR Detection Kit (MY01100) was used. U-2 OS, MDA-MB-231, and A375 cells were cultured in DMEM-high glucose (Sigma Aldrich, D6429) supplemented with 10% fetal bovine serum (FBS; Peak Serum) and 1% antibiotic-antimycotic (Gibco, 15240062) at 37 °C and 5% CO₂.

To complement the zebrafish macrophage marker Tg(mpeg1:EGFP)⁵⁰ and the vascular marker Tg(kdrl:Hsa.HRAS-mCherry)⁴⁶, we expressed the fluorescent marker pVimentin-PSmOrange⁴⁹ in the xenografted U-2 OS and A375 cancer cells. pVimentin-PSmOrange was obtained from Addgene (plasmid # 31922). The MDA-MB-231 cells stably expressed F-tractin-EGFP⁴⁵ to label the cells and reveal their actin organization. The F-tractin-EGFP construct was obtained from Addgene (plasmid # 58473).

To obtain fluorescently labelled cancer cells, U-2 OS cells were transfected with Lipofectamine LTX&PLUS Reagent (Invitrogen, 15338100) according to manufacturer’s protocol. On day 0, 300’000 cells were seeded per well of a 6-well plate (ThermoFisher, 140675). On day 1, cells were transfected in Opti-MEM Reduced Serum Medium (Gibco, 31985062) with Lipofectamine LTX&PLUS Reagent (Invitrogen, 15338100) at a concentration 3.5 ug DNA, 3.5 ul Plus and 12 ul LTX reagent. On day 2, the media was changed back to normal cell culture media and cells were selected using antibiotic resistance to neomycin (Gibco 10131035). A375 cells were transfected on day 1 with FuGENE HD (Promega, E2311) according to the manufacturer’s protocol. 150ul/well transfection mixture was prepared with 3 ug DNA and 9 ul FuGENE® HD Transfection Reagent in sterile water and 10 min complexing time at room temperature.

Zebrafish xenografts

To prepare for a successful xenograft and imaging experiment, zebrafish embryos were collected and treated with 0.1 mM N-Phenylthiourea (PTU, Sigma Aldrich P7629) starting at 1 day post fertilization (dpf) to prevent pigmentation. At 2.25 dpf, the dechorionated embryos were xenografted with cancer cells. For this, cells were harvested at 70-90% confluency and trypsinized for 3 min (Gibco, 15400-054). If cells were highly adherent, cells were run through a 70-μm cell strainer (Fisherbrand, 22-363-548). To achieve an optimal number and density of cells for microinjection, 4 × 10⁶ cells in 40 uL normal cell culture media were prepared and stored on ice until xenografting.

Xenografts were generated by injection of the prepared cell mixture into the yolk near the common cardinal vein (CCV) of zebrafish larvae with glass capillary needles (World Precision Instruments, 1B100-4), pulled on a micropipette puller (Sutter Instrument, P-1000). Thereby, 50-1000 cells were injected per fish. During the injection, zebrafish were anesthetized with Tricaine (Sigma Aldrich, E10521). Two hours after injection, zebrafish larvae with few single cancer cells in the zebrafish tail and intact circulation were selected under a fluorescent stereomicroscope for imaging.

Low-resolution image tile stitching

Acquisition of three individual low-resolution tiles was required to cover the entire zebrafish larva. To fuse the three individual tiles to one 3D volume, we developed our own Python based stitching workflow (Github: multiScale_stitching.py). Firstly, we determined a coarse positioning of the tiles using the physical coordinates of the microscope stage position. Secondly, we calculated the maximum intensity projections of each tile to reduce the computational workload. Next, we calculated Fast Fourier transform-based cross-correlations on the overlapping sections of the background subtracted maximum intensity projections and obtained the translational shift between tiles as the highest value of all cross-correlations. Lastly, we fused the tiles using a sigmoidal blending function.

Low-resolution image segmentation of xenograft data

We segmented the fused low-resolution images to obtain the positions and dissemination patterns of macrophages (Supplementary Fig. 18). To segment the low-resolution macrophage images, we used GPU accelerated segmentation with py-clesperanto based on CLIJ⁵² (https://github.com/clEsperanto/pyclesperanto_prototype). Segmentation parameters for the low-resolution segmentation were tuned with a graphical user interface in Napari (https://github.com/haesleinhuepf/napari-workflow-optimizer).

Specifically, before applying the segmentation workflow, we rescaled the data to isotropic voxel spacings to obtain accurate segmentations. Thereby, we downsampled the lateral pixel spacings to fit the larger axial plane-to-plane spacings. This allowed us to compress the data from ~7 GB to ~85 MB, enabling fast GPU accelerated processing on the local BioHPC cluster computer, equipped with Tesla V100 32GB GPU cards.

The segmentation workflow comprised pre-processing, thresholding, labeling with Voronoi-Otsu labeling on the binary image, and post-processing. To pre-process the data, we applied a top-hat filter for background subtraction with radius 6 in all directions. We then thresholded the image with a manually optimized threshold in Napari, and then applied Voronoi-Otsu labeling, which included two Gaussian blurs, spot detection, Otsu-thresholding and Voronoi labeling. Thereby, we used 2.0 as spot sigma and 1.0 as outline sigma as parameters in the Voronoi-Otsu labeling. Next, we removed spurious labels below a size of 10 voxels. Lastly, we saved the segmentation statistics and outcome parameters as an xlsx file for further analysis. The workflow can be found in the following Python file on Github: segment_lowres_3d_stitched.py.

Hopkins statistics for analysis of spatial distribution

To quantify the cluster tendency of macrophages, we obtained the centroid position of each cell in the 3D volume from the low-resolution macrophage segmentations (real positions). Moreover, we calculated the 3D zebrafish volume V as a space for possible locations of macrophages to compare the observed spatial distribution (real positions) against randomly generated distributions (random positions) (Supplementary Fig. 19). We used these data points to test for evidence that the observed macrophage distributions contained clusters. To this end, we computed the Hopkins statistics H⁵⁶, defined as^76,77:

$$H=\frac{{\sum }_{i=1}^m{u}_i^d}{{\sum }_{i=1}^m{u}_i^d+{\sum }_{i=1}^m{w}_i^d}$$

where u_i is the Euclidean distance from a random position within the shape of the zebrafish V to the nearest-neighbor location of a real segmented macrophage position, and w_i the Euclidean distance from a randomly selected real segmented macrophage position to the nearest-neighbor location of a real segmented macrophage position. The dimensionality d was set to three for the three spatial degrees of freedom associated with the macrophage positions. The number of random samples m calculated in the Hopkins statistic H was set to 10% of the total number of macrophages n for each time point. To account for variability in the random sampling, we calculated the Hopkins statistics 350 times, which we found to be a stable number of iterations, and determined the 95% confidence interval for display. The source code with a compiled Matlab version is available on Github.

High-resolution image segmentation of xenograft data

To analyze macrophage morphology from the high-resolution data (Fig. 3d, Extended Fig. 5a), a multi-step segmentation pipeline was required (Extended Fig. 5b, c). First, we segmented the high-resolution data using multi-otsu thresholding⁷⁸ from the scikit-image library⁷⁹, with subsequent connected component labeling (Github: ConnectedComponent_MacrophageSegmentation.py). To optimize the segmentation, we performed data preprocessing, including an optional registration of the raw dataset using PyStackReg⁸⁰, background smoothing, normalization, and Wiener-Hunt deconvolution with scikit-image. This segmentation performed well at identifying single, protrusive macrophages. Next, as cancer cell label (psmOrange) showed bleed-through to the macrophage channel (488 nm excitation), we removed the cancer cell signal from the images (postprocessing_removeCancersignal.py).

However, this approach did not distinguish individual macrophages that touch, e.g. during phagocytosis (Extended Fig. 5d). Therefore, we complemented the multi-otsu thresholding with deep learning based segmentation. For this, we used Cellpose⁵⁵ and its cytoplasm 2.0 model (“cyto2”) to compute segmentations slice-by-slice in x-y, x-z, y-z views, and aggregated the resulting 2D segmentation into a single consensus 3D segmentation using u-Segment3D⁵⁴ (Cellpose_based_MacrophageSegmentation.py).

To merge both segmentations (Merge_ConnectedComponents_Cellpose.py), we identified macrophage segmentation volumes in the multi-otsu thresholding approach that exceeded a defined threshold in size (40’000 voxels). The threshold was manually chosen based on a histogram distribution of the macrophage size distribution in the control data (i.e. no cancer cell xenograft), where macrophages rarely interacted with one another. We then replaced the label of the multi-otsu thresholding approach with the labels of the Cellpose-based segmentation. After replacement, we postprocessed the merged segmentation by merging labels below a defined threshold (5000 voxels) with the closest label.

Lastly, as macrophage segmentation was very challenging due to their clustering and shape differences, we performed manual curation of the dataset (Manual_curation.py) with a Python based script and Fiji²⁰.

Analysis of macrophage morphology over time

To analyze the morphological variation of macrophages over time, we performed global morphological feature analysis in Matlab vR2023b (MathWorks, Inc.), adapted from Segal et al.⁴². We included 12 geometric features, including volume, surface area, solidity, sphericity, longest length, extend, aspect ratio, roughness, volume sphericity, radius sphericity, ratio sphericity, and circumscribed sphere area ratio (Supplementary Table 3). To calculate the circumscribed sphere, we applied Matlab code written by Anton Semechko⁸¹ based on Ritter’s algorithm⁸². To visualize and analyze the resulting data in a 2D morphological shape space, we reduced the dimensionality of the features using principal component analysis (PCA).

To assess the shape variation of macrophages over time, we grouped data from one hour of imaging (three time points), and compared the spatial distribution of each group across all time points in the 2D principle component analysis (PCA) space. To assess the similarity between each pair, a permutation test with 3000 iterations was employed. This involved the random shuffling of the positions in the 2D PCA space, and the subsequent measurements of the Euclidean distance between the Tukey median of each new distribution (see Rouseeuw et al.⁵⁸). Similar distributions have a zero Tukey median difference. The source code with a compiled Matlab version is available on Github.

Analysis and rendering of macrophage curvature.

The high-resolution data provided enough resolution to analyze the curvature of the macrophage cell surface. For this, the surface mesh was extracted from the 3D image segmentation using marching cubes with the ‘isosurface’ function in Matlab vR2023b (MathWorks, Inc.). The resulting mesh was smoothed with the Taubin filter⁸³ using the ‘smoothSurfaceMesh’ function in MATLAB and the mean curvature was computed using principal curvatures as described by Rusinkiewicz^84,85. For visualization, we used u-shape3D⁸⁶ to color the triangle mesh using a blue, white, red colormap corresponding to negative, flat and positive curvature for a mean curvature range of [$-1\mathrm{\mu }{\mathrm{m}}^{-1}$, $1\mathrm{\mu }{\mathrm{m}}^{-1}$]. The mesh was rendered using ChimeraX v.1.2.5⁸⁷. The source code with a compiled Matlab version is available on Github.

Image and movie visualization

To visualize the high- and low-resolution data, we leveraged different software packages. For display, maximum intensity projections and 2D images were contrast, level and gamma adjusted in Fiji/ ImageJ²⁰ and Adobe Photoshop. Optionally, a Gaussian blur and/or sharpen operation in Fiji was applied. 3D renderings were generated with Fiji’s 3D Viewer, Oxford Instruments Imaris Viewer 10.1.0, Agave Renderer v15.0. (https://github.com/allen-cell-animated/agave) and Napari 0.4.19²⁶. Movies were annotated with boxes and arrows using a dedicated Fiji plugin⁸⁸.

Statistics and Reproducibility

Our experiments, demonstrating a self-driving, multi-scale microscope platform, were not hypothesis-driven. Therefore, sample sizes were not predetermined due to a lack in expectations for effect sizes. To ensure replicability, we made sure to at least perform n=3 independent repeats for technical characterizations of the microscope. Applications of the microscope to different biological questions, including subsequent segmentation and shape analysis, were performed once. Randomization and blinding were not applicable in this study and were not performed.

Data availability

Test data for all our analysis code is available on Zenodo⁸⁹: https://zenodo.org/records/12791724

All raw imaging data (several TB of data) is available on Synapse^90,91, the official storage space of the MC2 center, supported by NIH:

• Test data collection (same as Zenodo): https://doi.org/10.7303/syn61795850.2

• Experiment datasets collection: https://doi.org/10.7303/syn61795837.2

Code availability

All algorithms, code and software used in this study are available on Github. The microscope control code and all image processing software is available at:

https://github.com/DaetwylerStephan/self_driving_multiscale_control

https://github.com/DaetwylerStephan/multi-scale-image-analysis

Extended Data

Extended Figure 1. Schematic of the multi-scale microscope.

We introduce two designs for a multi-scale microscope with dual-sided illumination (grey), a remote focusing unit (light grey) and detection with two resolutions (orange): a one design operating with cylindrical lenses and using a motorized slit, and b one design using Powell lenses, where no motorized slit is required (Supplementary Note 1). The microscope hardware comfortably accommodated several samples in one experiment, mounted in low melting agarose within fluorinated ethylene propylene (FEP) tubes.

Abbreviations are as follows: 4 different lasers (LS1-LS4), dichroic beamsplitters (DC), 4x telescope (T1, f1=50mm, f2=200mm), pinhole (PI), halfwave plate (λ1/2), polarized beamsplitter cube (PBS), motorized flip mirror (FM), two cylindrical lenses (CT, f1=25mm, 100mm), vertical slit (VS), resonant galvanometer (RM), low-resolution cylindrical lens (Cl, f=200mm), tube lens (TLi, f=100mm) illumination objective (IL, NA 0.4), telescope (T2, 5x), motorized vertical slit (mVS), high-resolution cylindrical lens (Ch, f=50 mm), tube lens (Trm, f=200mm), remote focusing objective (RF), a quarter-wave plate (QWP), telescope (T3, f1=200mm, f2=75mm), 10° Powell lens (P), f=30 mm lens (L1), f=60 mm lens (L2), f=400 mm lens (L3), voice coil with a mirror (VCM), LED illumination (LED), 20x NA1.0 detection objective (DL), filter wheel (FW), 100 mm tube lens (TL1), 500 mm tube lens (TL2)

Extended Figure 2: Region tracking algorithm.

a Low-resolution maximum intensity projection of MDA-MB-231 breast cancer cells, labelled with F-tractin-GFP, in the zebrafish tail. Insets depict a high-resolution region of interest (blue, panel b) with its corresponding, enlarged (1.5x in both lateral dimensions) region used for better tracking. b 3D rendering of the high-resolution region of interest. c Schematic of the steps required for initialization of the region tracking algorithm. d X-Y, Y-Z, X-Z low-resolution maximum intensity projections saved in the image library at initialization, and each subsequent timepoint / update step. e Schematic of simultaneous data processing and acquisition (data streaming), enabled by a custom buffer architecture using shared memory arrays and tools from the concurrency tool library by Thayer, York et al.²⁵. f Schematic of the pipeline for the update step to calculate on-the-fly the lateral and axial shifts for region tracking. Scale-bar lengths are as follows: a 150 μm, b,d 50 μm.

Extended Figure 3: Region tracking using transmission images.

a Selected maximum intensity projections of transmission images of the larval zebrafish tail over 11 hours of observation, starting at 2.5 days post fertilization. The blue dashed line indicates the tip of the vasculature at the end of the tail, growing around 100 μm over the 11 hours window of observation (blue arrows). b Despite this growth, self-driving microscopy kept the vascular region of interest in focus over the observation window. Maximum intensity projections of the corresponding high-resolution region showcase details of the outgrowth of a single vessel (white arrowhead) with subsequent anastomosis (white arrows) with a neighboring vessel, including positioning of endothelial nuclei (asterisk). The vasculature was labelled with Tg(kdrl:EGFP). c Maximum intensity projection of the entire field of view of the low-resolution acquisition at start of the time-lapse imaging, with the inset highlighting the region used for tracking. d Representative X-Y, Y-Z, X-Z maximum intensity projections used in the region tracking algorithm. Scale-bar lengths are as follows: a, c, d 150 μm; b 50 μm.

Extended Figure 4: Phagocytosis of cancer cell by macrophages.

a Schematic of our in vivo zebrafish xenograft assay to study immune cell–cancer cell interactions in situ. Firstly, human cancer cells were cultured and modified as desired, e.g., by expressing a fluorescent marker to label the cells. Secondly, cells were harvested and injected near the common cardinal vein (CCV) into the yolk of zebrafish larvae (violet arrow). Then, xenografts were imaged on our self-driving, multi-resolution microscope, and subsequent analysis allowed visualization and quantification of cell spreading, cell-cell interactions, and cell morphological changes. b Low-resolution mSPIM images captured the distribution of macrophages (Tg(mpeg1:EGFP), top: magenta, bottom: grayscale image) in the entire zebrafish larvae after xenografting U-2 OS osteosarcoma cells (pVimentin-PsmOrange label, top: white, bottom: bright white). For tissue context, zebrafish also expressed the vascular marker Tg(kdrl:Hsa.HRAS-mCherry) (top: cyan). The image highlights how macrophages clustered around sites with cancer cells (Supplementary Movie 7). c In contrast, zebrafish without xenografts (control) displayed a uniform distribution of macrophages (Tg(mpeg1:EGFP), top: magenta, bottom: grayscale image) across the entire zebrafish embryo (top: vascular marker Tg(kdrl:Has.HRAS-mCherry in cyan). d The self-driving feature of the microscope enabled high-resolution imaging of selected cancer colonies in the zebrafish tail over many hours by keeping it in focus. Frequently, we observed how zebrafish macrophages (Tg(mpeg1:EGFP), top: magenta, bottom: gray) attached to the U-2 OS cancer cells (pVimentin-PsmOrange label, top and bottom: green), and phagocytosed them (Supplementary Movie 7, 8). Scale-bar lengths are as follows: b,c 500 μm; d 50 μm.

Extended Figure 5: Workflow to obtain a high-resolution segmentation of zebrafish macrophages.

a Maximum intensity projection of a 3D volume in the caudal hematopoietic tissue visualizing zebrafish macrophages, labeled with Tg(mpeg1:EGFP)⁶², after xenografting human U-2 OS cells into the zebrafish larvae. As the cancer cells (yellow arrowhead) were labelled with a psmOrange fluorophore, there was bleed-through into the macrophage channel (GFP). Moreover, macrophages tended to cluster around cancer cells and at locations where cancer cells resided before they were phagocytosed by macrophages (yellow arrow). b 3D rendering with Fiji’s 3D viewer to display the final segmentation by the proposed workflow. c Schematic of the segmentation workflow. d We initially segmented the data with multi-Otsu thresholding and connected component labeling. This segmented individual macrophages well, e.g. in the control experiment (without xenografts) or at the beginning of the xenograft experiment. However, it could not separate touching macrophages (white arrowheads). e To clean up the segmentation, we first removed the cancer cell signal (white arrowhead) from the segmentation. f To separate clustering macrophages, we constructed 3D consensus segmentation from Cellpose-2D⁴⁵ segmentations computed in orthogonal x-y, x-z, y-z views of the original raw data. While this segmentation separated clustering macrophages, it was observed to split individual macrophages into smaller fragments (white arrows) when the macrophages show extensive cytoplasmic extensions of several tens to hundreds of microns. g We merged the connected component segmentation and the Cellpose-based segmentation by replacing connected volumes that exceeded a defined threshold with the Cellpose-based segmentation. Moreover, we merged connected volumes that were below a certain threshold with neighboring volumes. Subsequent manual curation further cleaned up the segmentation of macrophages to obtain the finalized segmentation b. Scale-bar lengths are as follows: a 30 μm.