Cutting-edge innovations in biological microscopy are increasingly blurring the line between data acquisition and data analysis. Computational microscopy and machine learning-based methods take this paradigm to an extreme, often producing raw measurements that are not human-interpretable without post-processing [1, 2, 3]. Furthermore, new data-adaptive imaging methods rely on data processing during acquisition to actively control various hardware settings of the microscope.
Testing new ideas and applying them for biological discovery is often impeded by a lack of control software that is capable of meeting demands for speed and performance, integrating new and diverse types of hardware, providing the flexibility to adapt in real time to the data being captured, and providing user-friendly programming interfaces. As a result, researchers often end up developing bespoke software that works only with specific instruments, using closed-source and/or proprietary programming languages. In addition to slowing development, this lack of a consistent software ecosystem creates barriers to reproduction and replication of new techniques. These include the increased likelihood of bugs in code that is not re-used and tested in different contexts, the burden on new users of understanding new code bases, and the extra work of disentangling high-level code that is not properly abstracted from the underlying hardware. In many fields the Python programming language, in particular the NumPy/SciPy ecosystem [4, 5], has emerged as a common framework for reproducing research results that rely upon complex, multi-layer scientific workflows in a portable, scriptable form [6] that is accessible to researchers with various levels of programming experience.
The central challenge of such a consolidated approach in microscopy is diversity of hardware used in different types of microscopes, ranging from custom-built components on optical tables to turnkey commercial systems. μManager [7, 8] is an essential tool for controlling microscopes, thanks to an extensive library of ’device adapters’ for controlling different types of hardware, from cameras to complete microscopes, from a single programming interface. Community contributions of device adapters, plugins, and scripts provide hundreds of developer-years of microscopy automation.
Despite the power of these libraries, which are written in C++ and Java, they are often difficult to integrate with the latest developments in computer vision and scientific computing, which most readily interface with NumPy. This not only increases the difficulty of developing techniques that rely on both customized data capture and data analysis, but also hinders the dissemination and adoption of these techniques by fragmenting them across multiple tools/programming languages and making them harder to understand and test in new contexts.
Numerous groups are developing microscopy control software that is written mostly or entirely in Python, including, Python Microscope [9], Python Microscopy Environment (http://www.python-microscopy.org/), and Tormenta [10] (a complete list is detailed at https://doi.org/10.5281/zenodo.4433237). However, these tools are currently limited to specific types of microscopy (e.g. super resolution) and/or support a relatively small number of hardware devices. Thus, despite the potential benefits of such a design, they face an enormous task to match μManager’s flexibility and device support.
To address these needs, we present here Pycro-Manager. Pycro-Manager is built upon a translation layer that converts Java objects, functions, and data into language-agnostic messages that are reconstituted as Python objects/functions and NumPy arrays (Fig. 1a, Supplementary materials: Architecture and design). As a result, the existing capabilities of μManager can be called as if they had been written in Python, without users having to learn Java, Java developers having to learn Python, or the abandonment of the relative strengths of either language (Supplementary materials: Background and motivation).
Figure 1:
a) Software architecture overview. (Grey) The existing parts of μManager provide generic microscope control abstracted from specific hardware, a graphical user interface (GUI), a Java plugin interface, and an acquisition engine, which automates various aspects of data collection. (Orange) Pycro-Manager enables access to these components through Python over a network-compatible transport layer, as well as a concise, high-level programming interface for acquiring data. These provide integration of data acquisition with (purple) Python libraries for hardware control, data visualization, scientific computing, etc. b) Pycro-Manager’s high-level programming interface. The data acquisition process in Pycro-Manager starts with (blue) a source of acquisition events (from either a programming or GUI). These events are passed to (green) the acquisition engine, which optimizes them to take advantage of hardware triggering where available, sends instructions to hardware, and acquires images. (M4agenta) The resulting images are then saved and displayed in the GUI. The three main abstractions of the Pycro-Manager high-level programming interface (acquisition events, acquisition hooks, and image processors) enable fine-grained control and customization of this process. c) Code examples. Code snippets for implementing (blue) acquisition events, (green) acquisition hooks, and (magenta) image processors.
In addition, Pycro-Manager provides a new library of high-level programmatic building blocks (in Python) for customizing data acquisition. These are designed to facilitate creation of complex data acquisitions with concise, readable code, while maintaining the flexibility required for customization and the ability to handle multi-TB datasets acquired at speeds over 1 GB/s (Supplementary materials: Performance).
The high-level API contains three important abstractions: acquisition events, acquisition hooks, and image processors (Fig. 1b, c). These building blocks can be used individually or combined for more complex applications. Acquisition events enable customized instructions of how to adjust hardware when acquiring images and how to index those images along arbitrary axes (e.g. time, z, etc.) for storage and display. For common microscopy workflows like timelapses, z-stacks, etc., events can be automatically generated by high-level functions (Fig. 1c). Alternatively, they can be created manually from nested Python lists and dictionaries to allow for a greater degree of customization. Acquisition hooks enable the execution of arbitrary Python code concurrently at various stages of the data acquisition process. Image processors give access to the image data as soon as it is acquired, for modification or for use in data-driven feedback loops on the acquisition process.
The combination of Jupyter notebooks (https://jupyter.org/), interactive documents that consolidate text, code, equations, and results, and the Pycro-Manager high-level application programming interface (API) provides a means to describe the full workflow of a research project, from data acquisition to analysis to results, thereby facilitating understanding, dissemination, and reproduction of new microscopy technologies. For techniques that rely heavily on computation, these notebooks can be as (or even more) valuable than their corresponding research papers.
In the supplementary information and online documentation (https://pycro-manager.readthedocs.io/en/latest/#), we provide a wealth of tutorials describing the basic features of Pycro-Manager, as well as Jupyter notebooks outlining several sample applications in microscope alignment, light sheet microscopy, integrated sample processing, computational microscopy and machine learning, and the control of microcsopes over networks.
The source code and documentation for Pycro-Manager can be found in the Supplementary materials, and future updates will be posted at: https://github.com/micro-manager/pycro-manager and: https://pycro-manager.readthedocs.io/en/latest/#
Supplementary Material
Acknowledgements
We thank S. van der Walt and K. Marchuk for helpful discussion during development.
This project was funded by Packard Fellowship and Chan Zuckerberg Biohub Investigator Awards to L.W.; STROBE: A NSF Science and Technology Center; an NSF Graduate Research Fellowship awarded to H.P.; a Berkeley Institute for Data Science/UCSF Bakar Computational Health Sciences Institute Fellowship awarded to H.P. with support from the Koret Foundation, the Gordon and Betty Moore Foundation, and the Alfred P. Sloan Foundation to the University of California, Berkeley. K.W.E., B.L., M.T. were funded by the Chan Zuckerberg Initiative and NIH grant P41GM135019
Footnotes
Competing Interests The authors declare no competing interests.
References
- [1].Boominathan Vivek, Adams Jesse K., Asif M. Salman, Avants Benjamin W., Robinson Jacob T., Baraniuk Richard G., Sankaranarayanan Aswin C., and Veeraraghavan Ashok. Lensless Imaging: A computational renaissance. IEEE Signal Processing Magazine, 33(5):23–35, 2016. [Google Scholar]
- [2].Gustafsson MGL. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. Journal of Microscopy, 198(2):82–87, 2000. [DOI] [PubMed] [Google Scholar]
- [3].Pavani Sri Rama Prasanna and Piestun Rafael. Three dimensional tracking of fluorescent microparticles using a photon-limited double-helix response system. Optics Express, 16(26):22048, 2008. [DOI] [PubMed] [Google Scholar]
- [4].Harris Charles R., Millman K. Jarrod, van der Walt Stéfan J., Gommers Ralf, Virtanen Pauli, Cournapeau David, Wieser Eric, Taylor Julian, Berg Sebastian, Smith Nathaniel J., Kern Robert, Picus Matti, Hoyer Stephan, van Kerkwijk Marten H., Brett Matthew, Haldane Allan, del Río Jaime Fernández, Wiebe Mark, Peterson Pearu, Gérard-Marchant Pierre, Sheppard Kevin, Reddy Tyler, Weckesser Warren, Abbasi Hameer, Gohlke Christoph, and Oliphant Travis E.. Array programming with NumPy. Nature, 585(7825):357–362, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Virtanen Pauli, Gommers Ralf, Oliphant Travis E., Haberland Matt, Reddy Tyler, Cournapeau David, Burovski Evgeni, Peterson Pearu, Weckesser Warren, Bright Jonathan, van der Walt Stéfan J., Brett Matthew, Wilson Joshua, Millman K. Jarrod, Mayorov Nikolay, Nelson Andrew R.J., Jones Eric, Kern Robert, Larson Eric, Carey CJ, Polat Ilhan, Feng Yu, Moore Eric W., VanderPlas Jake, Laxalde Denis, Perktold Josef, Cimrman Robert, Henriksen Ian, Quintero EA, Harris Charles R., Archibald Anne M., Ribeiro Antônio H., Pedregosa Fabian, van Mulbregt Paul, Vijaykumar Aditya, Bardelli Alessandro Pietro, Rothberg Alex, Hilboll Andreas, Kloeckner Andreas, Scopatz Anthony, Lee Antony, Rokem Ariel, Woods C. Nathan, Fulton Chad, Masson Charles, Häggström Christian, Fitzgerald Clark, Nicholson David A., Hagen David R., Pasechnik Dmitrii V., Olivetti Emanuele, Martin Eric, Wieser Eric, Silva Fabrice, Lenders Felix, Wilhelm Florian, Young G, Price Gavin A., Ingold Gert Ludwig, Allen Gregory E., Lee Gregory R., Audren Hervé, Probst Irvin, Dietrich Jörg P., Silterra Jacob, Webber James T., Slavič Janko, Nothman Joel, Buchner Johannes, Kulick Johannes, Schönberger Johannes L., de Miranda Cardoso José Vinícius, Reimer Joscha, Harrington Joseph, Rodríguez Juan Luis Cano, Nunez-Iglesias Juan, Kuczynski Justin, Tritz Kevin, Thoma Martin, Newville Matthew, Kümmerer Matthias, Bolingbroke Maximilian, Tartre Michael, Pak Mikhail, Smith Nathaniel J., Nowaczyk Nikolai, Shebanov Nikolay, Pavlyk Oleksandr, Brodtkorb Per A., Lee Perry, McGibbon Robert T., Feldbauer Roman, Lewis Sam, Tygier Sam, Sievert Scott, Vigna Sebastiano, Peterson Stefan, More Surhud, Pudlik Tadeusz, Oshima Takuya, Pingel Thomas J., Robitaille Thomas P., Spura Thomas, Jones Thouis R., Cera Tim, Leslie Tim, Zito Tiziano, Krauss Tom, Upadhyay Utkarsh, Halchenko Yaroslav O., and Vázquez-Baeza Yoshiki. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods, 17(3):261–272, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Poldrack Russell A., Gorgolewski Krzysztof J., and Varoquaux Gaël. Computational and Informatic Advances for Reproducible Data Analysis in Neuroimaging. Annual Review of Biomedical Data Science, 2(1):119–138, 7 2019. [Google Scholar]
- [7].Edelstein Arthur, Amodaj Nenad, Hoover Karl, Vale Ron, and Stuurman Nico. Computer Control of Microscopes Using μManager. Current Protocols in Molecular Biology, 92(1):1–17, 10 2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [8].Edelstein Arthur D, Tsuchida Mark a, Amodaj Nenad, Pinkard Henry, Vale Ronald D, and Stuurman Nico. Advanced methods of microscope control using Micro-Manager software. Journal of Biological Methods, 1(2):10, 11 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Miguel David, Pinto Susano, Phillips Mick A, Hall Nicholas, Langerak Mateos, Stoychev Danail, Pinto Tiago Susano, Booth Martin J, and Dobbie Ian M. Python-Microscope : high performance control of arbitrarily complex and scalable bespoke microscopes. bioRxiv, 2021. [Google Scholar]
- [10].Barabas Federico M., Masullo Luciano A., and Stefani Fernando D.. Note: Tormenta: An open source Python-powered control software for camera based optical microscopy. Review of Scientific Instruments, 87(12):126103, 12 2016. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.