A feedback-driven IoT microfluidic, electrophysiology, and imaging platform for brain organoid studies

Abstract

The analysis of tissue cultures, particularly brain organoids, takes a high degree of coordination, measurement, and monitoring. We have developed an automated research platform enabling independent devices to achieve collaborative objectives for feedback-driven cell culture studies. Unified by an Internet of Things (IoT) architecture, our approach enables continuous, communicative interactions among various sensing and actuation devices, achieving precisely timed control of in vitro biological experiments. The framework integrates microfluidics, electrophysiology, and imaging devices to maintain cerebral cortex organoids and monitor their neuronal activity. The organoids are cultured in custom, 3D-printed chambers attached to commercial microelectrode arrays for electrophysiology monitoring. Periodic feeding is achieved using programmable microfluidic pumps. We developed computer vision fluid volume estimations of aspirated media, achieving high accuracy, and used feedback to rectify deviations in microfluidic perfusion during media feeding/aspiration cycles. We validated the system with a 7-day study of mouse cerebral cortex organoids, comparing manual and automated protocols. The automated experimental samples maintained robust neural activity throughout the experiment, comparable with the control samples. The automated system enabled hourly electrophysiology recordings that revealed dramatic temporal changes in neuron firing rates not observed in once-a-day recordings.

One-Sentence Summary:

An IoT laboratory robotics system that enables touch-free feeding, imaging, and electrophysiology of brain organoids.

INTRODUCTION

Recently, advances in biological research have been greatly influenced by the development of organoids, a specialized form of 3D cell culture. Created from pluripotent stem cells, organoids are effective in vitro models in replicating the structure and progression of organ development, providing an exceptional tool for studying the complexities of biology (1). Among these, cerebral cortex organoids (hereafter “organoid”) have become particularly instrumental in providing valuable insights into brain formation (2–4), function (5, 6), and pathology (7, 8). Despite their potential, organoid experiments present significant challenges. Brain organoids require a rigorous, months-long developmental process, demanding substantial resources and meticulous care to yield valuable data on aspects of biology such as neural unit electrophysiology (9), cytoarchitecture (10), and transcriptional regulation (8).

The primary methods for generating and measuring organoids depend on media manipulations, imaging, and electrophysiological measurements (11), which are all labor- and skill-intensive, limiting the power and throughput of experiments (12). Cell culture feeding and data collection occur at intervals realistic for researchers. Furthermore, during manual feeding and data collection, the cell cultures are removed from the incubator, which provides a controlled gas, temperature, and humidity environment (13). Ideally, feeding should be aligned with the cells’ metabolic cycles, and data should be collected at intervals on par with the biological phenomenon. The disturbance incurred by leaving the incubator environment is shown to increase metabolic stress and batch-to-batch variability, potentially impacting the quality of the experiment (14), as well as increasing contamination risk. These limitations hinder the depth of insights gained from these organoid models, particularly in studies focused on dynamic neural processes and disease modeling (11).

Laboratory robotics, most often liquid handling devices (15), offer increased precision and throughput but are primarily designed for pharmaceutical screens, limiting their adoption in research labs due to high costs, large footprints, and inflexible workflows (16). Moreover, many of these systems lack the ability to seamlessly integrate new technologies as they emerge. Conversely, academic research labs are benefiting from advancements in commercial and custom-made technologies, facilitated by in-house fabrication methods like 3D printing (17, 18), which are enhancing their capacity to manipulate and measure biological systems. However, without an easy-to-integrate, device-agnostic robotic platform, researchers are constrained to manual operations, restricting the power and scope of their experiments. By outfitting devices to carry out automated jobs and relay data through communication networks, they acquire around-the-clock functionality and increased fidelity (19). The flexibility in size (number of devices per integrated system) allows researchers to optimize for the experimental design and budget. Implementing programmable feedback loops derives precision and self-optimization by dynamically adjusting to real-time data (20–22), offering a practical alternative to complex mathematical modeling for experiment control. This approach would enable more integrated, flexible automation in research settings, broadening the scope and efficiency of experiments.

Automating multiple devices to report data presents a challenge for device management and communication, necessitating flexible and efficient infrastructure. Addressing this need for an interconnected ecosystem of devices, services, and technologies is possible through designing networks using standards defined by the Internet of Things (IoT). This approach has already impacted wearables (23), agriculture (24), city infrastructure (25), security (26), and healthcare (27). It was recently proposed to expand this approach to biology research (28). Previously, each researcher built a custom device and code from scratch with unique assumptions for communication and behavior. Each device operated in solitude, lacking integration and feedback with other devices. Here, we establish a platform that addresses these challenges, combining electrophysiology, microscopy, microfluidics, and feedback control, automated and integrated through IoT technology for touch-free, in-incubator tissue research.

RESULTS

An integrated microfluidic, electrophysiology, and imaging organoid research platform

We have developed an integrated platform (Fig. 1) that automates organoid culture and data collection in individual microenvironments. While microfluidics (Fig. 1A) controls the media environment, digital microscopy captures the morphogenic features. The neural activity is recorded by local field potential measurements using complementary-metal-oxide semiconductor (CMOS) high-density microelectrode arrays (HD-MEA)(29)(Fig. 1B). The IoT cloud network brokers the communication between all devices and facilitates data storage, processing, and presentation services including an interactive webpage (Fig. 1D). Through touch-free automation, samples remain undisrupted in the incubator, increasing the consistency of images and allowing for higher frequencies of feeding and recording.

Fig. 1. Schematic diagram of the integrated feedback platform.

(A) A syringe pump and valve system dispense fresh media and aspirate conditioned media at user-defined intervals. The blue background represents 4°C refrigeration. (B) In-incubator microscopy and HD-MEA electrophysiology unit for automatic recording, media exchange, collecting morphogenic and functional dynamics data of the biological sample. Exploded view: the 3D printed gasket-sealed culture chamber. The red background represents 37°C incubation. The microfluidic culture chamber has an access lid with a polished glass rod viewing port, a gasket-sealed microfluidic module with threaded microtube fluidic fittings, and an overflow catch tray. The microfluidic culture chamber attaches to the HD-MEA, which houses the sensor for recording electrophysiology signals. (C) An in-refrigerator camera setup captures images of the aspirated conditioned media drawn from each culture and relays them through cloud-based data processing for volume estimation feedback to the syringe pump system. (D) Devices communicate over MQTT (Message Queuing Telemetry Transport) protocol and automatically upload data to the cloud, where it is stored, processed, and presented on a web page.

At user-defined intervals, conditioned media is aspirated by a syringe pump through a system of distribution valves (Fig. 1A), stored in a collection reservoir (without passing through the syringe pump vial Fig. 1C), and replaced by an equivalent volume of fresh media. Both types of media are perfused through flexible fluorinated ethylene propylene (FEP) tubing at 110 mm/s, which leads to low shear forces (30) (see Methods, Automated fluidics organoid culture).

The digital microscope (Fig. 2A) is attached using 3D-printed parts on aluminum posts. The 3D printed culture chambers integrate the microfluidics and HD-MEAs. A liquid-impermeable O-ring gasket ensures media retention inside the chamber. The well lid includes a polished glass rod submerged in the media, improving image quality and removing the effects of condensation. Alignment grooves in the glass rod lid prevent rotation and incorrect fitting. The lid exchanges gas with the incubator conditions through ventilating air ducts (Fig. 2C), similar to a cell culture well plate. The removable and re-attachable lid reduces manufacturing complexity and enables future use of other lids with applications beyond imaging.

Fig. 2. The microfluidic culture chamber.

(A) The experimental setup inside an incubator shows two microfluidic culture chambers and two conventional membrane lids. (B) 3D printed culture chamber attached to the HD-MEA. (C) Cross-section showing the fluid path. Fresh media (red) enters the chamber, filling the internal cavity. Conditioned media (blue) is drawn out of the chamber. The media level, noted by the upper black arrow (559 μL) and lower black arrow (354 μL) on the glass rod, is the ideal operating range that keeps the rod immersed in media. The biological sample is adhered to the HD-MEA in the bottom of the cavity.

Fig. 2C shows the cross-section of the culture chamber attached to the HD-MEA. The media flows in (red) and out (blue). The sinuous media path and well geometry ensure minimum disturbance to the biological sample (30). Fresh media is delivered on top of the volume present in the chamber, similar to partial media changes found in manual feeding protocols (31, 32). The ideal operating range is between 350 to 700 μL (see Supplementary Figure 1 and Supplementary Table 1 for numerical volume limits). In the case of over-aspiration, media drops to a minimum of 170 μL before aspirating air from the chamber’s headspace. The 3D-printed catch tray guards against overflow, collecting up to 1.5ml (200% of the chamber’s capacity) to protect the recording equipment from liquid damage.

Computer vision for microfluidic flow feedback

We developed a computer vision volume estimation system to monitor the accumulation of aspirated media and identify anomalies during culture feeding events. Figures 3A and 3B provide a detailed view of the setup inside a refrigerator, which includes three main components: a collection reservoir support system, an LED panel, and a camera module (see Methods, Assembled devices, and custom 3D printed components). The camera module remains on standby for image capture requests made by other IoT devices or users. Upon request, computer vision techniques are employed to estimate the media volume within the reservoirs accurately.

Fig. 3. Computer vision for volume estimation.

(A) Example of a raw image captured by the camera module. (B) In-refrigerator volume estimation setup. The CMOS camera module (the white triangle) images the conical tubes with a diffused LED backlight for even illumination. (C) Fluid segmentation: a rectangular pixel patch down the center of the conical tube; Row-wise summations of the HSV channels are used to detect the location of the meniscus. The initial liquid potion segmentation is added to the meniscus portion to yield the final segmentation. (D) Calibration graph with a fitted relationship of segmented pixel count to ground truth volume. (E) The absolute error percentage: orange dots represent the average error at selected volumes. The shaded bar represents the minimum to maximum error range.

Figure 3C shows the computer vision process (see Methods, Computer vision for fluid volume estimation) for segmenting area related to the media in the reservoir. A calibration was required to establish the relationship between the segmented area in pixels and volume in milliliters. We captured 184 images of the collection reservoirs containing volumes of media ranging from 0 to 12 mL (several pictures for each volume), with each volume confirmed by a scale, accurate to 1 μL. For each specific volume in Figure 3D, multiple points overlap and are all accounted for to calculate the polynomial regression lines. To accommodate the reservoir’s conical section (volumes <1.5 mL) and cylindrical section (volumes >1.5 mL), two distinct regressions were applied, ensuring a high degree of precision for each geometrical shape.

A Leave-One-Out cross-validation (LOO) (33) approach was employed to quantify the model’s error. This method tests the model’s accuracy and generalizability in an unbiased manner, ensuring that the calibration results in a model that performs reliably across different samples. The effectiveness of the model is assessed quantitatively with the following metrics: an average Mean Absolute Error (MAE) of 0.56% (equivalent to 27μL), an average standard deviation of errors at 0.53% (22μL), and an average Root Mean Square Error (RMSE) of 0.77% (35μL). The polynomial models exhibit R-squared values of >0.99, denoting an optimal fit of pixel area to liquid volume. Figure 3E shows the average absolute error percentage at a specific volume, with the bar indicating the error range from minimum to maximum.

IoT infrastructure creates an ecosystem of devices and cloud-based services

We built a cloud-based IoT ecosystem that enables communication between users, devices, and services to implement actions, record data, and streamline upload, storage, and analysis. All devices (here: pumps, microscopes, and microelectrode arrays) run software using the device-class Python framework (Figure 4A and Supplemental Methods). Devices operate collectively with shared core software and complementary behaviors: they can request jobs from each other, yield during sensitive operations, and ensure collaborative functions and smooth operation (Figure 4D). Devices update their shadow in the database whenever their state information changes (i.e, assigned experiment, schedule, current job and estimated completion time, and other dynamic variables) to eliminate the need for device polling. Messages (i.e., job requests) between devices and services are sent through a centralized MQTT broker via the publish/subscribe protocol. This decoupled architecture allows for independent and extensible deployment of components. Data generated by devices is immediately uploaded to an S3 object storage in a predefined structure using an experiment Universally Unique IDentifier (UUID) as the top-level key. A ‘metadata.json’ file stores experiment details, sample information, notes, and an index of the produced data. Raw data is stored separately from analyzed data under different sub-keys. Cloud jobs, which operate as shared services, process raw data from S3 and write results back to S3, reporting status via MQTT messages. To utilize the IoT ecosystem, users initiate experiments, control devices, and visualize data through a website (see Methods, Website, and screenshots in Supplemental Figure 1), with the typical user workflow in Figure 4C.

Fig. 4. Cloud-based device interactions.

(A) The device-class is a generalized state machine framework of all IoT devices. The device participates in experiments by taking in job requests (from experimenters or other devices), scheduling and executing the jobs, and producing data files that are queued and uploaded to cloud storage. (B) IoT infrastructure. Device states (pink) are saved in a database and displayed on the website user interface. Device-generated data (gray) is saved and organized in cloud storage, where it can be accessed by user interface or analysis cloud jobs. Devices send communications (purple) through a message broker and use message bridges to translate messages to analysis pipelines or text messaging applications. (C) User workflow. Devices are physically primed in accordance with experimental procedures such as sterilization. On the ‘Initialize’ webpage, an experiment is created with a unique ID (UUID) and descriptive notes (metadata). On the ‘Control’ webpage, devices are called to start working on the experiment and are given a job schedule. The ‘View’ webpage and notifications allow the user to monitor the ongoing experiment. (D) Example of inter-device communication: (1) A RECORD job request is made from the ‘Control’ panel. (2) The message broker delivers the record request to the electrophysiology recording unit. (3) The electrophysiology unit pauses all other devices to ensure a quality recording. (4) All devices receive a pause request. The pump reschedules a feed until after the pause. (5) Upon finishing the recording, the electrophysiology unit delivers a spike sorting request to commence data analysis.

Automated study of cerebral cortex organoids

The integrated research platform was used to study the effects of automation on the neuronal activity of pluripotent stem-cell-derived mouse cerebral cortex organoids. Embryonic stem cells were aggregated, patterned, and expanded to generate organoids using a previously defined differentiation protocol (34, 35). Day 32 post-aggregation, 10 organoids were plated two-per-chip directly onto five HD-MEAs. For the 7-day study, the five chips were split into groups that were fed and recorded with standard manual procedures (Controls 1–3), automatic feeding and manual recording (AF), or automatic feeding and automatic recording (AFAR). Four chips (Controls 1–2, AF, and AFAR) were imaged in the incubator every hour, by their dedicated upright digital microscope (DinoLite).

Automated microfluidic feeds were used to increase the consistency and frequency of cell culture media replacement. We removed conditioned cell supernatant from the well and dispensed the equivalent volume of fresh media for each feed cycle. The controls had 1.0 mL media replacement every 48 hours, consistent with standard protocols. AF and AFAR were placed on a protocol in which 143 μL media were replaced every 6 hours, matching the total media volume turnover across groups for the 7-day study. The schedule of automated media feeds was defined at the experiment’s launch and initiated by a timed feeding job command sent to the microfluidic pump. The fidelity of feeding was controlled through a computer vision volumetric feedback loop on the aspirated conditioned media (Fig. 3, 5A).

Fig. 5. Volume feedback.

(A) Volume estimation feedback loop. After the pump completes a microfluidic action, it requests a picture of the media collection reservoir from the camera module. The picture is passed to the cloud-based computer vision program to estimate the current volume. The results is compared with the expected volume, and a decision is made: within tolerance (green checkmark), a microfluidic volume adjustment action is needed (red “x”), or an anomaly is detected (yellow question mark). Once the estimated volume is within tolerance (green check mark), the feedback cycle ends and proceeds to the next job. If this cannot be achieved or an anomaly is detected, such as out-of-range volumes, an alert is sent to the user messaging service to request assistance. (B-D) On these graphs, the “Day” x-axis summarizes the timeline: organoids were plated on the HD-MEA on Day 32, automation started 5 days after plating and continued to day 12. Above this axis, dots mark the occurrence of microfluidic events. (B) Graphs of the Expected Volume and Estimated Volume for the automated AFAR (left) and AF (right) during a period of feedback events. Event types are marked with dots below the graph. (C) The complete view of Expected and Estimated volume traces over the 7-day study. (D) Stacked histogram pump events per day organized by type.

Conditioned media has a high protein content, contains cellular debris, and is susceptible to forming salt crystals (36, 37). In microfluidic systems, this leads to clogs, error accumulation, and failure modes (38). To overcome this, a volume estimation feedback loop was initiated each time the pump performed a job. Once media was perfused to/from a specific well, the pump sent a job request to the camera module responsible for imaging the well’s collection reservoir. The image was captured, uploaded to the cloud, its volume estimated by the computer vision Estimator, and returned to the pump for feedback interpretation. Within tolerance, the action was declared a success (marked as a green check mark in Figure 5A), and no further action was taken. Outside of tolerance, the pump scheduled itself a new job proportional to the volume discrepancy and in relation to the number of previous feedback attempts (see Methods, Feedback interpreter).

The system strives to resolve discrepancies on its own using feedback. However, in extreme cases where volume estimation returns a value outside of reason (i.e., > expectation + 2 mL) or if the feedback iteration limit is reached (i.e., > 20 attempts), the system was programmed to send an alert to a Slack messaging channel and pause. During the 7-day experiment, the system resolved errors independently, and this condition was not reached.

The automated feeding and feedback results for AF and AFAR are visually represented in Figure 5B–D. Figure 5C shows the traces of expected volume and computer vision estimated volume for AFAR (left) and AF (right) for the 7-day study (Days 5 to 12 post-plating). There was a collection reservoir change on Day 8 in which the 15 mL conical was replaced with a fresh tube. In both samples, the drop in estimated and expectation reflects the collection reservoir exchange. For AFAR (Figure 5B, left), a zoomed-in view of the feedback loop following the scheduled feeding cycle at 7:12 on Day 9 highlights feedback actions taken to remedy a volumetric discrepancy. In this instance, the volume estimation was less than expected after the feed cycle. Five consecutive aspiration jobs were carried out, and the estimated volume still remained under expectation. At the 6th iteration of feedback, a pull job was sent to the pumps, which raised the collection volume above the expected volume. In the 7th and 8th iterations of feedback, two dispense jobs were engaged to supplement the well for the over-aspiration. In a similar case, for AF (Figure 5B, right), a total of 6 iterations of feedback were engaged to bring the estimated volume into tolerance with the expected volume; however, in this example, no dispense jobs were required. Figure 5D shows histograms of the sum of pump events per day by subcategory. Each feeding cycle (four per day) was scheduled, and all other events occurred through feedback.

High-frequency HD-MEA recordings do not disrupt neuronal activity

To interrogate organoid neuronal activity, extracellular field potentials were measured using 26,400 electrode HD-MEAs, which can record up to 1,020 electrodes simultaneously. We conducted daily activity scans to monitor neural activity. Heat maps derived from the first and final activity scans for each sample are presented in Figure 6B, with organoid body outlines for reference. To optimize electrode coverage, we generated specific configuration files for electrode selection based on the regions with the highest activity, which remained constant for four of the five chips. In one case (AF, Day 32+6), we adapted the configuration due to the emergence of a new high-signal area on the second day of recording. These maps allowed continuous electrode measurements for 10-minute intervals optimized for frequent, long-term sampling.

Fig. 6. Electrophysiology analysis of the 7-day cerebral cortex organoid study.

(A) Digital microscope images of each organoid sample (B) Boundaries of each organoid were outlined and overlaid for orientation on their activity scans from the first recording day (top) and last recording day (bottom). The samples are labeled underneath, with the color legend consistent throughout the figure. (C) Spike sorting analysis on 10-minute recording/day detected neural units and spiking frequency. Daily Violin plots. The black dashed lines represent the first (lowest) quartile, median, and third (highest) quartile. The firing rate of each neuron is represented along the y-axis, and the total number of neural units is labeled above each plot. (D) Plots of the number of units (top) detected on the daily activity scans, and their median firing rates (bottom). (E) The AFAR had an additional 23 (hourly) recordings per day (translucent dots/trendline) beyond the daily recordings (opaque dots/trendline). The trend of units detected over time (top graph) and median firing rates (bottom graph) was different with the inclusion of hourly recordings compared to daily recordings. Vertical lavender lines represent occurrences of automated feeds. (F) A dial schematic of the 6-hour automated cycle of recording, feeding, and imaging for the AFAR sample (top). The AFAR’s hourly recordings were organized in bins of ‘hours since feed,’ and the neuron firing rates are shown in violin plots (bottom).

Manual recordings involved an experimenter placing each HD-MEA on the recording unit and initiating 10-minute recordings via software. In contrast, the hourly recordings (AFAR) featured the HD-MEA remaining on the headstage while automated software handled the entire process, from power management to data uploading. AFAR amassed 161 recordings, totaling 26.8 hours (560 GB) of electrophysiology data. Conversely, all manually recorded samples (Controls 1–3 and AF) accumulated 7 recordings, amounting to 1.2 hours of electrophysiology data.

From these data, we analyzed the effects of our automated microfluidic, imaging, and recording system on the neuronal activity of the brain organoids housed therein. Imaging of the chips from above (Figure 6A) allowed us to align the body of the organoid with neural units detected (Figure 6B). In some instances, such as in Control 1, neurite outgrowths were evident in the images and activity scans.

The initial activity scan was used to assign samples for the experiment. In the first activity scan, AFAR and Control 1 exhibited the highest activity, followed by AF, Control 2, and lastly, Control 3. This specific categorization of samples was designed to address potential biases introduced by varying levels of starting activity. To ensure robust analysis, each chip was treated as an average of the two organoids. Chips demonstrated similar trajectories in the number of units and firing rate frequency over the 7-day experiment, irrespective of feeding or recording schedules (Figure 6C–D). Figure 6C shows the distribution of neuronal firing rates as a violin plot for each chip over seven days, labeled with the number of neurons detected in that recording. The number of detected units had a positive correlation over the 7-day study for all samples except Control 1, which also presented the highest number of units (Fig 6D, top). Similarly, the median firing rate was positively correlated with time for all samples except for Control 2 (Fig 6D, bottom). The fully integrated system, encompassing automated feeding, imaging, and recording, behaved like the controls, indicating minimal bias or distortion of data as a result of increased HD-MEA recording frequency.

High-frequency HD-MEA recordings reveal dynamic neuronal activity states in organoids

The hourly recording condition (AFAR) unveiled dynamic and transient states, not apparent with single daily recordings (Figure 6E). While the trendlines for hourly and daily recordings (for both units and firing rates) have similar upward trends, hourly recordings show more prominent oscillations around the trendline not captured by the single daily recordings. Median firing rates fluctuated as much as 3-fold over the course of a day, sometimes within a few hours of each other.

To determine if feeding cycles influenced this activity, we aligned recordings to a six-hour ‘time since feed’ cycle (Fig. 6F, top) and examined potential differences in the number of units and firing frequency. Figure 6F presents the composite graph of aggregated neuronal firing rates comprising 26 feeding cycles with all 161 recordings binned with respect to their time since feeding. Our results showed no trend across all superimposed feeding cycles: neither the number of units nor firing rate changed in relation to feeding cycles. The oscillations presented in Figure 6E do not align with feeding and appear to be a temporal dynamic intrinsic to the network. In summary, these findings underscore the robustness and reliability of our feedback-driven microfluidic platform for brain organoid studies.

DISCUSSION

We integrated the custom-built and commercially available instruments using the IoT device-class framework. This system ensures faster, consistent, and always available operations, increasing the overall throughput while allowing the researchers to focus on formulating questions and data interpretation. Running on a distributed IoT network offers dual benefits. Using a local MQTT broker ensures reliable performance even during internet outages. Cloud integration enables global collaboration across distant labs for shared or complementary research. This setup enhances both the continuity of individual experiments and the integration of worldwide scientific efforts. The reduction of human intervention enabled by the microfluidic feeding system reduces the risk of contamination and other human-introduced mishaps. This is particularly valuable in months-long organoid experiments, where the accumulation of small variations in sample handling can accumulate to generate large differences between experimental batches.

Our system has the capacity to increase the frequency of media collections, morphology assessments, and electrophysiological measurements beyond what is feasible under standard conditions. Feedback in experimental setups becomes essential for maintaining target operating zones in the absence of direct physical observation. In this paper we demonstrated one method of feedback, which was needed to maintain a consistent volume in the organoid growth chamber. During our 7-day run, the system achieved this feedback autonomously and did not experience break-downs or need to use the messaging alert system to overcome anomalies.

How frequently should data be collected? By providing the ability to record as frequently as desired, our system can uncover the optimal frequency for meaningful data capture for rare but significant events. Neural processes unfold with remarkable complexity and variability, yet for practical reasons, many experimental paradigms are limited to daily recordings at most (5, 39–41). Researchers can thereby miss crucial events that occur between observation points. From our results, the high-frequency recordings presented trends not captured in the once-a-day sampling. Hourly recordings like those conducted here can enable the detection of patterns, oscillations, and interactions that may be overlooked in sporadic recordings (42, 43). These benefits are particularly relevant to researchers wishing to study phenomena with a more immediate timescale, such as neuroplasticity (44). Additionally, many neurodevelopmental disorders have been hypothesized to be ‘connectopathies,’ characterized by abnormal connectivity (45). Frequent recordings can provide a nuanced view of the underlying changes in activity patterns during neurodevelopment, contributing to a better understanding of the etiology of neurodevelopmental disorders.

In the future, devices can use the flexibility of MQTT messaging to allow the creation of additional feedback loops to control the experiment. The computer vision techniques we applied to volume estimation could be extended to further applications such as colorimetric and absorbance sensing using the same setup to interrogate biochemical properties of the media. Such measurements could provide a more detailed and accurate analysis of organoid cultures and can lead to a more nuanced understanding of their behavior and responses to different stimuli.

The more the number and different kinds of measurements taken in an experiment, the more automation becomes essential to coordinate and manage the different modalities. The use of 3D printing technology enhances this flexibility, allowing for the seamless combination of multiple systems, such as the integration of our custom media exchange setup with the commercial HD-MEA and portable microscope. We foresee the integration of various sensory data and feedback mechanisms to analyze cell culture conditions. Our platform’s consistency and reliability are ideal for comparative studies involving organoids of different genotypes or subjected to various pharmacological manipulations. This capacity to facilitate direct comparisons between diverse experimental conditions holds promise for advancing our understanding of neurodevelopment and neurodevelopmental disorders.

MATERIALS AND METHODS

Cell culture protocols and organoid plating on HD-MEA, which occurred prior to the experiment, are described in Supplementary Materials.

Assembled devices and custom 3D-printed components

The Bill of Materials listing components and costs are provided in Supplementary Material Section 1. STL files for 3D printing are provided in PrintedAccessories.zip.

Automated fluidics organoid culture

The automated microfluidic pump system builds on previous work (30). The microfluidic system was configured to support two chips (AF and AFAR) and their respective collection reservoirs (right and left) were imaged by the camera setup.

Fresh cell culture media is kept at 4°C refrigeration and accessed by the pump through flexible FEP tubing routed into a benchtop refrigerator and to a media bottled with a reagent delivery cap (Cole-Parmer VapLock). Fresh media is kept refrigerated to increase longevity and may be replaced during experimentation. To dispense, the syringe pump and distribution valves draw fresh media into the syringe vial and distribute the programmed volume into flexible FEP tubing routed through an access port in the incubator. Here, the media is heated in incubator conditions prior to being delivered to the organoid inside the culture chamber. To keep media dispenses available on demand, a preheated 450 μL reserve (59% of the chamber’s volumetric capacity) of fresh media remains idle in the FEP tubing so that upon dispensing, 37°C media is delivered to the well in less than 10 seconds. The FEP tubing is interfaced with the fluidic module with threaded ferrule lock and nut fittings (Cole-Parmer VapLock). Outflow from the fluidic module is drawn away with FEP tubing routed out of the incubator and into a refrigerator containing the collection reservoirs and computer vision camera setup.

For the collection reservoirs, we selected 15 mL Polyethylene Terephthalate (PET) conical tubes (430055, Corning) for high optical clarity, ease of replacement, and durability in downstream analysis and cold storage. To enhance visibility for computer vision imaging, we removed the factory-printed writing area on the conical PET tubes using generic, multipurpose tape. Flexible FEP tubing was interfaced with the PET tubes using a rubber cork plug (#6448K95, McMaster-Carr). The cork was pierced with 8-gauge steel needles that served as supportive conduits for the tubing. The tubing was secured inside the needle with glue (Loctite 4011) to create a hermetic seal at the point of interface. The steel encasing of the needles ensures a smooth, unobstructed flow within the flexible FEP tubes. Each collection reservoir had two flexible FEP tubes: one for media coming from the fluidic module and one for pressurized operation connected to the syringe pump. This ensured that spent media never entered the syringe (only air). The air is expelled into a filtered (Millipore AA 0.22 μm syringe filter) safety container (not shown in Figure 1).

For the 7-day study described here, we designed for equivalent media exchange across conditions. The Controls 1–3 were fed 4 times at 1 mL per feed, totaling 4 mL of replacement media. AF and AFAR were fed 28 times at 143 μL per feed, totaling 4 mL of replacement media over the week. Summing the scheduled feeds and feedback adjustments, a single collection reservoir could store conditioned media for 2–3 weeks.

Priming the experiment

On the 5th day on chip (Day 32+5), membrane lids for two HD-MEAs (AF and AFAR) were replaced with microfluidic culture chambers. During the replacement process, all media was aspirated from the HD-MEA’s well with a P-1000 pipette. The microfluidic catch tray, followed by the culture chamber, was inserted inside the well, and 750μL of the original media was added back to the microfluidic culture chamber. Excess media was discarded. The glass rod lid was placed on top.

Flexible FEP tubes (idling with DI water) were flushed with 1.0 mL of fresh media. After priming the lines with media, the AF/AFAR chips were connected with fluidic fittings wrapped with Teflon tape. An initial aspiration leveled the media to the target fluidic operating range. The collection reservoirs were replaced with new empty conical tubes.

Running the experiment

During the experiment, the media was exchanged using a feed cycle operation consisting of an aspiration followed by fresh media dispense. Here, we performed 143μL aspirations and dispenses every 6 hours to match 1.0mL feeds every two days in the manual feeding controls. Feedback performed additional aspiration, dispense, and pull actions in addition to the basic feed cycle schedule to ensure the system stayed within normative error ranges. See section Feedback interpreter.

Teardown of the experiment

Once the experiment was stopped, chips were disconnected from the flexible FEP tubes by unscrewing the fittings. The flexible FEP tubes with fittings were sterilized in a flask containing disinfectant (Cydex) and covered with aluminum foil. The collection reservoirs with the experiment’s conditioned media were disconnected and taken for analysis. New collection reservoirs were inserted for the cleaning cycle. The pump ran a cleaning solution (Cydex) through the entire internal cavity for 1 hour to disinfect the system. Following disinfection, DI water and dry, sterile air were profused through the system for 12+ hours (overnight) to clear the disinfectant. The flexible FEP tubes were left resting with DI water until the next experiment.

Computer vision for fluid volume estimation

The computer vision setup, located inside a 4°C refrigerator, included a support for the collection reservoir, a camera module, and an LED panel positioned behind the conical tubes. The LED panel served as backlighting to enhance the clarity and contrast of the images. The reservoir support was a two-plex 3D-printed system capable of multiplexity to tailor alternate experiments (see Section Assembled devices and custom 3D printed component). The camera and LED panel were both controlled by a Raspberry Pi.

To generate the calibration dataset, the camera module captured images of media in the collection reservoirs at select volumes over the entire range of the tube (0–12 mL), totaling 184 images. The volumes associated with each image were measured using a high-precision scale (30029077, Mettler Toledo). This approach enabled a correlation between the visual representation of media in the images and its actual volume (see Results).

To ensure image quality, our study introduced two checks to validate the integrity of the captured images: Lighting and blurriness. A region of interest (ROI) was designated within the panel’s area to verify the lighting conditions by checking that the average RGB color values each exceeded a minimum threshold of 20 out of 255. Blurriness was assessed by computing the variance of the Laplacian for the image, with a necessary threshold of 50 to pass. The thresholds were empirically determined using the calibration dataset.

Figure 3C illustrates the methodology applied to fluid segmentation, outlined in the Results section. The process begins with capturing an RGB image of the collection reservoirs that are fixed in place by the setup. To facilitate better segmentation and feature extraction, the RGB image is transformed into the HSV (Hue, Saturation, and Value) color space. A summation of the HSV values row-wise from the bottom to the top of the collection reservoir results in three distinctive profiles that allow differentiation between the liquid and background. Each profile, as illustrated in Figure 3C, presents a vertex at the boundary. A row value was established by averaging three rows identified in each HSV channel: an abrupt rise in the curve for the Hue channel, the absolute maximum for the Saturation channel, and the absolute minimum for the Value channel. From the average row value, the first segmentation was created. Everything below this row was set as white pixels, and everything above it was set as black pixels. A local evaluation around the average row was made to incorporate the meniscus in this segmentation. Utilizing HSV thresholds, the meniscus was accurately characterized and incorporated into the initial segmentation, culminating in the final image segmentation, in which white pixels represented the liquid portion.

The estimated volume was given by Equation 1, where x represents the segmented area in pixels, and the resultant volume is in microliters. Two different curves are used to account for the conical section for volumes under 1.5 mL (and pixel area less than 4446) and the cylindrical section for larger volumes.

$$V(x)=\left\{\begin{array}{ll}5.09\times {10}^{-9}{x}^3+2.39\times {10}^{-5}{x}^2+0.13x-1.28& \text{if}x<4446\text{pixels}\\ 2.60\times {10}^{-11}{x}^3+5.38\times {10}^{-7}{x}^2+0.62x-1288.37& x\ge 4446\text{pixels}\end{array}\right.$$ (1)

The image segmentation and estimation based on the mathematical model (Equation 1) is carried out by a software program named the “Estimator.” The process initiates with a feeding cycle, which triggers a picture request. Upon receiving the image of the collection reservoir, the “Estimator” analyzes the image and returns the estimated value of the fluid volume. The volume is relayed to the next module for feedback interpretation within the pump system (see Feedback interpreter).

Feedback interpreter

Computer vision volume estimations were compared to expectation values based on the sum total of pump action jobs. The feedback interpreter classified estimations into four categories: within tolerance, out-of-tolerance, anomaly, and tube change. Tolerance was a static volume selected at the start of the experiment. For the results shown here, the tolerance was 143 μL. If the volume estimation received was within the expectation value +/− the tolerance, the pump action was determined a success, and feedback ceased. If the volume estimation received was beyond the expectation value +/− the tolerance and also less than +/− 2000 μL, another cycle of feedback was engaged. When the volume was less than expected, for the first 5 iterations of feedback, aspiration jobs were sent to the pump with the difference of expectation and estimation. For iterations 6 to 19, pull jobs were sent to the pump, increasing by one for each subsequent interaction. A “pull” is a 1000 μL aspiration at 10x the standard syringe speed (applying a 1.1 × 103 mm/s flow rate), shown to generate the force required to break through variably high resistance in the conditioned media. At 20 iterations, the feedback interpreter requests manual intervention via the messaging application, and all further pump actions are suspended until the issue is resolved. When the volume was more than expected, dispense jobs were sent to the pump with the difference of expectation and estimation. Dispense actions were limited to 200 μL per action and 2 iterations of feedback in total to prevent overflow. A volume estimation that was 2000 μL or more above the expectation value was determined as an anomaly and requested manual intervention via the messaging application, and all further pump actions were suspended until the issue was resolved. The feedback interpreter automatically detected collection reservoir tube changes when the volume estimation dropped by 2000 μL or more compared to the previous estimation and the total volume present was estimated as less than 2000 μL.

Computer vision for In-incubator Organoid Culture imaging

In-incubator imaging

A 5MP digital microscope (AM7115MZTL, Dino-Lite) was placed over the organoid culture on the HD-MEA using holders described in Assembled devices and custom 3D printed components. Imaging was performed from the top through a glass rod (quartz drawn rod, 5mm ± 0.20mm dia × 15mm ± 0.20mm long, UQG Optics) (in AF/AFAR chips) or through a membrane lid (in control chips). The image is captured using reflected light from a built-in brightfield LED source next to the camera sensor. The 3D printed alignment trays handle most of the chip placement, with initial minor focal plane adjustment required. The microscope remains shut off until the software triggers it to turn on the lights and take a photo.

Image Segmentation for Organoid

In the process of image segmentation for organoid analysis, the first step involves applying an image calibration to correct any distortion. This procedure requires identifying four source points and four destination points. The former were manually selected from the distorted image. The latter were calculated based on an initial pixel (left corner of the HD-MEA), the size of the electrodes, and the spacing between them, both in millimeter units. This relationship between pixels and millimeters was established by using known dimensions of the HD-MEA border and electrode pitch in the image.

The organoid segmentation within the rectified image was accomplished using the Segment Anything Model (SAM) (46). This model combines neural network architectures, allowing for precise and versatile image segmentation without the need for specialized training on new images. The segmented image is analyzed to detect variations in pixel intensity, which signify the presence of organoid contours. Both images with the organoid’s contour and electrode grid are overlayed. Each electrode area is checked for the presence of the organoid’s border. When a border is detected within an electrode’s bounds, that particular electrode is marked prominently on the grid image to signify contact with the organoid (see Figure 6B). The step-by-step illustration of the analysis process is shown in Supplemental Figure 5.

Plotting & alignment to neural activity data

Electrode numbers as (x,y) position were plotted in matplotlib and exported as SVG. The SVG aligns over other plots, such as activity heatmaps, which follow the same x:3580 by y:2100 axis dimensions. Since electrophysiology plots use the electrode coordinate system with the same (x,y) positions, the image segmentation grid and neural activity plots are aligned on the same coordinate system.

Measuring neural activity

Extracellular field potential recordings were performed using CMOS-based high-density microelectrode arrays (HD-MEAs) (MaxOne, Maxwell Biosystems). Each HD-MEA contains 26,400 recording electrodes within a sensing area of 3.85 mm × 2.1 mm (each electrode has a diameter of 7.5 μm, spaced 17.5 μm apart center-to-center). A subset of up to 1020 electrodes (defined spatially by a configuration) can be selected for simultaneous recording (47). Across one configuration, neuronal activity in microvolts was sampled over time at 20kHz and stored in HDF5 file format.

The experiment involved each chip’s daily activity scans and recordings (described below). Each chip underwent an activity scan and subsequent recording every day, consistently conducted within the same one-hour time window. All chips shared the same recording unit and were recorded one at a time. For the AFAR condition, beyond the daily recordings and activity scans, the chip remained on the HD-MEA for automated hourly recordings.

The gain was set to 1024x with a 1 Hz high pass filter for both activity scans and recordings. The recording was set up to save 5 RMS thresholded spike times as well as all raw voltage data for downstream analysis and plotting.

All neural activity measurements were performed inside the incubator at 36.5°C, 5% CO₂.

Internet of Things (IoT)

Cloud Infrastructure

The cloud infrastructure, including S3, MQTT messaging, and cloud processing within the IoT system, has been previously described (28). Additionally, we added a database service and defined a consistent organizational structure for MQTT messages and topics across devices and cloud jobs.

We use a combination of self-hosted services running on a server, and large data storage and analysis are performed on the National Research Platform (NRP) cloud compute cluster (48). The devices are integrated with these cloud services:

• S3 cloud data storage: file storage using S3 object store, hosted on NRP cloud.

• Database: Strapi database stores device states, is self-hosted on our server, and is backed up to S3.

• MQTT messaging: EMQX MQTT broker, self-hosted on the server, and a Python messaging library (braingeneers.iot.broker) utilized by all software endpoints to send and receive messages from the broker.

• Cloud jobs/processing: utilizes a Kubernetes cluster on NRP and launches jobs. Employs software modularized by Docker containers and orchestrated by Kubernetes.

• User interfaces: features a website and integration with messaging apps (e.g., Slack) for interaction with devices, self-hosted on the server.

All custom software functionalities run in Docker containers and operate in a microservice architecture: specialized to a specific task and interface with minimal dependencies. A reverse proxy shields all web services from direct exposure to the internet. For example, webpages are configured through a reverse NGINX proxy, which not only assigns a specific domain to each service but also handles SSL and authentication services.

Security

Devices initiate communication with the server and can be locked down to incoming traffic. Devices take MQTT commands in a specific format and are limited to the set of their defined commands, making them robust to command injection attacks. Accessing all cloud services requires authentication with user/device credentials. All web, MQTT messages, database, and S3 storage operations are encrypted. Access to the user interface website is restricted through the proxy with a login authentication step. On the server side, all web-based microservices are secured through an NGINX proxy. The proxy allows web-based services to be relatively untrusted by providing security (https, authentication, internet visible network listener) and keeping all other web-based services on an internal docker network inaccessible from the internet. This simplifies security for services that will change often and be written by programmers with minimal security training.

Data and materials availability:

All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.