Segmentation-free analysis of live-cell imaging data reveals how T cell modifications influence cancer cell aggregation dynamics

Abstract

Live-cell imaging (LCI) of modified T cells co-cultured with cancer cells is commonly used to quantify T cell anti-cancer function. Videos captured by LCI show complex multi-cell behavioral phenotypes that go beyond simple cancer cell fluorescence measurements. Here, we develop an unsupervised analysis workflow to characterize LCI data generated using the Incucyte imaging platform. Unlike most LCI analyses, we avoid cell segmentation due to the low spatiotemporal resolution of the LCI videos and high levels of cell-cell contact. Instead, we develop methods that identify global aggregation patterns and local cellular keypoints to characterize the multicellular interactions that determine cancer cell sensitivity to, or escape from, T cell surveillance. We demonstrate our segmentation-free live-cell behavioral analysis (SF-LCBA) methods on TCR T cells from four donors with varying proportions of cells with a beneficial RASA2 knockout and effector-to-target initial concentrations in a co-culture with A375 melanoma cells. We find that different T cell modifications affect the spatiotemporal dynamics of multicellular aggregate formation. In particular, we show that fewer and smaller cancer cell aggregates form with higher proportions of T cells with the beneficial RASA2 gene knockout in co-culture with A375 melanoma cells. Our SF-LCBA method identifies, characterizes, and tracks cellular aggregate formation in datasets that are unsuitable for cell segmentation and tracking, opening the door to more therapeutically-relevant measurements of modified T cell therapy cell behavioral phenotypes from LCI data.

Introduction

T cell cancer immunotherapies have in the past decade gained recognition for their therapeutic potential. The 2017 success of anti-CD19 chimeric antigen receptor (CAR) T cell therapy and subsequent FDA approval was a bell weather for this class of therapies for liquid cancers^1,2. Since then, research to improve T cell therapies and adapt them for use in solid tumors has accelerated^3,4. T cell receptor (TCR) based immunotherapies have been particularly promising; an anti-MAGE-A4 TCR T cell therapy for metastatic synovial sarcoma became the first FDA-approved T cell therapy for solid tumors in 2024⁵. TCR T cells differ from CAR T cells in that they bind to cancer-specific proteins presented on the cell surface by human leukocyte antigens (HLAs) instead of cancer cell surface proteins such as CD19. This advantage of TCR T cells increases the number of potential cancer-specific antigens that can be targeted with a T cell therapy.

Another active area of T cell cancer immunotherapy research is to improve anti-cancer activity of the modified T cell. Endogenous regulatory pathways and immunosuppressive tumor microenvironments can both trigger T cell dysfunction, which is characterized by reduced T cell proliferation and cytotoxic signaling. Targeted gene editing with CRISPR-Cas9 has the potential to overcome these limitations and enhance therapeutic efficacy^6–10. Several studies have performed genome-wide CRISPR knockout screens under in vitro^11–14 and in vivo^14–17 immunosuppressive conditions to identify genetic perturbations that reduce T cell therapeutic functions. These modified CAR and TCR T cells hold immense promise to improve anti-cancer cellular immunotherapy.

While genome-wide CRISPR screens successfully nominate promising T cell perturbations in a high-throughput fashion, these candidates are validated using lower throughput experiments. In particular, fluorescence-based killing assays remain the gold-standard in vitro functional assay for quantifying anti-cancer activity^11,13,17. In this assay, modified T cells are co-cultured with cancer cells that express a fluorescent reporter. The co-cultures are then subject to multi-channel live-cell imaging (LCI) to quantify the amount of fluorescent signal coming from cancer cells. T cell modifications associated with the largest reduction in fluorescent intensity across the well plate are determined to have the greatest anti-cancer potential, prompting follow-up in vivo studies. However, there is often a discrepancy between in vitro and in vivo cancer cell killing efficacy¹⁷.

To address the discrepancy between in vivo and in vitro efficacy and to better understand the mechanisms for increased anti-cancer potential, computational research from our group and others has been trying to extract additional features from co-culture live-cell imaging (LCI) data beyond cancer cell killing efficacy^18–23. Research in this area has been made possible through critical advancements from the deep learning computer vision field to develop models that segment individual cells from phase channel images^24–31. The resulting cell segmentation masks enable measurements of cell morphology, such as quantifying cell size and circularity, which are known to vary between cell types and phenotypic states^18,32–34. In LCI datasets with high temporal resolution, cell tracking models are often run downstream of segmentation, which allows for estimation of cell motility, cell division events, and changes in cell morphology over time^35–39.

Despite the promise of cell segmentation and tracking to provide informative features, these pre-trained deep learning models are often incompatible with existing cancer and T cell co-culture LCI data. Co-culture experiments test cancer cells and T cell products across many different experimental conditions and technical replicates. Given that automated LCI microscopes only have one camera per 96-well plate, there is a tradeoff between number of wells imaged and the spatiotemporal resolution of the images. The spatiotemporal resolution of co-culture experiment LCI data is commonly and 2 h between frames^11,13,17. This is too low to perform cell segmentation and tracking for T cells, which require (≥ 10x magnification) and ≤ 10 minutes between frames, respectively^18,36,38. Additionally, even when the spatiotemporal resolution criteria are satisfied, these models struggle to segment and track cells belonging to cancer cell aggregates— commonly found in co-culture LCI experiments—since co-cultures of interacting cell types are not used as training data for these tracking models^35–39. Therefore, there is a critical need to measure biologically-relevant features of co-culture LCI data, such as T cell activity and cancer cell aggregation, without the use of cell segmentation and tracking methods.

In this paper, we analyzed LCI data of RASA2 knockout (RASA2KO) TCR T cells co-cultured with the A375 melanoma cell line¹³ to characterize cancer cell aggregation across experimental conditions. Each LCI co-culture varied in the titration of RASA2KO mutant to wildtype T cells, the effector to target (E:T) ratio, and the primary T cell donor. Since the cancer cells expressed red fluorescent protein (RFP), each well had phase and red channel images taken once every 2 h for a total of 144 h. As expected based on known RASA2 biology, analysis of total RFP+ cancer cell area over time in prior work¹³ revealed that higher E:T ratios and higher proportion of RASA2KO T cells led to the greatest inhibition of cancer cell growth. However, previous analysis of this data failed to investigate any other properties as the low spatiotemporal imaging resolution prevented cell segmentation and tracking.

To circumvent these limitations, we formulated a segmentation-free live-cell behavioral analysis (SF-LCBA) framework to understand the multicellular interactions associated with the observed cancer cell aggregation processes. The two main components of SF-LCBA are the calculation of cancer cell spatial entropy across each well⁴⁰, and using scale-invariant feature transform (SIFT) to identify and characterize local keypoints in the phase channel images as one of five single or aggregate cell types⁴¹. SIFT extracted hundreds to thousands of keypoints per image—where each keypoint represents a single- or multi-cellular instance and is captured using a length-128 descriptor vector. We clustered SIFT descriptor vectors and interpreted the meaning of each SIFT cluster through visual inspection and comparison of RFP intensities. Statistical testing was applied to both the SIFT clusters and the spatial entropy scores to show their relationships with experimental conditions and temporal dynamics. Overall, our analysis demonstrated that cancer cell growth and, importantly, aggregation is most effectively inhibited in co-cultures with high starting doses of RASA2KO TCR T cells (high E:T ratio, high RASA2KO titration). We believe the SF-LCBA framework has broad potential to improve analyses of co-culture LCI data beyond total cancer cell counts, stratifying distinct conditions where total RFP expression would otherwise be equal in a wide variety of imaging scenarios, ultimately leading to better cellular cancer immunotherapies.

Methods

The proof-of-concept data for our segmentation-free live-cell behavioral analyses (SF-LCBA) methods come from LCI taken with the Incucyte platform from a study of TCR T cells in four donor backgrounds¹³. These TCR T cells with the RASA2 knockout are co-cultured with cells from the A375 melanoma cell line (Supplementary Fig. S1a). These LCI data allow us to showcase the association of our metrics with donor, E:T ratio, and RASA2KO mutant-to-wildtype TCR T cell titration. The SF-LCBA workflow that we develop may also be applied to LCI experiments from different experimental systems as long as there are phase contrast and fluorescence channels available.

Isolation of primary T cells from healthy donors

Leukopaks from deidentified healthy donors with Institutional Review Board-approved consent forms and protocols were purchased from StemCell Technologies (200-0092). For screens, residuals from leukoreduction chambers after Trima Apheresis from deidentified healthy donors with Institutional Review Board-approved consent forms and protocols were purchased from Vitalant (formerly known as Blood Centers of the Pacific). Primary Human T cells were isolated using EasySep Human T cell isolation kit (17951) according to the manufacturer’s protocol using the EasySep magnets. The cells were seeded in appropriate culture vessels and activated with Immunocult (Stem Cell Technologies, 10971) at 12.5 μl per ml. Cells were kept in culture at a 106 cells per ml density throughout, and cultured with IL-2 at 50 IU per ml (unless otherwise specified). Cells were cultured in X-Vivo-15 medium, which was supplemented with 5% fetal calf serum, 50 μM 2-mercaptoethanol, and 10 mM N-acetyl-l-cysteine. Peripheral blood mononuclear cells (PBMCs) were frozen down at cells per vial using Bambanker (Bulldog Bio) serum-free cell freezing medium.

CRISPR KO in primary human T cells using Cas9–RNP electroporation

48h after T cell isolation and stimulation, Cas9–sgRNA–RNP electroporation was performed using the Amaxa P3 Primary Cell 96-well 4D-Nucleofector Kit (Lonza, V4SP-3960). Lyophilized crRNA and tracrRNAs (Dharmacon) were resuspended in nuclease-free duplex buffer (IDT 1072570) at a concentration of 160 μM. Unless otherwise stated, control-edited T cells were targeted with the AAVS1 sequence GGGCCACTAGGGACAGGAT, and RASA2-edited T cells were targeted with the RASA2-targeting sequence AGATATCACACATTACAGTG. The crRNAs and tRNAs were complexed at 1:1 v/v ratio for 30 min at C. sgRNAs were mixed with Cas9 (Stock 40 μM) at a 1:1 v/v ratio and incubated at C for 15 min to form the RNP complex. T cells were counted, resuspended in P3 buffer at per 20 μl, mixed with 3 μl of RNPs and added to a 96-well electroporation plate. The cells were electroporated using the EH115 protocol and immediately recovered by adding 80 μl T cell medium (X-Vivo-15, Lonza) at C for 15 min. Once recovered, cells were transferred to appropriate culture vessels in X-Vivo-15 medium with IL-2 at 50 IU per ml.

Lentiviral production and T cell transduction of TCR

Lenti-X 293T cell line (Takara Bio 632180) cells were seeded at 18–20 million cells per 15 cm dish pre-coated with poly-l-lysine 16 h before transfection and cultured in DMEM + 5% FBS + 1% penicillin-streptomycin. Cells were transfected with the sgRNA transfer plasmids and second-generation lentiviral packaging plasmids, pMD2.G (Addgene 12259) and psPAX2 (Addgene 12260) using the Lipofectamine 3000 transfection reagent per manufacturer’s protocol (L3000001). 6 h after transfection, the transfection medium was replaced with DMEM + 5% FBS + 1% penicillin-streptomycin containing viral boost reagent at 500× per the manufacturer’s instructions (Alstem VB100). 24- and 48-h viral supernatants were collected and spun down at 300g for 10 min at C to remove the cell debris. The lentiviral particles were concentrated using Alstem precipitation solution (Alstem VC100) and stored overnight at C. The virus was centrifuged at 1,500g for 30 min at C and resuspended at 100× of the original volume in ice cold PBS and stored at C until further use. For T cell transduction, 24 h after TCR stimulation, the concentrated lentivirus was directly added to T cells at 1:25 v/v ratio with X-Vivo-15 medium and gently mixed by tilting.

Live-cell imaging of cancer versus T cell co-cultures

NY-ESO1-reactive 1G4 TCR T cells were co-cultured with pre-plated red fluorescent protein positive (RFP+) A375 tumor cells (ATCC, CRL-1619) in 96 well flat-bottom plates. Every well was seeded with target A375 cells and co-cultured in X-VIVO-15 plus supplements, 100 IU IL-2 per ml and 1X Glucose (Gibco). The plates were imaged every 2 h for 144 h using the IncuCyte Zoom live-cell imaging platform (Essen Bioscience). Across two channels images were acquired. These channels consisted of a phase (brightfield) channel and an RFP (red) channel. The plates were laid out in a matrix as follows: Across 12 columns, T cell titration was serially diluted from 100% RASA2KO T cells to 3.1% RASA2KO T cells and 96.9% wild type T cells. These titrations were duplicated across four technical replicates. Across 14 rows, effector T cell to target cancer (A375 cancer cells) cell ratio (E:T ratio) is progressively diluted from 2.8284 by a factor of 1.14142 to 0.2. These dilutions are duplicated (across four technical replicates) for all E:T ratios except for the lowest one. Each combination of E:T ratio and RASA2KO percentage is replicated four times, except for the lowest E:T ratio, which only has two technical replicates. There are four plates, the T cells of each plate across mutational status are derived from different donors. Each experiment has approximately 65 images; there is variation in time series length of ± 3 frames between donor replicates. The spatial resolution per image frame is 4.975 μm/pixel. To ensure time frames exactly matched across conditions, we restricted our analysis of this dataset to the first 64 time frames (128 h) from 5 RASA2KO titrations (100%, 50%, 25%, 12.5%, 6.25%), 5 effector to target (E:T) ratios (2.83, 2.00, 1.41, 1.00, 0.71), 4 donors, and 2 technical replicates; however, these filters are performed for simplicity of presentation and are not essential to the analyses.

Thresholding on RFP intensity to obtain cancer cell masks

Red fluorescent protein (RFP) was primarily localized to the nucleus and cytoplasm of A375 cancer cells; however, RFP was also detected at low levels throughout the well due to background auto-fluorescence. We therefore created a binary cancer cell mask M(i, j) by thresholding the red channel’s intensity values at ≥ 3.5. We quantified the binary cancer cell mask as follows:

1

where represents the red channel’s intensity value at pixel coordinates (i, j). M(i, j) is the resulting binary mask where M(i,j)=1 indicates RFP+ pixels, and M(i,j)=0 indicates RFP- pixels.

Cancer cell mask summary statistics

The two summary statistics we computed from cancer cell masks were the cancer cell area and spatial entropy. We approximated cancer cell area for a region of interest (ROI) R using the ROI’s fraction of RFP+ pixels as follows:

2

where:

• M(i, j) is the binary cancer cell mask,

• R is the set of pixel coordinates within the ROI, and

• |R| is the total number of pixels in the ROI.

To characterize spatial organization of RFP+ pixels within R, we computed four entropy-based statistics that capture heterogeneity at different spatial scales and definitions of spatial dependence.

Quadrat entropy. The primary spatial entropy statistic was based on dividing each image into quadrats (subregions) and computing the Shannon entropy in RFP intensity between quadrats⁴⁰. The ROI R was partitioned into fixed 100 × 100 pixel subregions . For each subregion , we computed the proportion of all RFP+ pixels in R that lie within that subregion:

3

where is the number of RFP+ pixels in subregion and is the number of RFP+ pixels in the entire ROI. Quadrat entropy applies the Shannon entropy formula to the spatial distribution of RFP+ pixels across coarse blocks of size s:

4

This measure is high when RFP+ pixels are evenly spread across quadrats and low when they are spatially concentrated.

Patch-size entropy. To quantify heterogeneity in the sizes of contiguous cancer-cell aggregates, we computed the entropy of connected components (patches)⁴². We defined an adjacency graph over RFP+ pixels, with nodes corresponding to coordinates (i, j) such that M(i,j)=1, and edges connecting all 8-neighbor pixel pairs:

5

The set of patches was defined as the set of 8-connected components of this graph. For each patch with size , we defined:

6

Patch-size entropy was computed using Shannon entropy:

7

This statistic increases with the diversity of aggregate sizes, distinguishing images with many similarly sized clusters from those dominated by one or a few large patches.

GLCM entropy. We computed a gray-level co-occurrence matrix (GLCM) over the binary mask M(i, j) using a fixed pixel offset⁴³. Let g(a, b) denote the normalized frequency of observing value a adjacent to value b, for . GLCM entropy was defined as:

8

This statistic captures local spatial disorder based on pairwise adjacency patterns.

Multiscale entropy. The quadrat entropy is computed across a set of different subregion sizes and then averaged across several values. The formula for this entropy statistic is:

9

We used values of for benchmarking on simulated data.

For all summary statistics, the ROI R could be either the entire image or a sliding or fixed window within an image.

Simulation of synthetic cancer cell masks

To benchmark spatial summary statistics and evaluate their sensitivity to spatial heterogeneity, we generated synthetic binary cancer cell masks M(i, j) of size 1000 × 1000 pixels using two complementary simulation frameworks: a point-cloud simulator that produces circular multi-cell clusters with tunable entropy, and a block-based simulator designed to directly manipulate coarse-scale entropy. In both cases, simulated masks had the same dimensions as the experimental regions of interest (ROIs) R. Point-cloud simulator. The point-cloud simulator generated masks by first sampling cluster centers and then sampling individual cells as spatial perturbations around each center. Let denote the set of cluster centers, where the number of clusters depends on a tunable entropy parameter . Cluster centers were sampled independently and uniformly over the spatial domain:

10

where L=1000 is the image side length. For each cluster center , we generated k(α) cells, where the number of cells per cluster scales inversely with α. Each cell location was sampled from an isotropic Gaussian distribution centered at : 

where the standard deviation is defined as , which increases linearly with α and controls the spatial dispersion of cells around each cluster center. The binary mask M(i,j) was then constructed by rendering each sampled cell as a filled disk of fixed radius r:

11

where is the indicator function. The parameter α jointly controls the number of clusters, the number of cells per cluster, and the spatial spread of cells, such that low α yields dense, highly aggregated clusters, while high α produces more dispersed, spatially uniform patterns.  Block-based simulator. The block-based simulator directly controls spatial heterogeneity at the scale of coarse subregions. The ROI R was partitioned into a grid of blocks , each of size 100 × 100 pixels and B is the number of blocks. To model variability in cell density across blocks, we first sampled a vector of block-level probabilities from a symmetric Dirichlet distribution:

12

where is a vector of length B and each element is the same value ζ > 0 which represents a global concentration parameter that controls the degree of spatial heterogeneity. Small values of ζ yield sparse, highly uneven allocations (low entropy), while large values produce more uniform block-level densities (high entropy).

Given a fixed total number of cells N, we then allocated cells to blocks by sampling block-specific counts from a multinomial distribution: Within each block , cell centers were sampled independently and uniformly:

13

The binary mask M(i,j) was constructed by rendering each sampled cell as a filled disk of fixed radius r: Much like α in the point-cloud simulator, lower values of ζ produce masks with clustered spatial distribution and high values produce uniform distributions.

Multiple realizations. For each simulator, we generated n=50 independent images at values of for the point cloud simulator and in the block simulator. This allowed consistent comparison of summary statistics across controlled spatial regimes and assessment of their stability with respect to stochastic variability.

Applying SIFT to LCI phase images

The scale-invariant feature transform (SIFT)⁴¹ was applied to all phase (brightfield) image frames in the LCI dataset using the scikit-image implementation with default settings of 8 octaves and 3 scales⁴⁴. The SIFT algorithm consists of two steps: (1) identifying keypoints and (2) computing descriptors. Each SIFT keypoint is a set of image coordinates (i, j) that represents a local maximum or minimum of a Gaussian pyramid transform of the image. SIFT identifies keypoints across three discrete scales σ, and 8 octaves o, whereby the radius r of each keypoint is defined by ⁴⁵. SIFT also captures the orientation angle of the gradient around every keypoint. Keypoint angle is used to standardize the orientation of the calculated descriptor histograms, insuring rotational invariance. After identifying all keypoints in an image, SIFT computes a length-128 descriptor vector for each keypoint. This vector is a flattened histogram of gradients calculated around the keypoint coordinates on the untransformed image.

Subsampling SIFT descriptor matrix

Since SIFT keypoints correspond to local maxima and minima in the feature space of a Gaussian pyramid transform of the image, we obtained keypoints (and descriptors) for each phase image n instead of a constant number of keypoints per image. To limit memory usage, we randomly subsampled 10% of each image’s keypoints when constructing the original descriptor matrix of shape D × 128, where . We further randomly subsampled the descriptor matrix down to D=50,000 keypoints when performing statistical tests and visualization to avoid overpowered p-values and improve interpretation, respectively.

K-means clustering of SIFT descriptors

We used K-means to cluster the D × 128 SIFT descriptor matrix. We used the scikit-learn implementation of K-means clustering⁴⁴ with default parameters and random seed of 0 unless otherwise stated. K-means clustering results were compared to agglomerative Ward clustering and HDBSCAN clustering using the adjusted Rand index (ARI).

To select the optimal number of clusters K, we performed a parameter sweep ranging from K=3 to K=10. We computed the silhouette score and sum of squared distances (SSD) to cluster center for each set of clustering results K. The silhouette score measures the quality of clustering by evaluating how similar a (key)point i is to its own cluster compared to other clusters. It first computes the the mean intra-cluster distance for each keypoint :

14

where:

• is the cluster to which keypoint i belongs,

• is the number of keypoints in cluster ,

• and are the length-128 descriptor vectors for the ith and jth keypoints, respectively,

• is the Euclidean distance between descriptor vectors and .

The second step of computing the silhouette score is to find the nearest cluster distance , which represents the average distance of keypoint i to all keypoints in the nearest neighboring cluster:

15

where:

• C is any cluster other than ,

• |C| is the number of keypoints in cluster C.

Using both and for all keypoints, we compute the mean silhouette coefficient over all keypoints in the SIFT matrix D:

16

Note that we only computed silhouette score on D=50,000 keypoints since computing silhouette scores has runtime complexity of . Silhouette scores range from -1 to 1, where 1 represents perfectly clustered data.

The other metric we computed was the sum of squared distances (SSD) from each keypoint to the cluster center, also known as the within-cluster sum of squares (WCCS) metric:

17

where

• D is the total number of keypoints,

• is the length-128 descriptor vector for the ith keypoint,

• is the cluster to which keypoint i belongs,

• is the length-128 centroid of cluster ,

• is the Euclidean distance between and cluster center .

We selected the optimal number of clusters by applying the elbow heuristic to WCCS scores and jointly examining the silhouette scores.

Principal components analysis

We used principal component analysis (PCA) to visualize the 128-length SIFT descriptors in a lower dimensional space. To do so, we ran PCA on the full D × 128 descriptor matrix with D=3,389,740 with 30 principal components (PCs). Only the first two 2 PCs (PC1 and PC2) were used in the SIFT PCA embedding plots, although we studied latter PCs for additional correlations with relevant held-out covariates without success.

Mixed effects linear models

We related several image-level summary statistics (cancer cell area, cancer cell spatial heterogeneity) to experimental conditions of the image (RASA2KO titration, E:T ratio, donor, technical replicate, time) using mixed effects linear models implemented in statsmodels⁴⁶. Each model performed linear regression where one summary statistic was the response variable. Across models, RASA2KO titration, E:T ratio, technical replicate, and time were the fixed effect covariates, while donor was the random effect variable. Coefficients were estimated for each fixed effect variable, a y-intercept term, and all possible combinations of 2-, 3-, and 4-variable interaction terms. Donor and technical replicates were treated as categorical variables while all other variables were continuous. Categorical variables were one-hot encoded. Since we only have two technical replicates, we showed the term for replicate ID of 1 in tables and figures while replicate ID of 0 was absorbed into the intercept term. We reported the coefficients and p-values for fixed effect estimates in these models, where the p-values represent the Wald test probability of accepting the null hypothesis that a true coefficient is equal to 0. All p-values from these tests were corrected with Benjamini-Hochberg false discovery rate (FDR) correction.

SIFT cluster enrichment testing

When testing whether certain SIFT clusters were enriched or depleted for certain experimental conditions, we had to treat categorical experimental covariates differently from continuous ones.

The categorical covariates were donor ID and technical replicate ID. We iterated through each cluster ID (0, 1, … , 6) and variable value (donor 0, donor 1, …, replicate 1). For each unique combination of cluster ID and variable value, we created a 2× 2 contingency table comparing that value’s presence inside the cluster versus outside of it. The contingency tables were then used to perform χ-squared tests and compute the odds ratios, yielding p-values and effect sizes, respectively.

The continuous covariates were time, RASA2KO titration, and E:T ratio. We iterated through each cluster ID and covariate type, performing a Kruskal-Wallis test to obtain a p-value and computing the Cohen’s d (standardized mean difference) effect size. Cohen’s d measures how much the mean covariate value differs between the points in a cluster and the points not in a cluster. Therefore, positive Cohen’s d means that the cluster had more points at large values for that covariate (e.g., enriched in later time frames) or fewer points at low values for that covariate (e.g., depleted in earlier time frames).

We used the subsampled set of SIFT keypoints (D=50,000) when performing these statistical tests to limit the number of p-values that were smaller than the machine float precision of . All p-values from these tests were corrected with Benjamini-Hochberg FDR correction.

Results

In this paper, we examine how segmentation-free analysis of live-cell imaging (LCI) data can be used to characterize cellular behavior in cancer and T cell co-culture assays. We focused on a dataset of co-cultured RASA2KO TCR T cells and A375 cancer cells¹³. This dataset contains Incucyte images collected every two hours from wells that varied in their RASA2KO titration, effector to target (E:T) ratio, T cell donor, and technical replicate number (see Methods). We constructed a fully-observed image tensor (with no missing images) from this dataset of the following shape:

• 5 RASA2KO titrations (100%, 50%, 25%, 12.5%, 6.25%),

• 5 E:T ratios (2.83, 2.00, 1.41, 1.00, 0.71),

• 4 donors,

• 2 technical replicates,

• and 64 time frames.

Two channels—phase and red—were collected for each image as the A375 cancer cells expressed red fluorescent protein (RFP; Fig. 1a). The red channel was thresholded to create a binary cancer cell mask where individual pixels were labeled RFP positive (RFP+) or RFP negative (RFP-). Qualitative assessment of cancer cell masks across E:T ratio, RASA2KO titration, and time suggested that cancer cells divided more and survived longer at low E:T ratios and low RASA2KO titrations (Fig. 1b; Supplementary Figs. S1–S3).

Fig. 1.

Cancer cell area and spatial entropy are computed via RFP masks. (a) Overview of the Carnevale et al. imaging dataset that was analyzed in this paper. 5D tensor represents the experimental metadata associated with each image. Red and brightfiend (phase) channels were captured in each image. (b) RFP+ masks over time at varying RASA2KO titrations and E:T ratios. Time is noted in hours and increases from left to right. RASA2KO titrations and E:T ratios vary between rows. (c, d) Spatial entropy of the cancer cell (RFP+) mask over time, stratified by (c) RASA2KO titration and (d) E:T ratios. Shaded areas represent 95% confidence intervals. (e) Mixed linear model results for predicting RFP spatial entropy using the covariate values for RASA2KO titration, E:T ratio, time, and technical replicate. Technical replicate ID is modeled as a categorical variable while the other covariates are modeled as continuous variables. The model is conditioned on donor ID. The intercept term is excluded here but shown in Table 2. Boxes represent mean values of each coefficient and whiskers represent +/- standard error. Interaction terms between multiple covariates are noted using colons in the x-tick labels. Adjusted p-values (Benjamini-Hochberg correction; false discovery rates) are annotated above each term if where the null hypothesis is that the term has no effect on entropy (coefficient = 0).

Spatial entropy distinguishes cancer cell aggregation patterns across experiments

The most commonly reported metric for T cell killing assays is total cancer cell area over time^11,13,17. Indeed, we were able to distinguish the different experimental conditions when examining the number of RFP+ pixels in each image over time (Supplementary Fig. S4a,b); however, solely looking at cancer cell area did not provide insights into the spatial distribution of cancer cells, as images with identical numbers of RFP+ pixels can have wildly different spatial organization. Measuring cancer cell spatial organization over time would improve the quantitative characterization of T cell killing assays, enabling researchers to link different T cell modifications with this new cellular response measurement.

We were interested in developing a metric to quantify different aggregation dynamics in live-cell imaging experiments. To do this, we first decided to calculate spatial entropy⁴⁰, which measures how evenly cancer cells are dispersed across the well. High entropy represents uniform cellular distributions and low entropy represents uneven distributions in two-dimensional (2D) space. Using simulated data, we demonstrated that our spatial entropy statistic, along with many others, accurately estimate true entropy of binary RFP masks (Supplementary Fig. S5, Methods). When analyzing experimental data, we found that the spatial entropy across RASA2KO titration (Fig. 1c) and E:T ratio (Fig. 1d) showed distinctive time-indexed curves in which entropy decreased over time, indicating that co-culture with T cells causes cancer cells to become less uniformly distributed across the well. These curves also revealed that wells with high E:T ratio or high RASA2KO titration underwent the largest decrease in spatial entropy over time. This result suggests that, while high E:T ratio and high RASA2KO titration led to the largest reduction in cancer cell area, certain areas of each well had more dramatic reduction of cancer cells than others. We also investigated the joint distribution of RFP+ area and spatial entropy across time, E:T ratio, and RASA2KO titration as we hypothesized that the presence of resistant cancer cell clones might be marked by low spatial entropy and high RFP+ area; however, we observed a consistent tradeoff between area and entropy across all covariates (Supplementary Fig. S4d-f), underscoring their limitation at detecting clone-level dynamics.

Linear mixed model relates experimental conditions to spatial metrics

We next wanted to quantify the relative contribution of each experimental condition to the spatial metrics of cancer cell area and entropy. We fit two mixed-effects linear models, one that predicts area and one that predicts entropy, using covariates of RASA2KO titration, E:T ratio, time, and technical replicate ID (see Methods). When modeling cancer cell area, we observed area increased over time across all conditions (; Supplementary Fig. S4c; Table 1). While the terms for E:T ratio and RASA2KO were not significant on their own, the interaction terms with time revealed that cancer cell area increased faster at low RASA2KO titrations () and low E:T ratios ().

Table 1.

General linear model regression results for cancer cell mask total area. RASA2KO titration, E:T ratio, technical replicate, and time were the fixed effect variables while donor was the random effect variable. Donor and technical replicate are modeled as categorical variables and all others are continuous. Interaction terms are denoted with colons between each variable name. Adjusted p-values represent the Wald test probability of accepting the null hypothesis that a true coefficient is 0 after Benjamini-Hochberg FDR correction. All decimals are rounded to three digits.

Model:	MixedLM	Dependent variable:	RFP area
No. Observations:	13446	Method:	ML
No. Groups:	4	Scale:	2609194480.4813
Min. group size:	3304	Log-Likelihood:	−164861.9334
Max. group size:	3400	Converged:	Yes
Mean group size:	3361.5

Term	Coef.	Std.Err.	z		[0.025	0.975]
Intercept	558818.236	283967.720	1.968		2251.733	1115384.740
Replicate	− 243968.339	152008.088	− 1.605		− 541898.716	53962.038
RASA2KO	− 214.891	2081.989	− 0.103		− 4295.515	3865.734
RASA2KO:replicate	5222.319	2944.377	1.774		− 548.554	10993.191
ET	− 12715.277	61043.281	− 0.208		− 132357.910	106927.355
ET:replicate	120463.687	86328.190	1.395		− 48736.456	289663.831
RASA2KO:ET	842.521	1182.508	0.712		− 1475.152	3160.193
RASA2KO:ET:replicate	− 1822.966	1672.318	− 1.090		− 5100.649	1454.717
Time	92824.668	1399.962	66.305		90080.794	95568.543
Time:replicate	− 220.291	1979.837	− 0.111		− 4100.700	3660.118
RASA2KO:time	− 400.075	27.106	− 14.760		− 453.202	− 346.948
RASA2KO:time:replicate	− 229.076	38.334	− 5.976		− 304.209	− 153.944
ET:time	− 27176.125	794.942	− 34.186		− 28734.182	− 25618.067
ET:time:replicate	− 3135.829	1124.216	− 2.789		− 5339.252	− 932.406
RASA2KO:ET:time	84.041	15.396	5.459		53.866	114.216
RASA2KO:ET:time:replicate	101.764	21.773	4.674		59.089	144.438

When modeling spatial entropy, the linear mixed-effects model revealed that cancer cell entropy was lower at high RASA2KO titrations (), high E:T ratios (), and later time frames (; Fig. 1c–e; Table 2). The fact that the RASA2KO and E:T solo terms showed greater significance than their interaction terms with time ( and , respectively) suggests that these experimental T cell parameters influence spatial entropy across all frames—not only in a time-dependent manner. Another significant term in the spatial entropy model was the three-way interaction term between RASA2KO, E:T, and time (), finding that cellular entropy was lowest when all three terms were high together. This three-way interaction also highlights that the changes to entropy across time are accelerated when RASA2KO and E:T ratio are both high, meaning the aggregates form faster in a non-additive sense than when one or the other of the experimental conditions are high. Together, these cancer cell mask statistics show that higher doses of RASA2KO T cells lead to less cancer cell growth and less uniform spatial configurations of cancer cells.

Table 2.

General linear model regression results for cancer cell mask spatial entropy. RASA2KO titration, E:T ratio, technical replicate, and time were the fixed effect variables while donor was the random effect variable. Donor and technical replicate are modeled as categorical variables and all others are continuous. Interaction terms are denoted with colons between each variable name. Adjusted p-values represent the Wald test probability of accepting the null hypothesis that a true coefficient is 0 after Benjamini-Hochberg FDR correction. All decimals are rounded to three digits.

Model:	MixedLM	Dependent variable:	RFP entropy
No. Observations:	13446	Method:	ML
No. Groups:	4	Scale:	0.0204
Min. group size:	3304	Log-Likelihood:	7062.6310
Max. group size:	3400	Converged:	Yes
Mean group size:	3361.5

Term	Coef.	Std.Err.	z		[0.025	0.975]
Intercept	4.564	0.032	143.553		4.502	4.626
Replicate	− 0.007	0.017	− 0.415		− 0.041	0.027
RASA2KO	− 0.001	0.000	− 4.256		− 0.001	− 0.001
RASA2KO:replicate	0.000	0.000	0.497		− 0.000	0.001
ET	− 0.082	0.007	− 11.948		− 0.096	− 0.069
ET:replicate	− 0.005	0.010	− 0.508		− 0.024	0.014
RASA2KO:ET	0.000	0.000	0.066		− 0.000	0.000
RASA2:ET:replicate	0.000	0.000	0.686		− 0.000	0.001
Time	− 0.001	0.000	− 3.497		− 0.002	− 0.000
Time:replicate	0.002	0.000	3.457		0.001	0.002
RASA2KO:time	0.000	0.000	1.978		0.000	0.000
RASA2KO:time:replicate	− 0.000	0.000	− 5.838		− 0.000	− 0.000
ET:time	− 0.001	0.000	− 5.170		− 0.001	− 0.001
ET:time:replicate	− 0.001	0.000	− 5.810		− 0.002	− 0.001
RASA2KO:ET:time	− 0.000	0.000	− 12.846		− 0.000	− 0.000
RASA2KO:ET:time:replicate	0.000	0.000	3.829		0.000	0.000

SIFT feature detection and descriptor vectors from phase images

While computing spatial statistics on the RFP cell masks showed that high concentrations of RASA2KO T cells led to reduction in both cancer cell area and spatial entropy, these statistics failed to quantify the number of cancer cells that appear as singlets or in multi-cellular aggregates between different RASA2KO T cell concentrations. If cancer aggregates were larger and more frequent at high RASA2KO T cell concentrations, this would indicate that aggregates mostly consist of debris clumps or damaged cancer cells that converge towards the same region after sublethal cytotoxicity⁴⁷. Conversely, if cancer aggregates were smaller and less frequent at high RASA2KO T cell concentrations, it would indicate that they mostly consist of proliferating cancer cells that evade T cell surveillance^48,49.

Additionally, computing spatial statistics on cell masks can only be applied to cells that express a fluorescent reporter. In the context of the RASA2KO LCI dataset¹³, we cannot examine how T cells grow, move, and change shape over time since they are not fluorescently labeled. Being limited to analysis of fluorescent channels would be particularly problematic for next-generation co-culture experiments where the number of immune cell subtypes (e.g., monocytes, B cells, dendritic cells) exceeds the number of available fluorescent channels⁵⁰ or patient-derived cancer cells lack a fluorescent reporter⁵¹. For analysis of the raw phase images, one could pass all images through an autoencoder^52,53 to create a low-dimensional embedding; however, each point in the resulting embedding would represent an entire image, and not each of the hundreds of cells within that image, preventing more granular, localized downstream analysis of cellular behavior (Supplementary Fig. S6). We therefore wished to explore analyses that could extract single- and multi-cell instances such as cancer cell singlets, cancer cell aggregates, and cancer-T cell interactions within individual phase images without using segmentation methods. Such analyses would allow us to tease apart the nuanced cellular interactions and aggregation dynamics that are unobservable from image-level statistics.

We chose to use the scale-invariant feature transform (SIFT) algorithm⁴¹ to analyze the phase images. For each image in our dataset N, SIFT extracts keypoints per image and computes a length-128 descriptor vector that encodes image properties surrounding each keypoint (Fig. 2a, left). We subsampled the number of keypoints and concatenated keypoints across all images, producing a SIFT descriptor matrix with D (total number of descriptors) rows and 128 columns (Fig. 2a, right; see Methods for full details). In our dataset, there were 2590 +/- 1528 (mean +/- stdev) keypoints per image (Fig. 2b). We found a positive correlation when cross-referencing the number of keypoints per image to the total cancer cell area obtained from the corresponding RFP mask (Pearson ; Fig. 2c). Such a high correlation between number of keypoints and approximate cell count suggests that SIFT correctly detects cellular keypoints under default parameters. Through a SIFT parameter grid search and image downsampling experiment, we observed that the number of detected keypoints increased with the number of scales parameter and decreased at lower image resolutions, indicating that SIFT should not be used for cases where image resolution approaches the size of cell diameters (Supplementary Text; Supplementary Figs. S7,S8). The original SIFT descriptor matrix had D ≈ 35 million descriptors across N=13,446 images; however, for computational expediency, we only included 10% of each image’s descriptors in the final SIFT descriptor matrix. The analyses proceeds identically when all of the keypoints are included.

Fig. 2.

SIFT extracts relevant keypoints and computes their corresponding descriptor vectors from phase images. (a) Schematic diagram for extracting matrix of SIFT descriptors from all phase images. Each image is passed through SIFT independently to obtain SIFT descriptors for each image n. We then concatenate the SIFT descriptor matrices across all N images to obtain a final SIFT descriptor matrix of shape D × 128. We commonly only use a subset of all rows in this SIFT descriptor matrix to speed up computational analysis. (b) Histogram of the total number of SIFT keypoints extracted per image () prior to any subsampling. Distribution is shown for all N=13446 images analyzed. (c) Relationship between total number of SIFT keypoints and the cancer cell RFP+ area of each image. Relationship is shown for all N=13446 images analyzed. The linear regression line of best fit is annotated with a dashed gray line alongside the Pearson correlation r value. (d) K-means clustering silhouette scores for different numbers of clusters K. K-means clustering was performed on all D=3,389,740 SIFT descriptors but the silhouette scores were computed on a D=50,000 subset. (e) Sum of squared distances from each point to its K-means cluster center for different number of clusters K. K-means clustering and the sum of squared distance calculation were both performed on all D=3,389,740 SIFT descriptors. (f) PCA embedding of D=50,000 SIFT descriptors colored by their K-means cluster ID. PCA was run on all D=3,389,740 SIFT descriptors prior to subsetting to D=50,000. Two representative ROI phase images are shown for each cluster. The circle in each ROI represents the radius of each SIFT keypoint and the line represents its relative orientation. ROIs are annotated with the SIFT keypoint’s x-y coordinate (in m units). along with the associated image metadata.

We next wanted to cluster the SIFT descriptor matrix to group keypoints by their similarity to one another. We performed K-means clustering of the SIFT descriptor matrix, performing a parameter sweep over the number of clusters K to find the optimal cluster size. For each clustering K, we examined the silhouette score, the sum of squared distances to cluster centers, and the cluster IDs annotated onto an embedding of the first two principal components (Fig. 2d,e; Supplementary Fig. S9a-h). The first two principal components cumulatively explained only 21% of the total variance in the SIFT descriptor matrix (Supplementary Fig. S9i-n), indicating that the clusters capture additional complexity beyond what is shown in the PC1 versus PC2 SIFT embeddings. K-means clustering was robust to random seed initialization and superior to Ward and HDBSCAN clustering algorithms (Supplementary Fig. S10). Evaluating these components led us to select K-means clustering with K=7 for downstream analysis (see Methods).

SIFT cluster characterization

We next define and give a label to the image properties that united keypoints within a SIFT cluster. We started by visually inspecting SIFT keypoint 30 × 30 pixel regions of interest (ROIs), both with and without RFP masks, and investigated 2000 × 2000 μm ROIs containing dozens of keypoints per ROI (Fig. 2f; Supplementary Figs. S11,S12). We made the following observations about each cluster from visual inspection:

• Clusters 2 and 3 were “edge” clusters as they were located near the polystyrene edges of each well;

• Clusters 5 and 6 were “aggregate” clusters with high cell density;

• Clusters 1 and 4 were “singlet” clusters as the ROIs were commonly centered around a single, often polar, cell rather than a group of cells;

• Cluster 0 was a mixture of “aggregates” and “singlets” with a subset of this cluster representing well-wide polystyrene reflections.

We next performed a series of analyses to quantitatively support these cluster definitions.

We first wished to quantitatively support our observation that keypoints in clusters 2, 3, and a subset of 0 captured properties of the polystyrene well rather than the cells inside. To support the observation that clusters 2 and 3 were truly at well edges, we computed the distance of each keypoint to the center of the well. We found that distance to the center of the well was substantially larger in edge effect clusters compared to all other clusters (independent t-test , t=-227.3) with the standard deviation in distance being 3.8-fold smaller in edge clusters (stdev of 37 in edge clusters, 142 in other clusters; Fig. 3a; Supplementary Fig. S13a). To determine which keypoints represented well-wide polystyrene reflections, we examined the radius (r) of each keypoint. We found that 3.4% of all keypoints belonging to cluster 0 had an ROI radius >250 pixels (72 / 2109) while the largest ROI radius outside of cluster 0 was 24 pixels, indicating that the well-wide polystyrene reflections were exclusive to a subset of cluster 0. The number, location, and size of cellular keypoints were only marginally influenced by cropping out well edges prior to running SIFT (Supplementary Text; Supplementary Fig. S13b,c), highlighting that these non-cellular image properties do not impair the detection of cellular keypoints.

Fig. 3.

Image properties of SIFT clusters. (a) Distance of SIFT keypoints from the center of the well and (b) fraction of RFP+ pixels in keypoint ROIs, grouped by cluster ID and cluster group. Edges consist of cluster IDs 2 & 3, singlets of cluster IDs 1 & 4, and aggregates of cluster IDs 5 & 6. (c–e) Relationship between each image’s RFP spatial entropy and the fraction of keypoints in that image that are (c) edges, (d) singlets, and (e) aggregates. Points are colored by the E:T ratio of the experiment. (f–h) Influence of E:T ratio on the correlation between RFP spatial entropy and fraction of aggregate keypoints. Scatter plots are the same as (c-e) except subset to experiments with (f) E:T ratio of 0.7071 and (h) E:T ratio of 2.8284 and coloring points by RASA2KO titration. (g) Pearson correlation between entropy and fraction of aggregate keypoints across all E:T ratios. All scatter plots are annotated with a linear regression fit, the Pearson r, and a p-value representing a Wald test where the null hypothesis is no correlation. P-values between boxes represent independent t-tests. P-values within each cluster group compare its two cluster IDs against one another. Other p-values compare the edges group versus non-edges group in panel (a) and compare adjacent cluster groups to one another in panel (b). Boxes represent the quartiles of the dataset while the whiskers extend up to 1.5× the interquartile range. Outlier points beyond the whiskers are drawn with small circles.

We next wished to more clearly define the singlet and aggregate clusters from one another. We hypothesized that aggregate clusters should have higher cancer cell density than singlet clusters. We therefore computed the fraction of RFP+ pixels within each keypoint’s ROI and compared these values between clusters. We found that aggregate clusters 5 and 6 had a substantially higher fraction of RFP+ pixels than singlet clusters 1 and 4 (independent t-test ; t=38.5), confirming our qualitative observation that there is higher cancer cell density in aggregate keypoints compared to singlet keypoints (Fig. 3b; Supplementary Fig. S13d).

We also compared the RFP+ fraction of ROIs within cluster types. Between the two singlet clusters, cluster 4 had a higher fraction of RFP+ pixels than cluster 1 (independent t-test ; t=22.1; Fig. 3b). Upon further exploration of keypoint ROIs, we believe this relationship reflects that cluster 1 has more T cell singlets than cluster 4 and that cluster 4 singlets contain neighboring cancer cells more often than cluster 1. Between the two aggregate clusters, cluster 6 had a substantially higher fraction of RFP+ pixels than cluster 5 (independent t-test ; t=17.7; Fig. 3b). This relationship suggests that cluster 6 aggregates have a higher cancer cell density than cluster 5. A summary of all cluster labels is shown in Table 3.

Table 3.

Summary of SIFT clusters. Cellular clusters can consist of either singlets (single cells) or aggregates (multiple cells). Edge effect clusters were located at the edges of each well and therefore were not centered on particular cells. Cancer cell abundance was determined based on fraction of RFP+ pixels in ROIs.

SIFT cluster ID	Name	Group	Group type	Cancer cell abundance
0	Well reflections	Mixed	Mixed	Low-medium
1	Cancer-poor singlets	Singlet	Cellular	Medium
2	Edge 1	Edge effect	Well	Low
3	Edge 2	Edge effect	Well	Low
4	Cancer-rich singlets	Singlet	Cellular	Medium-high
5	Cancer-poor aggregates	Aggregate	Cellular	Medium-high
6	Cancer-rich aggregates	Aggregate	Cellular	High

Next, we investigated whether the RFP spatial entropy statistic could support our interpretations of these clusters. To do this, we compared the fraction of keypoints belonging to each cluster category (singlets, aggregates, edges) to the RFP spatial entropy of each image. Across all experiments, we found there to be lower entropy at high edge fractions (Pearson r=-0.66, ) and higher entropy at high singlet fractions (Pearson r=0.59, ; Fig. 3c,d). These associations make intuitive sense as singlets must be spread out from one another, increasing entropy, and edge keypoints are most prevalent when there is a lot of empty space in the well, resulting in lower entropy. The correlation between entropy and fraction of aggregate keypoints was modest across all experiments (Pearson r=0.06, ) and insignificant at low E:T ratios (Pearson r=-0.03, ); however, this correlation was confounded by E:T ratio (Fig. 3e-g). When examining experiments with the highest E:T ratios, there was lower entropy at high aggregate fractions (Pearson r=-0.20, ; Fig. 3h). Only having significantly negative aggregate-entropy correlation at high E:T ratios makes biological sense because, while low E:T ratio experiments should have uniform cancer cell growth, leading to high entropy and a high fraction of aggregate keypoints, only high E:T ratio experiments have enough T cell mediated killing of cancer cell singlets to produce an uneven, low entropy spatial distribution of cancer cells while maintaining a high fraction of aggregate keypoints.

Fig. 4.

Enrichment of SIFT clusters between experimental conditions. (a-f) PCA embedding of D=50,000 SIFT descriptors colored by (a) K-means cluster ID, (b) technical replicate ID, (c) donor ID, (d) time (hour), (e) RASA2KO titrations, and (f) E:T ratios. (g-h) Volcano plots testing for enrichment and depletion of covariate values within clusters. The y-axis is adjusted p-values and the x-axis is effect size. Cluster × covariate value combinations are annotated with the color of the covariate and those with are also annotated with the cluster ID and covariate value. Benjamini-Hochberg correction is applied to all p-values. g) A χ-squared test and odds ratio effect size are used for the covariates treated as categorical variables (donor ID and technical replicate ID). (h) A Kruskal-Wallis test and Cohen’s d effect size are used for continuous variables (E:T ratio, RASA2KO, and time). Positive Cohen’s d for time means the cluster is enriched at late time points, high E:T ratios, or high RASA2KO titrations.

Enrichment of SIFT clusters between experimental conditions

We identified clusters that were enriched for certain experimental conditions over others. Such statistical testing is critical for quantifying how experimental perturbations alter the abundance of different cellular states captured by our phase images. In this dataset, we tested for cluster enrichment against the experiment variables of technical replicate ID, donor ID, time, RASA2KO titration, and E:T ratio (Fig. 4a-f). We treated time, RASA2KO titration, and E:T ratio as continuous variables while donor ID and technical replicate ID were treated as categorical for statistical testing (see Methods). After false-discovery correction, no cluster was enriched or depleted beyond for a specific donor or technical replicate, indicating that phase images were largely similar across donors and technical replicates. However, there were nine cluster × covariate interactions across time, E:T ratio, and RASA2KO titration covariates showing substantial cluster enrichment (; Fig. 4g,h). We demonstrate that the effect sizes of all relationships remain similar, but p-values become overpowered when repeating analysis shown in Figure 3 and Figure 4 using all D= 3,389,740 SIFT keypoints (Supplementary Text; Supplementary Fig. S14). A table of all p-values and effect sizes can be found in Supplementary File 1.

Looking into these enrichments more carefully, we found that edge effect clusters (2 and 3) were enriched at high E:T ratios and high RASA2KO titrations (, Cohen’s d =0.45 for cluster 2 × E:T; , Cohen’s d =0.28 for cluster 2 × RASA2KO; , Cohen’s d =0.29 for cluster 3 × E:T; , Cohen’s d =0.18 for cluster 3 × E:T), potentially because there are fewer cellular keypoints to be detected when T cells successfully limit cancer cell growth. Interestingly, both singlet clusters (1 and 4) were enriched at early time frames (, Cohen’s d =-0.49 for cluster 1 × time; , Cohen’s d =-0.19 for cluster 4 × time) but were not enriched for any RASA2KO titrations and E:T ratios. These results suggest that singlets disappear over time regardless of whether their absence is caused by (1) increased T cell killing of cancer cell singlets at high RASA2KO T cell concentrations, or (2) unconstrained cancer cell proliferation that turns singlets into aggregates at low RASA2KO T cell concentrations. We found that both aggregate cell clusters (5 and 6) were enriched at late time frames (, Cohen’s d =0.36 for cluster 5 × time; , Cohen’s d =0.21 for cluster 6 × time), depleted at high E:T ratios (, Cohen’s d =-0.09 for cluster 5 × E:T; , Cohen’s d =-0.25 for cluster 6 × E:T), and depleted at high RASA2KO titrations (, Cohen’s d =-0.09 for cluster 5 × RASA2KO; , Cohen’s d =-0.16 for cluster 6 × RASA2KO). The depletion of aggregates in wells with high RASA2KO T cell concentrations shows that the presence of these modified T cells limits the absolute number of cancer cells in aggregates. Moreover, these findings suggest that the lower entropy at high RASA2KO T cell concentrations (Fig. 1) were a consequence of anti-cancer T cell activity creating more empty space between cancer aggregates in the RFP masks rather than more cancer cells belonging to aggregates. A graphical summary of these findings and their relationship to underlying cellular dynamics is illustrated in Figure 5.

Fig. 5.

Graphical abstract of results. Schematic diagram of cancer cell aggregation dynamics from early (left column) to late time frames (right column) at low RASA2KO T cell concentrations (top row) and high RASA2KO T cell concentrations (top row). Arrow annotations along the x-axis represent SIFT singlets decreasing over time and SIFT aggregates increasing over time at both RASA2KO T cell concentrations. Arrow annotations along the y-axis represent features comparisons between the two experimental conditions at late time frames.

Discussion

Our segmentation-free live-cell behavioral analysis (SF-LCBA) framework enabled the characterization of cancer cell aggregation dynamics at varying RASA2KO TCR T cell concentrations in low-resolution LCI co-culture experiments. By applying spatial entropy to the RFP fluorescence channel, we captured spatial patterns of cancer cell proliferation and death that extend beyond total cancer cell burden. Additionally, by clustering scale-invariant feature transform (SIFT) keypoints derived from phase images, we identified recurring local cellular topologies such as well edges, cell singlets, and multi-cellular aggregates, and we associated their abundances with distinct experimental conditions. Together, these analyses revealed that cancer cell aggregation was most effectively disrupted in co-cultures with high RASA2KO TCR T cell titrations and high effector-to-target (E:T) ratios, corroborating the findings of earlier work¹³ and validating aggregation inhibition as a potentially valuable phenotypic marker of T cell therapeutic efficacy.

While the high E:T ratio and RASA2KO titration experimental conditions were faithfully stratified by RFP area alone in our dataset, the potential of SF-LCBA lies in the orthogonal information it offers. Whereas RFP area measures total cancer cell burden, spatial entropy and unsupervised enrichment testing of SIFT keypoints reflect the spatial organization of cancer cells and their morphological state–particularly the degree of aggregation versus dispersion. In solid tumor contexts, where immune evasion is often mediated by the formation of dense tumor cell aggregates⁵⁴, assessing the ability of T cell therapies to infiltrate and dismantle these evasive cancer cell aggregate structures is highly relevant. Our results suggest that SF-LCBA metrics may become a critical readout in future studies designed to differentiate T cell products that enhance elimination of cancer cell aggregates from those that merely clear cancer cell singlets more effectively.

This study has several important limitations. Chief among them is the relatively low spatial resolution of our LCI data (4.975~μm/pixel), which is near the lower limit for detecting individual T cells (typically 5–10 μm in diameter⁵⁵). As a result, most SIFT keypoints detected in singlet-rich images were centered on cancer cells (typically 10–20 μm in diameter⁵⁶), with very few SIFT cluster singlet keypoints being centered on T cells. Additionally, low image resolution likely hindered our ability to identify mitotic cells as distinct keypoint clusters. Downsampled images at even lower resolutions produced a substantial drop-off in the number of SIFT keypoints detected, indicating that users should not perform any experiments with resolution worse than 5~μm/pixel. We believe that applying SF-LCBA to images of ≥ 10x magnification could allow for more detailed description cancer and T cell morphologies, their patterns of co-localization, and, consequently, their phenotypic cell states.

A second limitation relates to the interpretation of statistical significance in our SIFT keypoint enrichment analysis. Because SIFT can detect thousands of keypoints per image, statistical comparisons between groups are performed on large sample sizes, which naturally yields extremely small p-values, even when biological effects are modest. Moreover, SIFT parameters directly influence the total number of detected keypoints, meaning that p-values can become smaller simply by increasing algorithmic sensitivity rather than by underlying biological differences. For these reasons, we caution that future applications of SF-LCBA should emphasize effect sizes over p-values. While varying E:T ratio or RASA2KO titration produced large effect sizes in this study, paying attention to effect size would be essential when studying subtler perturbations such as comparing cells from different donors.

Other limitations stem from the SIFT algorithm itself. SIFT was originally designed for robust image registration by placing keypoints at high-contrast regions of the image. Consequently, many SIFT keypoints are located at the highest contrast regions of cells, most notably their borders with other cells or background. It is possible that SIFT is capturing more subtle differences such as cancer-T cell interactions or morphology within cellular aggregate clusters; however, we lacked the image resolution or T cell fluorescent markers–such as CD4, CD8, CTLA-4, or TOX–to investigate such claims. In non-biological computer vision domains, SIFT “visual word” frameworks that aggregated many SIFT descriptor vectors together served as the backbone for many popular object detection classifiers before the rise of scalable neural networks^57–59, suggesting the potential of SIFT-based frameworks for detecting large multi-cellular structures in live-cell imaging data. Again though, data would be the primary limitation for these frameworks since cluster interpretation is much more challenging for brightfield cellular images compared to natural images containing highly-structured objects like mountains and highways. Future work where SIFT, or analogous unsupervised image encoder^60,61, embeddings represent image patches spanning tens to hundreds of microns could enable the detection of more complex spatial structures, such as immune infiltration gradients within aggregates, if used to analyze similar data with fluorescent T cell markers. For analysis of multi-cellular image patches, we believe unsupervised or pretrained image encoders that directly ingest 3-channel^60,61 or >3-channel^62,63 images would be superior to SIFT, which only operates here on single-channel images, enabling embeddings that better reflect local cell type composition and activity.

Despite these limitations, SF-LCBA opens up new opportunities for analyzing complex co-culture datasets in which traditional cell segmentation and tracking are infeasible. Our approach could be especially powerful in future experiments involving more complex microenvironments, including three or more interacting cell types. For example, adding dendritic cells or macrophages to T cell–cancer cell co-cultures would better recapitulate the tumor microenvironment⁵⁴ and allow for perturbation studies across multiple immune cell types, but segmentation in these complex cell cultures is infeasible due to their high cell density and prevalence of multi-cellular interactions. Similarly, the segmentation-free nature of SF-LCBA means it could also be extended from adherent cell lines shown in this manuscript to cancer cell suspensions and patient-derived organoids, further broadening the translational potential^22,23. Furthermore, incorporating additional fluorescent markers, such as cell viability stains, cell cycle reporters, or CD4/CD8 labels, would greatly enhance interpretability of keypoint clusters and support more nuanced phenotypic classifications^50,64,65. Notably, even sparse use of additional fluorescent channels would suffice, as SIFT embeddings are computed from the phase channel and require fluorescence data only for cluster annotation. Collectively, the methodological improvements presented in this paper move the field closer to an ideal unsupervised framework for behavioral phenotyping in complex, multicellular systems.

Author contributions

L.E., B.E.E., A.M., J.C., and A.V. conceived the study. J.C. and A.M. generated, preserved, and shared the imaging data. L.E., A.C.W., and M.S. conducted the computational experiments. L.E., A.C.W., B.E.E., J.C., and A.M. analyzed the results. L.E., A.C.W., and B.E.E. wrote the manuscript. All authors reviewed the manuscript.

Funding

The Marson laboratory has received research support from the Parker Institute for Cancer Immunotherapy, the Emerson Collective, Arc Institute, Sanofi, GlaxoSmithKline, and Gilead and reagents from Genscript and Illumina. J.C. was supported by NIH/NCI K08, 11271K08CA252605-01, a Burroughs Wellcome Fund Career Award for Medical Scientists, the Lydia Preisler Shorenstein Donor Advised Fund, and the Parker Institute for Cancer Immunotherapy, and the Pascarella Scholars Fund. L.E., A.C.W., and B.E.E. were funded in part by grants from the Parker Institute for Cancer Immunology (PICI), the Chan-Zuckerberg Institute (CZI), NIH NHGRI R01 HG012967, and NIH NHGRI R01 HG013736.

Data availability

All imaging data used in this manuscript are publicly available at: https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD2001

Declarations

Code availability

The SF-LCBA software package, along with accompanying code to reproduce the figures shown in this manuscript, is publicly available at https://github.com/bee-hive/sflcba. Instructions for installation and usage can be found in the repository’s README.md file.

Competing interests

A.V. currently works at Genentech. A.M. is a cofounder of Site Tx, Arsenal Biosciences, and Survey Genomics, serves on the boards of directors at Site Tx, and Survey Genomics, is a member of the scientific advisory boards of Site Tx, Arsenal Biosciences, Cellanome, Spotlight Therapeutics, Survey Genomics, NewLimit, Amgen, and Tenaya, owns stock in Arsenal Biosciences, Site Tx, Cellanome, Spotlight Therapeutics, NewLimit, Survey Genomics, Tenaya and Lightcast and has received fees from Site Tx, Arsenal Biosciences, Cellanome, Spotlight Therapeutics, NewLimit, Abbvie, Gilead, 23andMe, PACT Pharma, Tenaya, Lightcast, Vertex, Merck, Amgen, GLG, ClearView Healthcare, and AlphaSights. A.M. is an investor in and informal advisor to Offline Ventures and a client of EPIQ. B.E.E. is on the Scientific Advisory Board for Arrepath and Freenome. None of the other authors have competing interests to declare.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1038/s41598-026-50029-9.